Lexer
- C course
- Expert
- Lesson 13/47
- 5 min
Learn to write a lexer (tokenizer).
What is a lexer?
A lexer (scanner) turns source code from a character stream into a stream of tokens. It is the first stage of a compiler; it drops whitespace and comments and groups meaningful units.
Each token has a type and a value: NUMBER, IDENT, OP, EOF. For example 42 + x yields three tokens: number 42, operator +, identifier x. A lexer is written as a state machine or with regex. On errors (such as an invalid character) it reports line and column. Its output feeds the parser.
Token types
Token types follow the source language's syntax. In a C-like language common types are numbers (NUMBER), identifiers (IDENT), operators (OP) and end-of-file (EOF).
Some lexers tag keywords (if, while) as a separate type. Each token carries line and column info that error messages use. Token types are usually an enum. The lexer's job is only lexical structure; syntactic rules belong to the parser. This split keeps the compiler modular.
Practice Print tokens
Define enum token_type { NUMBER, IDENT, OP, EOF_TOK }. Define void print_token(int type, const char *text) with body printf("T%d:%s\n", type, text);. In main call it for four tokens: (NUMBER, "42"), (IDENT, "x"), (OP, "+"), (EOF_TOK, "").
Starter code
#include <stdio.h>
// Define token_type enum
// Define print_token function and call it for four tokens
int main(void) {
return 0;
}Sample solution
#include <stdio.h>
enum token_type { NUMBER, IDENT, OP, EOF_TOK };
void print_token(int type, const char *text) {
printf("T%d:%s\n", type, text);
}
int main(void) {
print_token(NUMBER, "42");
print_token(IDENT, "x");
print_token(OP, "+");
print_token(EOF_TOK, "");
return 0;
}Expected output
T0:42 T1:x T2:+ T3:
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What does a lexer produce?
- Tokens Correct answer
- AST nodes
- Symbols
- Assembly
Explanation: The lexer splits the source text into a token stream.
2 Complete the token_type enum
enum token_type { NUMBER, IDENT, OP, ____ };enum token_type { NUMBER, IDENT, OP, EOF };Explanation: EOF marks the end of the token stream.
3 Match token types
| Term | Meaning |
|---|---|
NUMBER |
Numeric literal |
IDENT |
Identifier |
OP |
Operator |
EOF |
End of stream |