TokenStream
- Rust course
- Advanced
- Lesson 52/81
- 4 min
Produce streaming code with TokenStream.
TokenStream
TokenStream is the form in which the compiler sees source code: a sequence of token trees. There are four kinds of tokens: Ident (a name, a keyword), Punct (:, +, ->), Literal (42, "metin") and Group (a sub-sequence inside parentheses, brackets or braces):
fn kare(x: i32)
// Ident(fn) Ident(kare) Group(( Ident(x) Punct(:) Ident(i32) ))Whitespace and comments aren't tokens; the macro doesn't see them.
A small tokenizer
The compiler's first step is splitting text into tokens. A simple version: join runs of letters, digits and underscores, split other symbols one by one, skip whitespace:
fn belirtecler(s: &str) -> Vec<String> {
let mut sonuc = Vec::new();
let mut ad = String::new();
for c in s.chars() {
if c.is_alphanumeric() || c == '_' {
ad.push(c);
continue;
}
if !ad.is_empty() {
sonuc.push(std::mem::take(&mut ad));
}
if !c.is_whitespace() {
sonuc.push(c.to_string());
}
}
if !ad.is_empty() {
sonuc.push(ad);
}
sonuc
}Practice Split into tokens
Write the belirtecler function above. Print the result for "fn kare(x: i32)" with {:?}.
Starter code
fn main() {
// buraya yaz
}Sample solution
fn belirtecler(s: &str) -> Vec<String> {
let mut sonuc = Vec::new();
let mut ad = String::new();
for c in s.chars() {
if c.is_alphanumeric() || c == '_' {
ad.push(c);
continue;
}
if !ad.is_empty() {
sonuc.push(std::mem::take(&mut ad));
}
if !c.is_whitespace() {
sonuc.push(c.to_string());
}
}
if !ad.is_empty() {
sonuc.push(ad);
}
sonuc
}
fn main() {
println!("{:?}", belirtecler("fn kare(x: i32)"));
}Expected output
["fn", "kare", "(", "x", ":", "i32", ")"]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What does <code>$x:tt</code> match in declarative macros?
- A type
- An expression
- A line
- A single token tree: one token or a bracketed group Correct answer
Explanation: tt is the most flexible fragment; with $($t:tt)* anything can be forwarded as is.
2 Take the value and empty it
sonuc.push(std::mem::____(&mut ad));sonuc.push(std::mem::take(&mut ad));Explanation: mem::take takes the value and leaves the default (an empty String) in its place.
3 Token and kind
| Term | Meaning |
|---|---|
kare |
Ident |
: |
Punct |
42 |
Literal |
( x: i32 ) |
Group |