Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Learnt from: CRRepo: RustPython/RustPython PR: 0File: .github/copilot-instructions.md:0-0Timestamp: 2025-09-11T05:06:59.971ZLearning: Applies to {vm,stdlib}/**/*.rs : Use RustPython macros (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust

Learnt from: CRRepo: RustPython/RustPython PR: 0File: .github/copilot-instructions.md:0-0Timestamp: 2025-09-11T05:06:59.971ZLearning: Applies to {vm,stdlib}/**/*.rs : Use RustPython macros (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust

Learnt from: CRRepo: RustPython/RustPython PR: 0File: .github/copilot-instructions.md:0-0Timestamp: 2025-09-11T05:06:59.971ZLearning: Applies to Lib/**/*.py : Minimize modifications to CPython standard library files; only make minimal changes to work around RustPython limitations

Learnt from: CRRepo: RustPython/RustPython PR: 0File: .github/copilot-instructions.md:0-0Timestamp: 2025-09-11T05:06:59.971ZLearning: Applies to extra_tests/**/*.py : Use ruff to lint custom Python code

stdlib/src/tokenize.rs (4)

39-42:Don't swallow non-StopIteration errors fromreadline.

Catching every exception and treating it as EOF hides real failures (TypeError, UnicodeDecodeError, etc.). OnlyStopIteration should terminate the stream; all other errors must propagate.

Apply this diff to fix the error handling:

-            let raw_line = match self.readline.invoke((), vm) {-                Ok(v) => v,-                Err(_) => return Ok(String::new()),-            };+            let raw_line = match self.readline.invoke((), vm) {+                Ok(v) => v,+                Err(exc) => {+                    if exc.fast_isinstance(vm.ctx.exceptions.stop_iteration) {+                        return Ok(String::new());+                    }+                    return Err(exc);+                }+            };

---

116-138:End-of-file tuple needs to come from the real EOF token.

Synthesizing the EOF tuple here hardcodes(line, -1) and reuses the previous line text, so consumers see the wrong coordinates and line content. Instead, retrieve the parser'sEndOfFile token, set it asprev_token, and let the existing emission path produce the correct tuple.

Apply this diff to use the parser's EOF token:

                 if nline.is_empty() {                     state.eof = true;-                    *zelf.state.write() = state.clone();--                    let line_num = &state.start().0;-                    let out = vm-                        .ctx-                        .new_tuple(vec![-                            token_kind_value(TokenKind::EndOfFile).to_pyobject(vm),-                            vm.ctx.new_str("").into(),-                            vm.ctx-                                .new_tuple(vec![line_num.to_pyobject(vm), (-1).to_pyobject(vm)])-                                .into(),-                            vm.ctx-                                .new_tuple(vec![line_num.to_pyobject(vm), (-1).to_pyobject(vm)])-                                .into(),-                            vm.ctx.new_str(state.current_line()).into(),-                        ])-                        .into();-                    return Ok(PyIterReturn::Return(out));+                    if let Some(eof) = state+                        .tokens+                        .iter()+                        .rev()+                        .find(|token| token.kind() == TokenKind::EndOfFile)+                    {+                        state.prev_token = Some(*eof);+                        break *eof;+                    }+                    *zelf.state.write() = state.clone();+                    return Ok(PyIterReturn::StopIteration(None));                 }

---

151-154:Don't trim token text whenextra_tokens is false.

trim() removes indentation whitespace and other significant padding, causing INDENT tokens to become empty strings. CPython preserves the exact lexeme. Use the slice as-is without trimming.

Apply this diff to preserve the exact lexeme:

-            let mut token_repr = &state.source[state.range()];-            if !zelf.extra_tokens {-                token_repr = token_repr.trim();-            }+            let token_repr = &state.source[state.range()];

---

178-180:extra_tokens must default toFalse.

CPython'sTokenizerIter allows callingTokenizerIter(readline) without naming the flag. Requiring the argument here breaks API compatibility.

Apply this diff to makeextra_tokens optional with a default:

         #[pyarg(positional)]         readline: ArgCallable,-        #[pyarg(named)]-        extra_tokens: bool,+        #[pyarg(named, default = "false")]+        extra_tokens: bool,         #[pyarg(named, optional)]         encoding: Option<PyStrRef>,

stdlib/src/tokenize.rs (2)

108-109:Reminder: Address the TODO comment for error handling.

The TODO indicates that error checking (e.g., raising SyntaxError for parse errors) is missing. Thestate.errors field is populated but never checked.

Do you want me to generate code to checkstate.errors and raise appropriate SyntaxError exceptions, or open a new issue to track this task?

---

200-207:Re-parsing the entire source on each line may be inefficient.

Thepush_line method re-parses the entire accumulated source text each time a new line is added. For large files, this could become a performance bottleneck with O(n²) behavior.

However, this approach may be necessary for correct tokenization since Python's grammar is context-sensitive (e.g., for indent/dedent tracking). Consider profiling to determine if incremental parsing or caching strategies would be beneficial.

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Learnt from: CRRepo: RustPython/RustPython PR: 0File: .github/copilot-instructions.md:0-0Timestamp: 2025-09-11T05:06:59.971ZLearning: Applies to {vm,stdlib}/**/*.rs : Use RustPython macros (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust

Learnt from: CRRepo: RustPython/RustPython PR: 0File: .github/copilot-instructions.md:0-0Timestamp: 2025-09-11T05:06:59.971ZLearning: Applies to Lib/**/*.py : Minimize modifications to CPython standard library files; only make minimal changes to work around RustPython limitations

Learnt from: CRRepo: RustPython/RustPython PR: 0File: .github/copilot-instructions.md:0-0Timestamp: 2025-09-11T05:06:59.971ZLearning: Applies to {vm,stdlib}/**/*.rs : Use RustPython macros (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust

stdlib/src/tokenize.rs (5)

121-143:EOF tuple must use the parser's actual EndOfFile token.

The code synthesizes the EOF tuple with hardcoded(-1) positions (lines 132, 135) and reusesstate.current_line(), which produces incorrect coordinates and line content. Instead, retrieve the parser'sEndOfFile token fromstate.tokens, set it asprev_token, and let the standard token-emission path (lines 145-175) produce the tuple with correct metadata.

Apply this approach:

                 let nline = zelf.readline(vm)?;                 if nline.is_empty() {                     state.eof = true;-                    *zelf.state.write() = state.clone();--                    let line_num = &state.start().0;-                    let out = vm-                        .ctx-                        .new_tuple(vec![-                            token_kind_value(TokenKind::EndOfFile).to_pyobject(vm),-                            vm.ctx.new_str("").into(),-                            vm.ctx-                                .new_tuple(vec![line_num.to_pyobject(vm), (-1).to_pyobject(vm)])-                                .into(),-                            vm.ctx-                                .new_tuple(vec![line_num.to_pyobject(vm), (-1).to_pyobject(vm)])-                                .into(),-                            vm.ctx.new_str(state.current_line()).into(),-                        ])-                        .into();-                    return Ok(PyIterReturn::Return(out));+                    if let Some(eof) = state+                        .tokens+                        .iter()+                        .find(|token| token.kind() == TokenKind::EndOfFile)+                    {+                        state.prev_token = Some(*eof);+                        break *eof;+                    }+                    *zelf.state.write() = state.clone();+                    return Ok(PyIterReturn::StopIteration(None));                 }

---

156-159:Remove trim() to preserve exact token lexemes.

Callingtrim() whenextra_tokens is false (line 158) strips significant whitespace, causing INDENT tokens to become empty strings. CPython's tokenizer preserves the exact lexeme. Use the slice as-is.

Apply this diff:

-            let mut token_repr = &state.source[state.range()];-            if !zelf.extra_tokens {-                token_repr = token_repr.trim();-            }+            let token_repr = &state.source[state.range()];

---

183-184:Makeextra_tokens optional with defaultfalse.

CPython allowsTokenizerIter(readline) without specifyingextra_tokens. The current declaration requires the argument, breaking API compatibility.

Apply this diff:

-        #[pyarg(named)]-        extra_tokens: bool,+        #[pyarg(named, default = "false")]+        extra_tokens: bool,

---

227-243:Token ordering skips tokens that overlap the current range.

Whenprev_token isNone,self.range() returnsTextRange::default() (0..0). The comparisonrange.ordering(self.range()) > Ordering::Greater (line 236) only matches tokenscompletely after the current range. Tokens starting at position 0 (e.g., 0..1) overlap with 0..0, returningOrdering::Equal, and are skipped. The first token in the source is never yielded.

Fix by comparing start positions:

-                if matches!(range.ordering(self.range()), Ordering::Greater) {+                if range.start() > self.range().end() {                     self.prev_token = Some(*token);                     return self.prev_token;                 }

---

286-391:Critical token mapping errors: duplicates and CPython conflicts.

Multiple critical issues remain:

1. Duplicate value 62 (lines 384-385): BothComment andTStringStart map to 62
2. Incorrect Comment mapping (line 384): Maps to 62, but CPython 3.14 hasCOMMENT=61
3. FString conflicts (lines 381-383): Values 59, 60, 61 correspond toSOFT_KEYWORD,ERRORTOKEN, andCOMMENT in CPython 3.14
4. Exclamation collision (line 380): Maps to 54, which is CPython'sOP token
5. Misleading comments (lines 385-387): "3.14 compatible" claims are inaccurate

These mappings must be corrected to either:

• Match CPython 3.14 standard token values exactly, or
• Use reserved/unassigned values (e.g., 100+) to avoid collisions

Run this verification to confirm no remaining duplicates after fixes:

#!/bin/bash# Extract all token value assignments and check for duplicatesrg -n"TokenKind::\w+ => \d+" stdlib/src/tokenize.rs| \  gawk'{print $NF}'| sort| uniq -d| \  xargs -I {}echo"Duplicate token value found: {}"

stdlib/src/tokenize.rs (2)

189-212:State structure is correct; consider performance implications.

Thepush_line method reparses the entire accumulated source on each call (line 208). While correct, this has O(n²) complexity for n lines. This is acceptable for now but may need optimization for large files.

---

113-118:TODO: Implement syntax error handling.

The TODO on line 114 indicates that parse errors fromstate.errors are not currently raised asSyntaxError. This means invalid Python syntax may be silently ignored. Consider implementing error checking in a follow-up to ensure proper error reporting.

Based on learnings.

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Learnt from: CRRepo: RustPython/RustPython PR: 0File: .github/copilot-instructions.md:0-0Timestamp: 2025-09-11T05:06:59.971ZLearning: Applies to {vm,stdlib}/**/*.rs : Use RustPython macros (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust

Learnt from: CRRepo: RustPython/RustPython PR: 0File: .github/copilot-instructions.md:0-0Timestamp: 2025-09-11T05:06:59.971ZLearning: Applies to Lib/**/*.py : Minimize modifications to CPython standard library files; only make minimal changes to work around RustPython limitations

Learnt from: CRRepo: RustPython/RustPython PR: 0File: .github/copilot-instructions.md:0-0Timestamp: 2025-09-11T05:06:59.971ZLearning: Applies to {vm,stdlib}/**/*.rs : Use RustPython macros (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust