mirror of
https://github.com/atom/atom.git
synced 2026-04-28 03:01:47 -04:00
Don't terminate tokenization if stack size changes
Previously Python import blocks were not tokenizing correctly since the loop was prematurely terminating when a match at the end of the line was reached and no tokens were generated for it. This approach was incorrect since the tokenizer may have just popped a rule and another loop could possibly pop more rules. Now this early termination is only performed if the stack size hasn't changed.
This commit is contained in:
@@ -146,7 +146,7 @@ class TextMateGrammar
|
||||
|
||||
tokens.push(nextTokens...)
|
||||
position = tokensEndPosition
|
||||
break if position is line.length and nextTokens.length is 0
|
||||
break if position is line.length and nextTokens.length is 0 and ruleStack.length is previousRuleStackLength
|
||||
|
||||
else # push filler token for unmatched text at end of line
|
||||
if position < line.length
|
||||
|
||||
Reference in New Issue
Block a user