Corey Johnson
83329567b9
Use the file's first line to determine the grammar.
...
Only when it cannot be determined by the file's extension fails. Fixes #75
2012-11-01 17:47:31 -07:00
Nathan Sobo
031da13316
Remove unused methods from native OnigRegExp
...
`OnigScanner` takes over these duties
2012-10-25 12:23:37 -06:00
Nathan Sobo
33c6fe8009
Remove regex from TextMateGrammar's Pattern class
...
We build a native `OnigScanner` object at the `Rule` level which tries to match every regex for the entire rule, rather than using an individual regex for each pattern from the JS side.
2012-10-25 12:07:13 -06:00
Corey Johnson & Nathan Sobo
253a07e2c4
Add comment explaining the need for appending \n when scanning lines with a TextMate grammar
2012-10-16 16:07:54 -07:00
Corey Johnson
6171ea33e1
Add support for TextMate grammars with newlines in their regexes
2012-10-16 15:58:11 -07:00
Kevin Sawicki
8ce6810179
Support bundles with foldingStopMarker under Preferences/
2012-10-09 16:57:43 -07:00
Nathan Sobo
4e49fbec72
TextMate grammars can include other grammars
2012-09-28 17:28:40 -06:00
Corey Johnson & Nathan Sobo
2722fe7d72
Rename stack to ruleStack
2012-09-28 09:41:04 -07:00
Corey Johnson & Nathan Sobo
3a66aab59d
getLineTokens must clone the ruleStack
2012-09-28 09:40:06 -07:00
Corey Johnson & Nathan Sobo
ae4db9d238
💄
2012-09-28 09:36:37 -07:00
Corey Johnson & Nathan Sobo
66ac4503a7
Return empty token array for zero-length matches
2012-09-28 09:31:15 -07:00
Corey Johnson & Nathan Sobo
255e58b4f8
Remove unnecessary Array coercion
2012-09-28 09:06:41 -07:00
Kevin Sawicki
4ccaa6f714
Terminate when tokens end position is undefined
...
This ensures the position is advanced for zero-length
matches and only terminates when the end position for
the tokens is no longer valid.
2012-09-27 19:07:08 -07:00
Nathan Sobo
32e36700ae
Add native OnigScanner, which handles matching multiple regexes on a line
...
This cuts the tokenization time from 250ms to 70ms by avoiding js <-> native overhead
2012-09-27 13:04:14 -06:00
Nathan Sobo
2dbba203a0
Allow begin/end rules with back-references to be nested
2012-09-26 12:10:00 -06:00
Nathan Sobo
1f8ce50261
Memoize Rule.prototype.getIncludedPatterns to shave a cheap 2% off profile
2012-09-26 09:57:35 -06:00
Corey Johnson
70e31b9677
getNextMatch no longer called
2012-09-24 11:17:31 -07:00
Corey Johnson & Nathan Sobo
2d3c06b86d
Handle begin/end patterns with no name or contentName
...
I'm looking at you ruby.
2012-09-07 18:54:54 -07:00
Corey Johnson & Nathan Sobo
12e2fbdb14
Add support for back-references in TextMate grammars
2012-09-07 15:27:13 -07:00
Corey Johnson & Nathan Sobo
b00830cbc8
A pattern with no name or contentName does not insert undefined into scope array
2012-09-07 11:20:56 -07:00
Corey Johnson & Nathan Sobo
10c36191ec
Zero-length start patterns work.
...
Add zero-length captures back into OnigRegEx.BuildCaptureIndices to see zero length captures. Handle zero-length captures by continuing to scan line. Does not handle infinite loop possibility yet.
2012-09-07 10:07:54 -07:00
Corey Johnson
f75315a568
Remove Rule.compileRegex
2012-09-06 15:30:50 -07:00
Corey Johnson
4f2cc1f856
Add OnigRegExp.captureIndices(string, index, regexes)
...
Allows us to know if a zero-length regex matched.
2012-09-06 15:26:42 -07:00
Corey Johnson
afbe99454a
Rename getRegexPatternPairs to getIncludedPatterns and getAllPatterns
2012-09-06 09:24:30 -07:00
Nathan Sobo
c3e748a17c
WIP: Use rake to start compiling resources (like require.coffee)
2012-08-26 16:29:46 -05:00
Nathan Sobo
c2c8724ad0
Initial commit
2012-08-10 13:32:19 -06:00
Corey Johnson
7477511a9f
Create empty token on empty lines
2012-08-09 15:17:46 -07:00
Corey Johnson & Nathan Sobo
7fe6a05594
Folding no longer relies on Ace.
...
Indentation is used to determine foldable regions, if the last line matches TextMate's foldEndRegex it is included in the foldable region.
2012-08-08 17:42:11 -07:00
Corey Johnson & Nathan Sobo
2a945838f5
Return absolute minimum capture data from OnigRegExp
2012-08-08 12:13:54 -07:00
Corey Johnson & Nathan Sobo
472e5348e2
Update TextMateGrammar to use optimized array-based capture trees
2012-08-08 11:04:36 -06:00
Nathan Sobo
f559ad73a7
Make grammar handle lines that don't match *any* of the current rule's patterns
...
It's possible to get a match based on a lookahead, but not actually have any non-empty capture groups. This should be considered a failed match.
2012-08-08 08:59:13 -06:00
Corey Johnson & Nathan Sobo
39dbcb15b7
Include end pattern in compiled regexes. Also Compile regexes for pushed rules.
2012-08-07 12:07:24 -06:00
Nathan Sobo
d520d6c859
WIP: Optimization: compile a single regex for each grammar rule
...
We compile a giant regex out of all the individual regexes for each pattern by or'ing together a capture group for each one. Then we use the index of the matched capture group to determine which pattern actually matched, and adjust the capture indexes of the subtree to make it appear to start from index 0, so the capture indices on the pattern align properly. There is still broken-ness on more complex patterns, but basic patterns and patterns w/ captures work.
2012-08-06 18:32:55 -06:00
Nathan Sobo
7bc6a5f24b
TextMateGrammar handles nested capture groups and trailing lookahead captures
...
Our previous implementation only allowed for a single layer of capture groups. Now we can have captures within captures. I achieved this by converting the match into a tree before generating tokens. If there are any capture scopes specified, then we will always emit a token for every capture group in the match. This may create some redundant tokens (a serious of 2 or more tokens with the same scopes), but it will at least be technically correct. I think the overhead of removing these redundancies exceeds the cost of maintaining them for now.
2012-08-06 16:35:33 -06:00
Nathan Sobo
0d6d16a438
Move TextMateGrammar.buildCaptureTree to .getCaptureTree method on OnigRegExp
2012-08-06 11:43:44 -06:00
Nathan Sobo
32fc042929
Add TextMateGrammar.buildCaptureTree, which organizes nested captures in a tree
...
This will be a weapon in dealing with capture groups that nest within other capture groups, and also helps deal with trailing lookahead groups that don't belong in the main match. I made it a class method because it's stateless and that made it easier to test.
2012-08-03 23:52:58 -06:00
Corey Johnson & Nathan Sobo
0e1d3fe3a9
wip: fixing capture group problems
2012-08-03 11:38:45 -07:00
Corey Johnson & Nathan Sobo
1a243adfcf
All specs pass w/ TextMateGrammar for tokenization (auto-indent disabled)
2012-08-03 12:00:05 -06:00
Corey Johnson
b378620c54
Add TextMateBundle which handles all bundle loading
2012-08-02 11:38:14 -07:00
Corey Johnson & Nathan Sobo
0cdc042e71
Use text as the default grammar
2012-08-01 17:24:28 -07:00
Corey Johnson & Nathan Sobo
370e0b896e
Add TextMateGrammar.loadFromBundles()
2012-08-01 16:58:31 -07:00
