Netbeans lexer problem : Fix the lexer to not return null token in this state
2018-03-13
I have met the problem “Fix the lexer to not return null token in this state.” (see below) and used one week to fix it by hacking netbeans source code. I found out Netbeans will not pass the whole file to lexer to parse. So if your antlr grammar used “skip” syntax, it has a probability of the skipped token become the last token to be parsed, then this exception happens. To fix this you need to check if info.input().readLength() still larger than 0, if so, just return a fake token.
java.lang.IllegalStateException: Lexer com.github.mcheung63.netbeans.antlr.syntax.antlr3.Antlr3EditorLexer@2057cbb3 returned null token but lexerInput.readLength()=1 lexer-state: null tokenStartOffset=1004, readOffset=1005, lookaheadOffset=1006 Chars: "\n" - these characters need to be tokenized. Fix the lexer to not return null token in this state.
