Netbeans lexer problem : Fix the lexer to not return null token in this state

Netbeans lexer problem : Fix the lexer to not return null token in this state

2018/03/13 Netbeans 0

I have met the problem “Fix the lexer to not return null token in this state.” (see below) and used one week to fix it by hacking netbeans source code. I found out Netbeans will not pass the whole file to lexer to parse. So if your antlr grammar used “skip” syntax, it has a probability of the skipped token become the last token to be parsed, then this exception happens. To fix this you need to check if info.input().readLength() still larger than 0, if so, just return a fake token.

java.lang.IllegalStateException: Lexer com.github.mcheung63.netbeans.antlr.syntax.antlr3.Antlr3EditorLexer@2057cbb3
  returned null token but lexerInput.readLength()=1
  lexer-state: null
  tokenStartOffset=1004, readOffset=1005, lookaheadOffset=1006
  Chars: "\n" - these characters need to be tokenized.
Fix the lexer to not return null token in this state.

read count : 1

Leave a Reply

Your email address will not be published. Required fields are marked *