When we produce tokens, we produce the same arrays that Ripper
does. However, we add a couple of convenience methods onto them to make them a little easier to work with. We delegate all other methods to the array.
Methods
Instance Public methods
event() Link
The type of the token.
location() Link
The location of the token in the source.
state() Link
The state of the lexer when this token was produced.