- Needed a shift counter, rather than a read counter, as ::token was
lexing '-10d' into '-' and '-10d', which when evaluated is '--10d',
which yields 10d.
- Added downgrade from typeNumber to typeIdentifer if a number is followed by a
letter, thus supporting "9th" as an identifier. This causes much breakage.
- Lexer now makes a speculative legacy dateformat parse whenever it encounters
a decimal digit. This assumes that rc.dateformat begins with a numeric date
element, which is a restriction, but not a big one.
- When a Lexer::typeIdentifier is found, it can be compared to a list of
other tokens, with the possibility of changing the type. This applies to
tokens that are longer than the four-character lookahead in the Lexer.
With this change, the Lexer can now identify all operators supported by Eval,
and therefore the Lexer can be used on all Eval input expressions. This is
because all the evaluator needs to know is the distinction between operators
and operands.
- Implemented Lexer::word, which is just like ::token, but does not
understand dates, durations or operators.
- Implemented Lexer::split, which uses Lexer::word.
- Added unit tests.
