Announcements

The TERMINATING_STRING setting, a new (and quite minor!) feature

Some days ago, I added a new setting. If, at the top of your grammar, you write: TERMINATING_STRING=”some string”; this means that the input you’re parsing is guaranteed to end with that string. If the file ends with that string already, then it does nothing. Otherwise, it tacks that string to the end. In actual …

The TERMINATING_STRING setting, a new (and quite minor!) feature Read More »

“You can’t get there from here!” — The Problem of Context-Sensitive Tokenization

(N.B. Note added 13 June 2021: This article is useful in terms of understanding how to add token hooks to code. However, in terms of solving the specific problem outlined, the article is obsolete. See here for the updated solution.) Since I picked up my work on the JavaCC codebase at the end of 2019, …

“You can’t get there from here!” — The Problem of Context-Sensitive Tokenization Read More »

New Feature: The =>|| delimiter stands for “scan up to here”

Revisiting LOOKAHEAD Redux Not so long ago, I had a sort of eureka moment when I realized that the legacy LOOKAHEAD construct was fundamentally half-baked or broken. Not only that, but I started narrowing in on how to definitely address the issue! Well, let’s get concrete. Suppose we have a production that looks kind of like this: FooBar : …

New Feature: The =>|| delimiter stands for “scan up to here” Read More »

Turning to Semantic Lookahead

In my last post I reported that I had finally fixed the way syntactic lookahead works in JavaCC. This leads to the natural question: what about semantic lookahead? Well, what is historically called semantic lookahead in JavaCC is actually misnamed. It’s really just a bit of java code that is used to determine whether to enter an expansion or not. Something like: …

Turning to Semantic Lookahead Read More »

Bugfix! Nested syntactic lookahead now works correctly!

Fixed a Longstanding Bug Known Issue in JavaCC: Nested syntactic lookahead now works! There is a quite serious bug (let’s just call it by what it is, shall we?) of very long standing in JavaCC. Basically, syntactic lookaheads do not nest. This bug is documented in Theodore Norvell’s JavaCC FAQ, which by the way, is the best …

Bugfix! Nested syntactic lookahead now works correctly! Read More »