Month: July 2023

All sorts of things you always wanted to know about tokenization but were afraid to ask (Part I)

Let’s consider the multiline comment in Java (and C/C++/C#, among others) which, you surely know, looks like this: /* * Comment text. */ This is an interesting construct. Paradoxically, it is extremely simple — I mean to describe in natural, human language (English or whatever) — but shockingly difficult to express in CongoCC. Or, that …

All sorts of things you always wanted to know about tokenization but were afraid to ask (Part I) Read More »