Revisiting the whole third-party dependencies problem

Now, after dusting off the code from the Sourceforge trunk in preparation for a new release (which I intend to name FreeMarker NG) I am faced with this whole issue of the various third-party libraries needed to compile all the code.

Finally, I think this is a big mess that needs to get cleaned up. I took the whole codebase into IntelliJ and, of course, it didn’t compile (over a thousand errors) because I did not have a whole slew of dependencies. Well, the solution is obvious. I had to hunt down the various .jar files and stick them in a libs directory. But really, the whole exercise raises various questions. For starters, there is just the whole versioning problem. All these various libraries have different versions. This project has been neglected for about 10 years, so back then, presumably we were compiling against fairly recent versions of the various libraries, but now those are very old and obsolete. So what to do?

There is also the more general issue of whether the subcomponent that requires whatever library should really be supported moving forward anyway.

So, okay, breaking this down, we have:

  1. The Servlet/JSP API’s.
  2. The XML-related libraries
  3. Logging (or really, various logging abstractions!)
  4. support for other JVM-based languages, like Jython, JRuby, Rhino
  5. JUnit for the unit tests.

Am I missing something? (Well, if I am, I’ll fill it in later.)

Servlet/JSP API

On this point, it seems obvious that the main existing usage of FreeMarker is in the web space, so we have to keep the various Servlet related stuff in shape and, I believe, the JSP support as well. There is no question of removing this, so the real problem is which version (or versions) we are supporting. And whether what we have is up to date.

XML related API’s

This one makes me nervous. We had Xalan 2.7.0 as a build dependency and Xalan is not just a DOM/XPath thingy, but is a full implementation of XSLT. (Who uses XSLT nowadays, I wonder.) There is built-in support for the various XML support in the latest JDK’s, I’m sure (as there is support for just about anything) and my sense is that we should try to just use that and get rid of any external dependencies. There is also this thing called Jaxen, an XPath implementation. I want see about getting rid of that as a dependency.

Now, as a more general comment, I scoured the net looking for people discussing the use of FreeMarker to transform XML. Now, it does work very well for that task, I do know that, but what is the community (if any) of people using FreeMarker for this?

Logging API’s.

First some history. (As best I understand it.) Once upon a time in the early days of Java, some guy (actually, he did have a name, Ceki Gülcü ) came up with a logging library called Log4J that became a sort of de facto standard out there. Later, Sun Microsystems, rather than simply incorporated that library into the JDK, rolled up their own version, the java.util.logging API that was introduced in JDK 1.4. Some people thought that what Sun did was quite inferior (not only Ceki, apparently, other people) and preferred to keep using Ceki’s library and there were some other logging libraries and then some people saw the need for a logging abstraction, which is what this SFL4J thing is.

My own instinct (but I don’t absolutely trust it) is to use whatever is currently in the JDK and get rid of all these logging abstractions. But I really don’t know and would like some people to weigh in on it. (I really hate this logging API topic because it just seems like such a much ado about nothing.)

Supporting alternative languages for the JVM (Jython, JRuby, Rhino)

This is actually quite an interesting topic and I got into this just because I saw that it was just very much fun. So, I believe that I wrote some initial wrappers for Jython and JRuby objects as a sort of proof of concept. Also, an entire Jython runtime could be embedded in a FreeMarker template, but it is hard to really see the use case for this. I did it because it could be done and it remained in the code repository. Later, Attila Szegedi got very interested in this whole alternative JVM language issue and took ownership of that initial work I did and surely polished it off quite a bit.

There is, however, a problem with all that. There is not the slightest evidence I can find that anybody actually uses any of this!

Now, supporting all these various things has all sorts of versioning problems and I have not even checked whether any of this compiles against more recent versions of Jython, JRuby.

Well, to make a long story short, my solution for building everything on my own box was just to rip it all out. I don’t think anybody is using it and if, contrary to my belief, people are using it and will complain, then they can show up and complain and maybe we’ll put it back. Or maybe some other people will take ownership of all this and keep it working. I don’t think I am willing to do so, certainly not in a situation where there is zero evidence that anybody uses it. In fact, there is close to zero evidence that anybody (at least any significant number) are using FreeMarker to process/transform XML. However, in that case, the most basic usage patterns should still be kept working.

JUnit

We were compilng the test cases against the junit 3.7 jarfile and, on my local box, I have that as a dependency. But I have to assume that this is something very old and we either have to see about updating to a more recent version or switching to some other testing framework. This, ideally, would be something that some eager person wanting to get involved could take ownership of.

Oh, and I ripped out support for JRebel as well. I’m not actually a hundred percent sure what that is, but I don’t want to keep carrying these things forward. I’d rather get the codebase down to a more lean, mean level so that we can do some fundamental things without carrying forward all this cruft that, basically, nobody uses. Still, I start this as a topic so that people can put in their two cents on all this.

Hi,
nowadays basically nobody uses ANT anymore.
The most common alternative is Maven (mvn) but there are millions of alternatives available.
Using Maven has the benefit, that you don’t need to ship the libraries in a “lib” folder but they ghet downloaded automatically.

The choice of libraries depends on the minimum Java version you want to use. When e.g. using at least Java 8, you don’t need an explicit XML library any more. Additionally JUnit 4 is preferred over version 3. Logging facade default is to my knowledge “SLF4J” and as an implementation Log4J 2.x… If you want I can try to provide you with a basic Maven pom.xml to get started.

hth, Philip

As far as JSP/Servlet is considered, I think you can safely ditch anything that’s old. Most of the world uses Tomcat to this day, I’d presume. The current Tomcat 9 release says it supports Servlet 4.0 and JSP 2.3, I guess those are the only two versions that you should compile against and support.

While you’re at it, you’ll need to decide what’s the minimum Java level you want to support. I think the realistic choices are either Java 8 or 11. People today tend to stick with Java 8 because they experience pains with Java 9+ module system, but sooner or later it’ll get an untenable position. If I had to choose, I’d go with Java 11 as baseline but for wider adoption you might want to stick with Java 8. I think going below Java 8 today is silly.

Logging is indeed a chore. FM logging adapters were made so it could interact with multiple logging systems. These days it seems like SLF4J has pretty much won and nobody uses j.u.logging (that I know of). So if you wanted to choose a single logging API to adhere to, I’d choose SLF4J.

I think extra language supports can be safely removed. They could live as add-on modules if absolutely necessary.

XML support is now built into java through javax.xml APIs. There’s absolutely no need for FreeMarker to bind explicitly to any XML parser implementation.

Hi Philip, thanks for the comments. I appreciate you taking the time. (You and Attila both.) Because I set up this forum quite recently and, well, you know, it’s quite hard psychologically to be here all alone, feeling you’re being an idiot talking to yourself…

But, anyway, getting back to the actual topic, first of all, I assume you realize that the topic of discussion here is FreeMarker. I say that, because up until this point, every conversation we have had was about FreeCC. Though, actually, also, FreeCC and FreeMarker are kind of sister projects.

As for nobody using Ant, I can’t help but think that is an exaggeration, but I don’t disagree either really. The situation with FreeMarker, the whole history is really this kind of dark, sordid drama, and I’ll probably end up writing it up soon. It is an Apache project, Apache FreeMarker, but the individual (idiot…) who is the main guy who brought it to Apache after both Attila and I drifted away, only brought the 2.3.x version, which was a very old codebase that was largely completed in 2003 (or maybe 2004). And the idiot in question just left the trunk of the code there on Sourcefore with a note telling people not to look at it or to pretend it doesn’t exist or something. So, at this point, after 10 years, I’ve decided to dust it off and put out the release that should have happened over 10 years ago. And that was supposed to be FreeMarker 2.4 but since so much time has gone by, it will be FreeMarker 2.4.

So, yes, Ant was the big thing then, and it uses Ant and probably, moving forward, ought to move to something more modern… BUT… that is not a particularly high priority at this point in time either. I just want to take the path of least resistance at this point to getting a release out, so I may stick with Ant or even just do a stop-gap of rolling up some very bloody-minded simple build script to use as a stop-gap for this initial release, which will be FreeMarker 3, preview 1, I guess.

My initial idea was just to go with the JDK’s built-in logging, but from the other feedback from Attila, I guess this SFL4J is the way to go, if that’s what other people are using. This whole logging question is not something that really lights my fire either, but I just have to figure out what to do and just put out the release, so it’s good to have some input from people haven’t done a Rumplestiltskin of 10 years wrt the Java world.

One thing, I would say, coming back to all of this, and it was a tendency already quite notable a decade ago, is that Java development has really become something very baroque, hasn’t it? Java was a breath of fresh air, just a lot of fun back in the late nineties but now… well… I dunno, that’s a big conversation. LOL.

I may take you up on your offer to get an initial maven thingy going, but I have to figure out exactly what I’m going to do about all the libraries first obviously. And I could leave that for a Preview 2 anyway.

Thanks for the input. As I said to Philip, it really is important for me now not to be in this “shouting into the void” sort of thing. So, Servlet 4.0 and JSP 2.3 it is.

As for JDK level, I guess I’ll compile to a Java 8 target for now. But, actually, yes, I think going with 11 pretty soon would also make sense. The thing is that, with FreeCC, I have a high degree of confidence that the included Java Grammar (which is embedded in FreeCC, so you can put Java code actions in there) is quite solid through Java 8. Once I achieved that milestone, I decided to put my energies elsewhere. I don’t think getting up to Java 11 or even Java13 is that hard. I notice that there is a “var” keyword that currently FreeCC doesn’t “know” about. (Though actually, JavaCC doesn’t know about it either! FreeCC’s Java language support supersedes that of JavaCC at this point, and actually, it’s safe to say that, moving forward, this will ALWAYS be the case!)

As for the logging stuff, yesterday, I was grepping through the code and there is exactly ONE instance where I log. I log once at the DEBUG level when there is an attempt/recover block and the code inside the attempt block fail. I guess I just don’t log very much because I just figure that if the code is working, why log anything? And if the code is not working, well, then you throw an exception. Of course, whatever app or framework is using FreeMarker then can choose what to do with the exception – okay, presumably log it somewhere, but IMHO, working at the FreeMarker level, the core parser/rendering machinery, it’s not even my problem.

The funny thing about the extra language support stuff is that it actually is appealing. A priori, it was not obvious that there would be so little uptake from the Jython or JRuby or Rhino people for something like this. But… there is the question of whether people even knew about it. So, in a way, it’s a pity to jettison all that, but the fact remains that there was never a shred of proof that anybody was really using that stuff. I think there was one blog entry of some jython guy playing with it and being quite impressed, but there wasn’t any kind of real usage out there, like in production and stuff, nah.

The XML processing is a funny thing because I implemented all that when there was all this XML hype. And XSLT hype. And I think all that is kinda dead. There was this guy, Kirk Rasmussen (I’ve been looking back at all the ancient history) and Kirk worked for Lucasfilm and was using FreeMarker. Back in 2003 and he was on our mailing list. But Kirk was heavy into all this XML API stuff. But he’d talk about all these performance problems he was hitting, and I’m pretty sure that all of the bottlenecks were in the various XML libraries, XPath and all that. FreeMarker, you just give it a DOM tree and it recursively walks the thing and outputs stuff and really, it’s fast like greased lightning. It was all that over-engineered XML stuff that was his problem. He couldn’t do more than dozen transactions a second on God knows what powerful hardware for those days. (I assume that starwars.com does get some serious traffic…)

Ah, the good old days. But the XML stuff also does not have much usage. The thing is, though, that the tree traversing TemplateNodeModel stuff is useful outside XML and is useful in conjunction with FreeCC. There are some very interesting, powerful ideas, lots of low hanging fruit to pluck. So, obviously, if we keep the tree-related stuff, then we keep the XML related stuff, because… ya know…

I’m thinking I want a FreeMarker 3 release by the end of this month at the latest. So it’s a question of figuring these things out and polishing it up and putting it out.