Stream Processing in Your Native Tongue

Editors. Tabs vs. spaces. Programming languages. Some hills are worth dying on.

I’ve often wrestled with the last one when it comes to stream processing. The basic situation is that people always want to use their language of choice. But stream processors, like Kafka Streams, Apache Flink®, and Apache Spark™, are largely geared to work best in exactly one programming language.

That has tended to leave people with two options: (1) use SQL (like FlinkSQL/KSQL) as a neutral means of expression embedded in your preferred language, or (2) use something a bit hacky, like inter-process communication, to sync data across language runtimes. Both can work, but neither are great general solutions.

Could GraalVM be the answer to this problem? Graal is a JVM replacement from Oracle in its early days. It has a lot of exciting improvements, but the most important for this discussion is the Truffle framework. Truffle is a library for building programming language implementations as interpreters for abstract syntax trees. Put simply, it allows languages to efficiently talk to one another without resorting to hacky workarounds.

I think this is a big deal for three reasons.

First, you’d be able to write stream processing programs in any language that has a Truffle implementation. You wouldn’t have to make trade-offs based on what the processing framework wants or needs.

Second, all of the tooling that you need to profile, debug, and monitor stream processing programs and state can be written once in a language-agnostic way. Each language won’t need to reinvent the wheel with local tooling.

Last, effort towards improving performance can be concentrated in one place. Today, framework maintainers spent time speeding up odds and ends across a large, multiple run-time project. But because Graal acts as a singular just-in-time compiler, that work can be reaped across whatever language you’re using. It’s worth noting this is a general benefit of Graavl, but a great example of it in action.

I for one am looking forward to a future where I can grumpily stay in my language of choice.