Posts

Event stream processing unplugged - part 1

-
Image
Introduction Welcome to event streaming unplugged, this is the first in a series of articles covering the topic of realtime evemt stream processing. This is a practical programming series with resources at unplugged part 1 . What you will learn in these articles: The basics of unbounded event processing Functional reactive programming introducing concepts like map, filtering, groupBy, flatMap Constructing directd acyclic execution graphs Managing event streams of heterogenous types Imperative event programming, integrating application code Input and output Building, testing and deploying Auditing, logging and debugging event execution flow This article starts with a simple hello world and then moves onto a more complex example. The second example calculates the mark to market value for a set of traded currency pairs. Two independent event streams, instrument market price and executed trades are processed. Batches of trades and reset signals are also supported. A more complex example se
Post a Comment
Read more

Serialize your lambdas for a rainy day - save to file

-
Introduction A short post describing how a Java lambda can be persisted to a file for re-use in a different process. I then discuss how we use this technique in Fluxtion. Serialising Lambdas Lambdas, introduced in Java 8 make functions first class citizens(nearly) in the Java language. They remove the need for a dedicated class to hold a function. But how does this work under the covers? In reality javac hijacks the class containing the function adding a static method that contains the implementation of the lambda. The lambda call site is replaced with an invocation to the newly added static method. For a full description of the lambda implementation magic see this article . Oracle helpfully provides the SerializableLambda class that implements the serializable form of a lambda, providing enough meta information to reconstruct the call. All we have to do is cast the lambda to a Serializable and then use the standard java machinery to marshal the lambda. Below are a couple of
Post a Comment
Read more

Easy event processing with var, Lombok and Fluxtion

-
Image
Introduction In this article I am combining two products Lombok and Fluxtion to demonstrate how tools can reduce both the code written and time to delivery while improving the readability of the code. The use of var from java 10 improves the situation even further. Both products and var use inference at build time to accelerate development. Fluxtion's ethos is to minimise waste, our goal here is to remove boiler plate code, reduce code-noise, and simplify integration tasks. We want to expends as little development time as possible while still delivering an efficient and high performance solution capable of processing millions of messages per second. Using the techniques described I compare a Fluxtion/Lombok implementation to a scala example using Akka streams, the Java version requires less code and is simpler to build. Housekeeping, apologies for not acknowledging Richard Warburton of  Opsian , in my first blog . Code Signal to Noise ratio When we code we address tw
Post a Comment
Read more

Waste free coding

-
Image
This article documents solving a meaningful event processing problem in a highly efficient manner through the reduction of waste in the software stack. Java is often seen as a memory hog that cannot operate efficiently in low memory environments. The aim is to demonstrate what many think is impossible, that a meaningful java program can operate in almost no memory. The example processes  2.2 million csv records per second in a 3MB heap with zero gc on a single thread in Java . You will learn where the main areas of waste exist in a java application and the patterns that can be employed to reduce them. The concept of zero cost abstraction is introduced,  and that many  optimisations can be automated at compile time through code generation. A maven plugin simplifies the developer workflow. Our goal is not high performance, that comes as a by-product of maximising efficiency. The solution employs Fluxtion which uses a fraction of the resources compared with existing java event proce
6 comments
Read more