Data intensive applications are now the bread and butter of our everyday lives in Software Engineering. Modern applications are able to handle amounts of data we didn’t think possible 15 years ago. Even more with the rise of more powerful Data Streaming platforms and Machine Learning tools, it seems to be the one thing giving engineers a good competitive advantage.
These systems are big, complex, and require a good understanding of certain principles of Computer Science. Many books have been written about principles, design patterns, architectures etc and there’s a lot of information out there, however it can be overwhelming at times.
I recently found myself reading a great book (if not THE book) about Data Intensive Applications, from Martin Kleppmann (One of the original creators of Kafka) and was able to relate a lot of the content to my everyday work. However, this is a 600 page typical O’Reilly book, which can get really dense and technical at times, so I decided to write a little series that summarises my understanding of the principles, components and most relevant parts (to me).
If you find that this series sparks your curiosity, feel free to look for the O-Reilly edition of the book and give it a full read yourself.
I’ve divided the series into 3 sections. Each one corresponds to a topic I decided was relevant to me and wanted to talk about in a bit more detail. Each of these sections are articles of its own, so make sure you go back and forth between them, or just go to the section you think will be relevant to you:
Replication & Partitioning in Distributed Systems
Batch Processing vs Stream Processing
As you can imagine, this series is not comprehensive, and there’s lots more out there to figure out. I like to learn as I do, and fortunately my job offers me opportunities to apply patterns and principles, in order to better learn them. If you don’t have such a luxury, there are still great tools out there to put together nice thought experiments and see these principles in action.
Feel free to reach out if you wanna have a catch-up about any of it.
Happy designing!
