Recently, in BookingSync, we were performing a migration from Karafka 1.4 to 2.0, which we use for communication with Kafka. One of the great features available in version 1.4 was a custom partition assignment strategy for consumers. It was particularly useful for us as we’ve had some topics that had a way higher throughput than the other ones, so just a round-robin strategy with even distribution of topics for consumers was not a suitable choice as we needed dedicated consumers for these specific topics/partitions. Unfortunately, custom partition assignment strategy for consumers is no longer available in Karafka 2.0. Nevertheless, we managed to perform the migration and replaced the custom partition assignment strategy with a more straightforward and robust solution.
Recently, I’ve had an unfortunate opportunity to deal with a very unexpected issue with Kafka that had quite terrible consequences. The exact origin of the issue is yet to be discovered; nevertheless, the process leading to the final solution to the problem and the fix itself were interesting enough that I decided to write a quick blog post about it as it might be potentially valuable to someone who also encounters a similar problem.
I’ve been recently planning to record a couple of tutorials for my team in BookingSync, focusing mostly on monitoring and some other more advanced aspects of Kafka and Karafka. But if I was already recording something, then why not make it available publicly so that more developers could benefit from it? So, in the end, I decided to make a bit more effort and recorded this mini-course, which will be available for everyone, for free ;).
In the first part of this series, we were exploring some potential options for communication between services - what their advantages and disadvantages are, why HTTP API is not necessarily the best possible choice and suggesting that asynchronous messaging might be a better solution, using, e.g. RabbitMQ and Kafka. Let’s focus this time entirely on the latter.
Microservices, Service-Oriented Architecture (SOA) and in general, distributed ecosystems, have been on hype in the last several years. And that’s for a good reason! At certain point, The Majestic Monolith “pattern” might start causing issues, both from the purely technical reasons like scalability, tight coupling of the code if you don’t follow Domain-Driven Design or some other practices improving modularity, maintenance overhead, and also from organizational perspective since working in smaller teams on smaller apps is more efficient than working with huge team on an even bigger monolith which suffers from tight coupling and low cohesion. However, this is only true if the overall architecture addresses the potential problems that are common in the micro/macro-services world. One of these problems I would like to focus on is communication between apps and how the data flows between them.