2022

An alternative approach to custom partition assignment strategy for Kafka consumers with Karafka

Recently, in BookingSync, we were performing a migration from Karafka 1.4 to 2.0, which we use for communication with Kafka. One of the great features available in version 1.4 was a custom partition assignment strategy for consumers. It was particularly useful for us as we’ve had some topics that had a way higher throughput than the other ones, so just a round-robin strategy with even distribution of topics for consumers was not a suitable choice as we needed dedicated consumers for these specific topics/partitions. Unfortunately, custom partition assignment strategy for consumers is no longer available in Karafka 2.0. Nevertheless, we managed to perform the migration and replaced the custom partition assignment strategy with a more straightforward and robust solution.

Read on →

2021