#Message queue code#

If a message is not acknowledged for a given period of time, it will be redelivered. The goal is to create an acknowledgment mechanism for individual messages.

If you are familiar with how Amazon SQS works, the message queue described below follows a similar philosophy. Each additional component adds up to the overall complexity of your system. But, that’s still to be verified.įinally, convenience - if you already have a working Kafka cluster, and you just need a simple message queue (without the advanced routing capabilities, for example), it might not make sense to set up yet another cluster, and maintain it later. Secondly, although I didn’t do any benchmarks yet, and while the message redelivery tracking does add a significant amount of overhead, the queue still should be pretty fast, taking advantage of the very good performance of the underlying Apache Kafka topics. in RabbitMQ’s documentation, it is explicitly stated that the system doesn’t cope well with network partitions. Other systems don’t put so much emphasis on clustering and replication - e.g.

?īefore diving into technical details, if Apache Kafka is a publish-subscribe system, why would we want to implement a message queue on top of it, instead of using a 'proper mq', such as RabbitMQ or ActiveMq?įirst of all, Kafka has reliable and proven clustering and data replication mechanisms if you want to make sure your data is safe, and that each message is processed at-least-once - even in case of losing a number of servers - Kafka should be a good candidate as a base for your message processing system.

#Message queue code#

A proof-of-concept-quality code is available on GitHub, with both embedded and stand-alone examples which should make experimentation quite straightforward. In this post, I’d like to outline a possible implementation of a Kafka-based Message Queue. That works great for many scenarios, but every now and then there are situations when you’d like to have selective message acknowledgment, and more important, redelivery of individual unacknowledged messages. That’s because of the streaming nature of a topic: you can only signal Kafka that all messages up to a given point have been processed, so that in case of failure stream processing restarts from the last committed point. It is not possible to acknowledge individual messages.

However, there is one important limitation: you can only commit - or, in other words, acknowledge the processing of - all messages up to a given offset. By committing processed message offsets back to Kafka, it is relatively straightforward to implement guaranteed “at-least-once” processing. The central concept in Kafka is a topic, which can be replicated across a cluster providing safe data storage. Apache Kafka is a very popular publish/subscribe system, which can be used to reliably process a stream of data.