Short Notes about Messaging Protocols, Apache Kafka and RabbitMQ.
Before choosing a messaging queue, there are some concepts, based on your requirements, that should be taken into consideration. Some of them are:
- Broker Scale: The number of messages sent per second in the system.
- Consumer Capability: Whether the broker is capable of managing one-to-one and/or one-to-many consumers.
- Data Persistency: The ability to recover messages.
- Subscription Types: Whether messages are lost or not if a consumer is down. These are:
Ephemeral subscription: When the consumer stops running, then the subscription and any unprocessed messages are lost.
Durable subscription: The subscription is maintained even if a consumer shuts down. When the consumer is running again, the message processing is resumed.
RabbitMQ
- A traditional general purpose message broker.
- Supports multiple messaging protocols like AMQP, MQTT, and STOMP.
AMQP: (Advanced Message Queuing Protocol) An open standard application layer protocol for message-oriented middleware
MQTT: A lightweight, publish-subscribe network protocol that transports messages between devices
STOMP: A simple (or streaming) text orientated messaging protocol - Delivers messages through both point-to-point and pub-sub methods.
Point-to-point: It is based on the concept of sending a message to a named destination.
Pub-sub: Any message published to a topic is immediately received by all of the subscribers to the topic. - Supports legacy applications (with plugins)
- Best at complex routing logic.
Scale: around 50K msg per second.
Persistency: both persistent (Kafka stores a persistent log which can be re-read and kept indefinitely) and transient messages.
Subscription Types: both durable and ephemeral.
Main Use Cases:
- Background jobs or long-running tasks (like file scanning, image scaling or PDF conversion, etc).
- As a middleman between microservices (for example, notifications for order handling, at each order update — order placed, update order status, send order, payment, etc.).
Architecture:
Components: producers, exchanges, queues, and consumers.
- A producer pushes messages to an exchange (direct, topic, etc.), which then routes messages to queues (or other exchanges).
- A consumer then continues to read messages from the queue, often up to a predetermined limit of messages.
Message Order: The queues are sequential and FIFO (first in, first out) with RabbitMQ. FIFO ordering is not guaranteed for priority and sharded queues.
Kafka
- Handle high throughput, low latency processing.
- Replicates a publish-subscribe service.
- Best at large amounts of data
Scale: can send up to a millions messages per second.
Persistency: both persistent and transient messages.
Subscription Types: both durable and ephemeral.
Main Use Cases:
- High-throughput activity tracking
- Stream processing (stream of events)
- Event sourcing
- Log aggregation (ELK stack)
Architecture:
Components: producers, consumers, clusters, brokers, topics, and partitions.
- Producers send records to clusters, which store those records and then pass them to consumers.
- Each server node in the cluster is a “broker,” which stores the data provided by the producer until it is read by the consumer.
Message Order: Kafka guarantees order within a partition (in a round robin fashion), but not across partitions in a topic. It is also possible to have producers add a key to a message — all messages with the same key will go to the same partition.
For more details about Apache Kafka’s architecture, you can read my other post:
References:
https://otonomo.io/redis-kafka-or-rabbitmq-which-microservices-message-broker-to-choose/
https://www.instaclustr.com/rabbitmq-vs-kafka/
https://dattell.com/data-architecture-blog/kafka-vs-rabbitmq-how-to-choose-an-open-source-message-broker/
https://stackoverflow.com/questions/42151544/when-to-use-rabbitmq-over-kafka
https://en.wikipedia.org/wiki/MQTT
https://stomp.github.io/
https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol
https://www.oreilly.com/library/view/java-message-service/9780596802264/ch04.html
https://aws.amazon.com/pub-sub-messaging/
https://www.confluent.io/blog/okay-store-data-apache-kafka/
https://www.rabbitmq.com/queues.html
Happy Coding!
