Apache Kafka in ASP.NET 6 Core In Brief

Apache Kafka Using C#  & Asp.net 6 Core 

Introduction

the purpose of this article is to briefly illustrate what Kafka is and why we need to use it

also will gonna add a code snippet to cover the whole image quickly.

What is Kafka??

Kafka is the most popular distributed message broker or streaming data

- use publish-subscribe the stream messages or data

- store messages or data in a fault-tolerant way

- can persist data for a particular period

- the ability to grow easily with zero downtime

- capable of handling a vast number of read and write operations/second

Kafka characteristics:

- Scalability: can scale up easily by adding new brokers (nodes) in a cluster also add multiple partitions inside each topic to enable high scalability in reads and writes.

- High Throughput: handle the massive amount of incoming messages at a high velocity per second

- High performance: deliver messages at high speed and high volumes with low latency and high availability

- Durability: it saves data on the disk not in memory so it keeps it for a long time without modification.

Kafka differ from RabbitMQ in:

Kafka saves a message after it has been consumed so you can get it or consume it again. 

RabbitMQ deletes messages immediately after they have been consumed.

Kafka Taxonomy:

Kafka has :

• Cluster: It comprises one or more brokers (nodes). For high availability, each of them has its own partitions, each partition considered to be a folder that contained a set of offsets each offset considered as a key for the message stored in the partition.
• Broker: acts as a middleman between producers and consumers, hosting topics and partitions, enabling sending and receiving data between them
• Topic: a channel for the transmission of data
• ZooKeeper manages brokers in a cluster. It also notifies producers and consumers in a cluster if a new broker is added or failure occurred of existing brokers
• Producer: is responsible for writing, and publishing messages on those topics.  can connect to cluster through Zookeeper
• Consumer: consumes data through reading messages on the topics they've subscribed to, each Kafka consumer belongs to a particular consumer group, and Kafka sends messages to the consumers within the same  group from different partitions of a topic

Topic partitions:

- topic is divided into partitions and each one has multiple offsets for storing a lot of messages.

- we need to put in our mind that num of partitions depends on number of consumers and must be less than or equal consumers numbers .

- each consumer in same group can consume from one or more  partitions but cannot multiple consumer within same group consume from same partition .

- if number of consumer within same group was bigger than number of partitions in this case each consumer will be assigned to one partition and the other not assigned consumers will be idles until partition became free . 

- what if multiple consumers within same group need to consume from same partition in this case those consumers will be assigned to another brokers which holds same data through load  blancer .