About Kafka

Apache Kafka has proven to be an extremely popular event streaming platform, as its scalable distributed architecture, high performance, and use cases, some key terms and concepts as bellow:

  • Kafka clusters and Kafka brokers

  • Kafka clients and servers

  • Producers, and Consumers, and Consumer groups

  • Kafka topics & Kafka partitions, offsets

  • Kafka topic replication, leaders, and followers

  • ZooKeeper or not

Typical Kafka Topology:

image tooltip here

  • Helm install bitnami/kafka

Here we use helm to install Kafka, then validate statefuleset storage and cluster avalibility by create topic, producer and consumer

helm repo add bitnami https://charts.bitnami.com/bitnami
helm pull bitnami/kafka
kubectl create ns kafka
helm install zz-kafka . -n kafka
  • Kafka-cluster status
kubectl get all | grep zz-kaf
pod/zz-kafka-client                                          1/1     Running   0               6m27s
pod/zz-kafka-controller-0                                    1/1     Running   0               8m52s
pod/zz-kafka-controller-1                                    1/1     Running   0               8m52s
pod/zz-kafka-controller-2                                    1/1     Running   0               8m52s
service/zz-kafka                                  ClusterIP   10.96.58.62     <none>        9092/TCP                        8m52s
service/zz-kafka-controller-headless              ClusterIP   None            <none>        9094/TCP,9092/TCP,9093/TCP      8m52s
statefulset.apps/zz-kafka-controller                                    3/3     8m52s

kubectl get pvc
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
data-zz-kafka-controller-0   Bound    pvc-f3831a5c-c9cf-46bb-a47d-58ea80f82e28   8Gi        RWO            rook-ceph-block   9m19s
data-zz-kafka-controller-1   Bound    pvc-a45fd063-7979-4c0d-8ae4-93b4b4b0bf7f   8Gi        RWO            rook-ceph-block   9m19s
data-zz-kafka-controller-2   Bound    pvc-ed48b71d-81be-4067-b05f-9e8dc64fab04   8Gi        RWO            rook-ceph-block   9m19s
  • Validate cluster by set key create client.properties with SASL authentication details, copy to client
kubectl run zz-kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.6.1-debian-11-r0 --namespace default --command -- sleep infinity
kubectl cp --namespace default client.properties zz-kafka-client:/tmp/client.properties

open 2 bash window to access kafka client

kubectl exec --tty -i zz-kafka-client --namespace default -- bash

window1:
    PRODUCER:
        kafka-console-producer.sh \
            --producer.config /tmp/client.properties \
            --broker-list zz-kafka-controller-0.zz-kafka-controller-headless.default.svc.cluster.local:9092,zz-kafka-controller-1.zz-kafka-controller-headless.default.svc.cluster.local:9092,zz-kafka-controller-2.zz-kafka-controller-headless.default.svc.cluster.local:9092 \
            --topic test

window2:
    CONSUMER:
        kafka-console-consumer.sh \
            --consumer.config /tmp/client.properties \
            --bootstrap-server zz-kafka.default.svc.cluster.local:9092 \
            --topic test \
            --from-beginning

test topic and PRODUCER with CONSUMER

image tooltip here