Apache Kafka with Node.js: Complete Setup Guide (2026)

What Kafka Actually Solves

The Core Concepts, One Example Throughout

• Producer is any code that sends events. When a customer checks out, your order service is the producer.
• Topic is the named log events get written to, for example order-placed. Producers write to it, consumers read from it, and it is the only name either side needs to agree on.
• Consumer is code that reads from a topic. Your email service and your inventory service are both consumers of order-placed.
• Partition is how one topic is split so it can be read in parallel. A topic with 3 partitions can be processed by 3 workers at once.
• Offset is a consumer's bookmark: the position it has read up to in a given partition. Kafka stores this per consumer group, which is how a restarted consumer resumes instead of starting over.
• Broker is one Kafka server. A cluster is several brokers sharing the work and replicating each other's data.
• Consumer group is a set of consumers that share a topic's partitions between them, so each event is handled once by the group rather than once per instance.

What You Would Actually Use Kafka For

Running Kafka Locally With Docker and Node.js

1. 1

   Start a single-broker Kafka in KRaft mode

   Create a docker-compose.yml in an empty folder. This runs one container acting as both broker and controller, which is exactly what you want locally and never what you want in production.

   yaml — docker-compose.ymlCopy

   services:
     kafka:
       image: apache/kafka:latest
       container_name: kafka
       ports:
         - "9092:9092"
       environment:
         KAFKA_NODE_ID: 1
         KAFKA_PROCESS_ROLES: broker,controller
         KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
         KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
         KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
         KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
         KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
         KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
         KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
         KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

   Bring it up in the background:

   bashCopy

   docker compose up -d

2. 2

   Create the topic with more than one partition

   Kafka can auto-create topics, but it gives them one partition, which caps your consumer group at one active instance forever. Create it explicitly instead:

   bashCopy

   docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
     --create \
     --topic order-placed \
     --bootstrap-server localhost:9092 \
     --partitions 3 \
     --replication-factor 1

   Confirm it exists and check what Kafka actually gave you:

   bashCopy

   docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
     --describe --topic order-placed \
     --bootstrap-server localhost:9092

   You should see three lines, one per partition, each listing a leader and its replicas. If that command returns cleanly, Kafka is up and your local setup is done.

3. 3

   Install KafkaJS

   There are three Node.js clients worth knowing about, and the choice is easier than it looks.

   • `kafkajs` is pure JavaScript with no native build step, actively maintained, and has the best documentation. This is the default choice.
   • `node-rdkafka` wraps the C library librdkafka. It is faster under heavy load but needs native compilation, which turns Docker builds and CI into a chore.
   • `kafka-node` is the original client and has been effectively unmaintained for years. Do not start a new project with it.

   bashCopy

   npm init -y
   npm install kafkajs

4. 4

   Write the producer

   The important detail here is the key. Passing the order ID guarantees every event for that order lands in the same partition and is therefore processed in order.

   javascript — producer.jsCopy

   const { Kafka } = require("kafkajs");
   
   const kafka = new Kafka({
     clientId: "order-service",
     brokers: ["localhost:9092"],
   });
   
   const producer = kafka.producer();
   
   async function main() {
     await producer.connect();
   
     const order = { id: "1001", item: "Mechanical Keyboard", total: 89.99 };
   
     await producer.send({
       topic: "order-placed",
       messages: [{ key: order.id, value: JSON.stringify(order) }],
     });
   
     console.log("Published order", order.id);
     await producer.disconnect();
   }
   
   main().catch(async (err) => {
     console.error("Producer failed:", err);
     await producer.disconnect();
     process.exit(1);
   });

5. 5

   Write the consumer

   The groupId is the most consequential line in this file. It is how Kafka tracks your offsets, so it must be stable across restarts and deploys. Generating it at runtime means every restart is treated as a brand new consumer.

   javascript — consumer.jsCopy

   const { Kafka } = require("kafkajs");
   
   const kafka = new Kafka({
     clientId: "email-service",
     brokers: ["localhost:9092"],
   });
   
   const consumer = kafka.consumer({ groupId: "email-service-group" });
   
   async function main() {
     await consumer.connect();
     await consumer.subscribe({ topic: "order-placed", fromBeginning: true });
   
     await consumer.run({
       eachMessage: async ({ topic, partition, message }) => {
         try {
           const order = JSON.parse(message.value.toString());
           console.log(
             `[p${partition} @${message.offset}] confirmation email for order ${order.id}`
           );
         } catch (err) {
           // A malformed record must not stall the partition forever.
           console.error("Skipping unprocessable message:", err.message);
         }
       },
     });
   }
   
   main().catch(console.error);
   
   for (const signal of ["SIGINT", "SIGTERM"]) {
     process.on(signal, async () => {
       await consumer.disconnect();
       process.exit(0);
     });
   }

6. 6

   Run both and watch the message flow

   Start the consumer first so it is subscribed and assigned partitions before anything is published. In a second terminal, run the producer.

   bashCopy

   # terminal 1
   node consumer.js
   
   # terminal 2
   node producer.js

   The consumer logs the order within a few milliseconds of the producer publishing it, along with the partition and offset it came from. Those two numbers are worth logging in real systems too: when something goes wrong, the partition and offset are how you find the exact record.

   

Making the Producer Actually Reliable

const producer = kafka.producer({
  // Refuse to report success until every in-sync replica has the record.
  idempotent: true,
  maxInFlightRequests: 5,
  retry: { retries: 8, initialRetryTime: 300 },
});

await producer.send({
  topic: "order-placed",
  acks: -1,
  messages: [{ key: order.id, value: JSON.stringify(order) }],
});

• `acks: -1` (equivalent to acks: 'all') makes the broker wait until every in-sync replica has written the record before acknowledging. With the default of acks: 1, only the partition leader has to confirm, so a leader crash immediately after a write loses the message silently.
• `idempotent: true` attaches a producer ID and sequence number to every record so the broker can discard duplicates. Without it, a retry after a timeout that actually succeeded writes the same order twice.
• `retry` controls how long KafkaJS keeps trying through a broker restart or leader election. The default of 5 attempts is usually short for a rolling cluster upgrade.

Consumer Groups and Scaling, In Practice

Monolith or Microservices? Where Kafka Fits

• Two or more teams are blocked on each other's release schedule
• One module needs to scale independently and measurably differently from the rest
• You already have centralised logging with correlation IDs across processes
• You have someone who can own Kafka in production, including partition sizing and consumer lag alerts
• Your deploy pipeline can roll out services independently without manual coordination
• Event schemas are versioned, with a plan for consumers running older versions

Common Mistakes That Cost People Messages

• Leaving `acks` at the default. The producer reports success, the leader dies, the record is gone. Set acks: -1 on anything that matters.
• Creating topics with one partition. Auto-creation gives you one, and it permanently caps that consumer group at a single working instance.
• Generating the `groupId` at runtime. Something like email-service-${Date.now()} means every restart is a new group with no stored offsets, so it reprocesses the entire topic or skips everything, depending on fromBeginning.
• Publishing ordered events without a key. Related events scatter across partitions and get processed out of sequence. Key by the entity ID.
• Letting `eachMessage` throw. An unhandled rejection stops the partition, and a genuinely unprocessable record blocks every message behind it forever. Catch, log, and route to a dead letter topic.
• Doing slow work synchronously inside `eachMessage`. Kafka rebalances the group if you exceed maxPollInterval, so a long database call can trigger an endless rebalance loop.
• Connecting and disconnecting the producer per request. Each cycle is a fresh connection and metadata fetch. Create it once at startup.
• Assuming Kafka is a task queue. There is no per-message ack, no delayed delivery, and no priority. Those are job queue features, not Kafka features.

import { Kafka, type EachMessagePayload } from "kafkajs";

interface OrderPlaced {
  id: string;
  item: string;
  total: number;
}

const kafka = new Kafka({ clientId: "email-service", brokers: ["localhost:9092"] });
const consumer = kafka.consumer({ groupId: "email-service-group" });

await consumer.run({
  eachMessage: async ({ message }: EachMessagePayload) => {
    if (!message.value) return;
    const order = JSON.parse(message.value.toString()) as OrderPlaced;
    console.log(order.id);
  },
});

docker exec -it kafka /opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name order-placed \
  --alter --add-config retention.ms=2592000000