What is Apache Kafka?
Apache Kafka is an open-source, distributed event streaming platform that stores events as a durable, replayable log and delivers them in real time from producing to consuming systems. Kafka was developed at LinkedIn by Jay Kreps, Neha Narkhede and Jun Rao, open-sourced in early 2011 and has been a top-level project of the Apache Software Foundation since October 2012, under the Apache License 2.0. In OT convergence, Kafka mainly serves as a central data bus between OT systems such as SCADA and historians and IT systems such as MES, ERP, cloud platforms and analytics, above the control layer.
🧠 How does Apache Kafka work?
Kafka is based on a distributed publish-subscribe model with an append-only log at its core. Producers write messages to a topic; consumers read them back at their own pace. Unlike a classic message broker, a message does not disappear once it has been read: it stays until the retention period expires.
| Component | Function |
|---|---|
| Producer | Publishes messages (events) to a topic |
| Broker | Server that receives messages, stores them on disk and serves them |
| Topic | Logical data stream, for example machine.telemetry |
| Partition | Ordered part of a topic; the unit of parallelism and replication |
| Offset | Sequence number of a message within a partition; the consumer tracks its position |
| Consumer group | Group of consumers that share the partitions of a topic between them |
| Cluster | Set of brokers, managed by the KRaft controllers |
Ordering is only guaranteed within a single partition. You therefore choose a key (for example the asset ID) so that all events from one machine land in the same partition and stay in order. Two standard components surround the core: Kafka Connect (since version 0.9) for integrating databases and other systems, and Kafka Streams (since version 0.10), a Java library for stream processing. Whereas a classic SCADA environment mostly polls data cyclically and keeps it only briefly, Kafka is event-driven: every state change, alarm or measurement is an event that several consumers can read independently of one another.
🔄 How does Kafka differ from MQTT and a historian?
In industrial architectures Kafka is often confused with MQTT or a historian. In practice they complement each other:
| Property | MQTT broker | Apache Kafka | Historian |
|---|---|---|---|
| Purpose | Lightweight telemetry from many devices | Event backbone and integration between systems | Long-term storage of process data for trends |
| Typical location | Edge and device level | Data centre, IDMZ or cloud | Site and enterprise level |
| Messages after reading | Gone (except retained message) | Kept according to retention | Kept and compressed |
| Replay of history | No | Yes, from any offset | Yes, via time-series queries |
| Typical clients | Many small devices, sometimes on unstable links | Back-end applications, connectors and stream processing | HMIs, reporting and analysis tools |
A common pattern is therefore: machines publish via MQTT or Sparkplug B to a broker in a Unified Namespace, a connector forwards the relevant topics to Kafka, and Kafka distributes them to analytics, time-series databases, data lakes and AI applications. Data from PLCs and sensors often arrives via OPC UA and an edge gateway.
🏭 Where does Kafka sit in the Purdue architecture?
Data originates at the lower levels of the Purdue Model, but Kafka itself does not belong there.
| Purdue level | Role around Kafka |
|---|---|
| Level 0-1 | Source of sensor and PLC data; no Kafka |
| Level 2 | SCADA events and alarms, read by an edge gateway |
| Level 3 | Local Kafka cluster or edge broker at site level, link to MES |
| Level 3.5 (IDMZ) | Replication or forwarding to IT, often one-way |
| Level 4-5 | Central Kafka cluster for enterprise analytics and ERP |
| Cloud | Managed Kafka services, data lakes and AI platforms |
Kafka is not suitable for hard real-time control, motion control or safety loops. Those still require EtherCAT, ProfiNET, Ethernet IP and fieldbuses. Typical end-to-end latency is in the range of milliseconds to tens of milliseconds and is not deterministic. Industrial real-time networks run fixed cycle times from tens of microseconds to a few milliseconds, with predictable latency.
⚙️ Which settings matter for reliability?
Kafka’s defaults are designed for an easy start, not for production. The Kafka 4.x configuration includes the following:
| Setting | Default | Production advice |
|---|---|---|
log.retention.hours |
168 (7 days) | Match to recovery and replay needs, for example 14-30 days |
log.retention.bytes |
-1 (no limit) | Limit per partition when disk space is constrained |
default.replication.factor |
1 | 3 brokers and replication factor 3 |
min.insync.replicas |
1 | 2, so every write lands on at least two brokers |
acks (producer) |
all |
Keep it; combine with idempotence (on by default) |
message.max.bytes |
about 1 MB | Do not send large files through Kafka; reference object storage instead |
listeners |
PLAINTEXT://:9092 |
Encrypted, authenticated listeners only |
Kafka can retain data for hours, days, weeks or indefinitely. With log compaction (cleanup.policy=compact) Kafka keeps only the latest value per key, which is useful for the current
state of every asset. Thanks to retention, consumers can re-read historical events for forensic investigation, monitoring, retraining models or predictive maintenance. For high availability, Kafka replicates every partition to several brokers; if the leader fails, an
in-sync replica takes over.
Kafka’s high throughput comes from sequential disk writes, batching of messages, compression (gzip, Snappy, LZ4 or zstd) and zero-copy transfer from disk to network. Actual performance depends on the number of partitions, storage speed, network capacity, the replication factor and the CPU load from compression. So measure with your own message sizes and volumes rather than relying on benchmark figures.
🕰️ What changed with Kafka 4.0?
Kafka 4.0 was released on 18 March 2025 and is the first version that runs entirely without Apache ZooKeeper. Metadata is now managed by Kafka’s own Raft protocol (KRaft), which means one fewer component to operate and secure. The new consumer rebalance protocol (KIP-848) also became generally available, and Queues for Kafka (KIP-932) arrived as early access. Since 4.0, brokers require Java 17 and clients Java 11. Existing clusters on ZooKeeper must first migrate to KRaft on a 3.x release before moving to 4.x. Version 4.4 followed in October 2026.
🔐 How do you secure Kafka in an OT environment?
A key point: security in Kafka is off by default. A non-secured cluster is allowed and the default listener is PLAINTEXT, with no encryption and no authentication. Anyone who can reach port 9092 can then read and write topics.
The main risks are unauthorised access, manipulation of process data, credential misuse, lateral movement, denial of service and supply chain risk from third-party connectors and plug-ins. Measures:
- Encryption - TLS on all listeners, including between brokers, with sound certificate management
- Authentication - mutual TLS or SASL with SCRAM-SHA-256/512, Kerberos (GSSAPI) or OAUTHBEARER; avoid SASL/PLAIN without TLS
- Authorisation - ACLs per topic and per consumer group; commercial distributions add RBAC on top
- Administrative access - MFA on management consoles, cloud portals and jump hosts
- Segmentation - place Kafka in a DMZ or IDMZ following the zones and conduits model of IEC 62443; let OT systems only publish outwards, or use a data diode for strict one-way traffic
- Patching - Kafka Connect, for example, was affected by CVE-2023-25194 (versions 2.3.0 to 3.3.2), where an attacker with rights to configure connectors could achieve code execution; fixed in 3.4.0
- Logging and detection - audit logs and broker metrics into a SIEM, with alerts on unusual clients
These measures support Defense in Depth and Zero Trust.
🧭 How do you deploy Kafka for OT data step by step?
- Start with the use case - for example OEE reporting, energy monitoring or condition monitoring, not with the technology.
- Design the topics - names following the ISA-95 hierarchy (site, area, line, asset), with the asset ID as key.
- Choose the ingestion route - OPC UA or MQTT to an edge gateway for edge computing; Kafka Connect or a bridge into Kafka.
- Place the cluster - three brokers in a dedicated zone above Level 3, on Kubernetes with an operator, or a managed cloud service such as Amazon MSK or Azure Event Hubs (Kafka endpoint from the Standard tier upwards).
- Secure before go-live - TLS, authentication and ACLs on, PLAINTEXT listeners off.
- Define data governance - schemas, ownership per topic and retention per data type.
- Test failures - shut down a broker and check that producers buffer and consumers carry on without data loss.
A simple topic design for a single plant might look like this:
| Source | Topic | Key | Consumer |
|---|---|---|---|
| PLC via edge gateway | site1.line2.telemetry |
asset ID | Time-series database, analytics |
| SCADA | site1.alarms |
alarm ID | SOC, monitoring |
| Energy meters | site1.energy |
meter ID | Energy dashboard |
| MES | site1.production.events |
order number | ERP, AI models |
For Dutch organisations that fall under the Cyberbeveiligingswet (the Dutch implementation of NIS2, in force since 15 August 2026), a Kafka cluster processing OT data is part of the network and information systems covered by the duty of care. Guidance such as NIST SP 800-82, ISO 27001 and the NIST CSF helps you organise operation and monitoring.
❓ Frequently asked questions
Is Apache Kafka suitable for real-time control?
No, Apache Kafka is not designed for deterministic, hard real-time control of machines or safety functions. Its latency is low but not guaranteed. Kafka therefore belongs above the control layer as an integration and analytics platform, while PLCs and fieldbuses handle control.
Does Apache Kafka still need ZooKeeper?
No, since Kafka 4.0 (March 2025) Apache Kafka runs exclusively in KRaft mode without ZooKeeper. Metadata is managed by controllers inside the Kafka cluster itself. Older clusters must migrate to KRaft before they can upgrade to version 4.
How long does Apache Kafka keep messages?
Apache Kafka keeps messages for 7 days (168 hours) by default, but retention can be set per topic to hours, weeks or indefinitely. With log compaction, Kafka keeps only the latest value per key. This lets consumers replay historical events for analysis or investigation.
What is the difference between Kafka and MQTT?
MQTT is a lightweight protocol for many small devices on sometimes unstable links, while Apache Kafka is a distributed log for large data streams between back-end systems. MQTT messages disappear after delivery, whereas Kafka retains them and supports replay. In industry the two are often combined: MQTT at the edge, Kafka as the backbone.
Is Apache Kafka secure by default?
No, Apache Kafka allows a PLAINTEXT listener on port 9092 by default, without encryption or authentication. For production you enable TLS, SASL or mTLS authentication and ACLs. In OT environments you also place Kafka in a separate zone such as the IDMZ.
Which cloud services offer Apache Kafka?
The major cloud providers offer managed versions of Apache Kafka, such as Amazon MSK (Managed Streaming for Apache Kafka), a managed Kafka service from Google Cloud and a Kafka-compatible endpoint in Azure Event Hubs. Confluent, among others, also provides a commercial Kafka platform. You no longer manage brokers yourself, but you remain responsible for access and data flows.
📌 In summary
Apache Kafka is a distributed, replayable event log that brings OT data to IT, cloud and analytics in a scalable and reliable way, above the control layer. Use Kafka as a backbone alongside MQTT and historians, place it in the IDMZ or higher, enable security explicitly and choose replication and retention deliberately. Kafka then becomes a solid building block for IIoT, digital twins and secure IT/OT integration with high scalability and clear segmentation.