Packet loss

What is packet loss?

Packet loss is the phenomenon in which data packets or Ethernet frames are lost or discarded during network communication before they reach their destination, expressed as a percentage of all packets sent. In an office network, one per cent loss shows up at most as a stuttering video call or a slower download. In OT and industrial automation the picture is different: cyclic I/O traffic over ProfiNET or IP tolerates only a few missed cycles, after which a PLC drives its outputs to a safe state and the machine stops. That makes packet loss one of the most important indicators of network health in OT convergence.

🧠 How does packet loss arise and how do you express it?

Network communication runs on packets that carry a source address, a destination address, a payload and control information. A packet is lost when it:

• Is discarded - a switch, router or firewall has no buffer space left or cannot process it fast enough.
• Arrives corrupted - every Ethernet frame ends with a CRC-32 check value (the Frame Check Sequence); a receiver silently drops a frame with a bad checksum. CRC errors therefore do cause loss for the higher layers.
• Never arrives - because of a broken cable, a dropped wireless link or a routing error.
• Arrives too late - strictly speaking not network loss, but for a cyclic application a frame that arrives after the monitoring time counts as missed.

The loss rate is (sent - received) / sent × 100. With 1% loss, 100 of every 10,000 packets sent are lost. Packet loss is one of a set of network properties that you should always assess together:

🌐 What causes packet loss?

The most common cause is network congestion: when traffic exceeds the available bandwidth of a port, the queues fill up and new packets are dropped (tail drop). Microbursts are treacherous: short peaks lasting milliseconds overflow the buffers, while the five-minute average utilisation in your monitoring looks harmlessly low.

On wireless networks the radio layer hides part of the loss: Wi-Fi acknowledges every unicast frame and repeats it when the acknowledgement is missing. The loss then surfaces as jitter and latency rather than as loss, so a high retry rate is an early warning sign. Incorrect QoS settings, such as a queue that is too small for the priority class of real-time traffic, are an often overlooked configuration cause.

🔄 How do industrial protocols respond to packet loss?

Not every protocol recovers lost packets. The key distinction is whether the transport is TCP (retransmission, so loss becomes delay) or UDP or raw Ethernet (no retransmission, so loss becomes a missed value).

Cyclic real-time protocols therefore do not recover lost data: the next cycle simply brings current values. The monitoring time decides how many missed cycles are acceptable. In PROFINET the watchdog is the product of the watchdog factor, the reduction ratio and the send clock, so a device with a 4 ms update time and a factor of 3 drops out after 12 ms without data.

TCP does recover loss, but at a cost in time. A lost segment is resent quickly after three duplicate acknowledgements (fast retransmit); if that fails, TCP waits for the retransmission timeout, which starts at 1 second under RFC 6298. Loss also throttles throughput. According to the well-known Mathis rule of thumb, TCP throughput is roughly MSS / RTT × 1.22 / √p. With 1,460-byte segments, a 20 ms round trip and 1% loss, that comes to about 7 Mbit/s, regardless of whether the link runs at 100 Mbit/s or 1 Gbit/s. For a historian export or cloud link across a WAN, that is immediately noticeable.

🏭 What are the consequences for control, SCADA and historians?

• Process control - lost process data leads to unstable control loops, incorrect measured values, PLC timeouts and wrong actuation. When an I/O connection drops, the whole machine or process unit often stops.
• Motion control - with synchronised axes, loss causes synchronisation and position errors, vibration and safety stops. That is why motion often runs on deterministic networks such as PROFINET IRT, EtherCAT or TSN.
• SCADA and HMI - delayed trends, missing HMI values (”####” or greyed-out fields), alarm delays and communication errors. An operator acting on stale data is a safety risk.
• Historian - missing data points, disrupted trends and unreliable analyses, for example for OEE or energy reporting.
• Safety communication - a safety protocol does not trust the network and goes to the safe state when telegrams are missing. That is safe, but it produces unexplained production stops (“nuisance trips”).

🧩 How does redundancy limit the effect of packet loss?

Redundancy prevents a cable break or failed switch from causing sustained loss. Packets are still lost during the switchover time, however, and that time must be shorter than the monitoring time of your I/O.

A practical lesson: an MRP ring with a 200 ms switchover does not protect a PROFINET device with a 12 ms watchdog. That device still drops out briefly when a cable breaks. Raise the watchdog factor as far as the process allows, choose a faster variant, or use PRP or HSR. See also ring redundancy.

🔐 How are cybersecurity and packet loss related?

Cybersecurity incidents regularly cause packet loss. A DDoS attack saturates links, an aggressive network scan overloads weak field devices, and a broadcast attack or a rogue device creates congestion. Conversely, security measures can introduce loss themselves: deep packet inspection and TLS inspection on an overloaded firewall, or an inline IDS that cannot process packets fast enough.

IEC 62443-3-3 therefore requires denial-of-service protection in system requirement SR 7.1 and resource management in SR 7.2, so that the system keeps performing essential functions under load. In practice that means segmentation and VLANs to keep broadcast domains small, storm control on switch ports, rate limiting towards the control network and priority for I/O traffic using QoS. Security Monitoring in line with IEC 62443 preferably works passively through a SPAN port or a network TAP, so that the monitoring itself causes no packet loss on the production network.

🔧 How do you track down packet loss step by step?

1. Establish the scale - run a long ping with many packets (Windows ping -n 1000 -l 1400 <address>, Linux ping -c 1000 -i 0.2 <address>). Test large packets too, to expose MTU problems. Note that routers and firewalls often rate-limit ICMP replies, which shows up as false loss.
2. Read the switch counters - through SNMP (IF-MIB: ifInErrors, ifInDiscards, ifOutDiscards) or the web interface of the managed switch. Rising CRC errors point to cable, connector or EMC; output discards point to congestion.
3. Localise per port - compare counters along the path from PLC to I/O device. The problem usually sits on a single link or port.
4. Analyse the traffic - in Wireshark, the filter tcp.analysis.retransmission shows retransmissions and tcp.analysis.lost_segment shows missed segments. With PROFINET you see gaps in the cycle counter of the RT frames. Prefer a TAP: an overloaded SPAN port drops frames itself.
5. Check the physical layer - certify cables, check the shield earthing and the distance from power cables.
6. Monitor continuously - collect counters in network monitoring with tools such as InfluxDB and Grafana, and forward anomalies to a SIEM or the OT team’s monitoring platform.

As rules of thumb for typical values:

For links to the cloud and the IIoT, real-time control stays local. Edge computing with local buffering (store-and-forward) and MQTT QoS 1 prevents data gaps when the WAN connection falters. MQTT Sparkplug B additionally uses sequence numbers to detect missed messages.

❓ Frequently asked questions

How much packet loss is acceptable in an OT network?

On a wired industrial network carrying cyclic I/O, the standard is 0% packet loss, because PROFINET and EtherNet/IP tolerate only a few missed cycles. Any rising error counter on a switch port is a reason to investigate the cable, the connector or the load. On WAN links to the cloud some loss is normal, and TCP and buffering absorb it.

What is the difference between packet loss, latency and jitter?

Packet loss is the share of packets that do not arrive, latency is the delay of a packet, and jitter is the variation in that delay. The three are linked: congestion first raises latency and jitter and, once buffers are full, leads to packet loss. In OT networks you should therefore always monitor them together.

Can I measure packet loss reliably with ping?

Ping gives a first indication of packet loss, but it is not conclusive. Routers and firewalls often give ICMP low priority or limit the number of replies, and ping does not measure the application’s own traffic. Switch counters via SNMP and a Wireshark capture through a TAP give a more reliable picture.

Why does a PROFINET device drop out on packet loss?

PROFINET RT does not repeat lost frames but monitors the cyclic data with a watchdog. If the device misses more frames than the watchdog factor allows (often 3 by default), it breaks the connection and drives its outputs to a safe state. With a 4 ms update time that happens after just 12 ms without data.

Can a firewall cause packet loss?

Yes, a firewall can cause packet loss when deep packet inspection, TLS inspection or logging overloads its CPU or buffers. In OT networks with real-time traffic you should therefore size firewalls for peak load and preferably keep cyclic I/O within a single zone. Passive security monitoring through a TAP or SPAN port adds no loss.

Does QoS help against packet loss?

QoS does not prevent packet loss caused by a faulty cable, but it does protect critical traffic during congestion. By giving I/O, safety and alarm traffic a higher priority than backups or video streams, the less important packets are discarded first when queues fill up. QoS only works when it is configured consistently on every switch along the path.

📌 In summary

Packet loss is the share of data packets lost in transit, and in OT networks just a few missed cycles lead to timeouts, substitute values and production stops. Causes include congestion, physical faults, configuration, wireless interference and overloaded security equipment. With clear watchdog settings, redundancy that switches faster than the I/O monitoring, QoS, segmentation and continuous monitoring of switch counters, you keep packet loss in industrial networks under control.