Time Series Database

What is a Time Series Database?

A time series database (TSDB) is a database optimised for storing, compressing and querying measurements that are each tied to a timestamp, such as sensor values, process parameters and machine states. Where an ordinary database mainly updates records, a TSDB almost only appends new values and retrieves them by time window. That makes it the storage layer beneath SCADA trends, historians, condition monitoring and predictive maintenance within industrial automation.

🧠 How does a time series database work?

Every data point consists of three parts:

• Timestamp - the moment of measurement, usually in milliseconds or nanoseconds and preferably in UTC
• Value(s) - the measurement itself, for example 72.3 °C, often accompanied by a quality status
• Labels or tags - descriptive attributes such as plant, line or sensor ID, which you use to filter and group series

Each unique combination of measurement and labels forms a series. The number of series is called the cardinality. Labels are indexed, values are not. So never put unique or highly variable data, such as order numbers or random IDs, in a label: every new value then creates a new series. High cardinality is one of the main causes of high memory use and slow queries.

The write pattern is append-only: new values are added at the end and old values are rarely changed. Queries almost always ask for a time window, such as “the average flow per minute over the last 24 hours”. The entire storage design is tuned to that.

🔧 How does a TSDB store time series so compactly?

A TSDB combines several techniques that a relational database lacks or offers only in limited form:

The best-known implementation of delta-of-delta and XOR is Gorilla, the in-memory TSDB that Facebook described at the VLDB conference in 2015. Gorilla reduced data points of 16 bytes to an average of 1.37 bytes, a factor of 12, at roughly 12 million new data points per second. Prometheus and many other TSDBs use variants of this technique and average 1 to 2 bytes per value.

A Historian often works differently: it uses deadband and swinging door compression to discard values that lie on a straight line within a tolerance. A TSDB keeps every value by default and compresses losslessly.

🔄 What is the difference between a TSDB, a historian and a relational database?

Technically, a historian is a TSDB with industrial collectors, tag management and process functions on top. In practice you choose a historian when you want ready-made connections to PLCs, RTUs and DCS systems. You choose an open TSDB when you want flexibility and can organise collection and contextualisation yourself.

📦 Which time series databases are used in OT?

For visualisation, an open TSDB is often combined with Grafana. Some TSDBs are part of broader SCADA or MES environments.

🏭 What is a TSDB used for in industry?

• Trending - comparing temperature, pressure and flow over hours, days or years
• Performance - calculating OEE, downtime and cycle times per line
• Maintenance - tracking vibration, current draw and running hours for condition monitoring
• Analytics - supplying training data for machine learning and anomaly detection
• Digital Twin and Edge Computing - a local TSDB buffers data at the edge and later synchronises with a central environment
• IT monitoring - watching servers, networks and applications with monitoring tools

Typical data sources are PLCs, RTUs, sensors, drives and HMIs, via OPC UA, MQTT, Sparkplug B or Modbus gateways. In IIoT architectures the TSDB often subscribes to a Unified Namespace, so every new value arrives without polling.

🧭 How do you set up a TSDB for OT data?

1. Design the data model - decide which attributes become labels (plant, line, unit) and which are values. Keep cardinality limited.
2. Capture time at the source - use the timestamp of the PLC or field device, not the moment of arrival, and synchronise clocks.
3. Choose the sampling rate - 1 second is common for process data; vibration analysis needs much higher rates.
4. Set retention per tier - for example raw data for 90 days and one-minute averages for five years.
5. Buffer during outages - let the collector store data locally when the connection drops (store and forward).
6. Plan availability and backup - with replication for high availability and a tested backup.

Worked example: 2,000 tags that each deliver one value per second produce 172.8 million values per day. At roughly 1.5 bytes per value after compression, that is about 260 MB per day, over 90 GB per year. If after 90 days you keep only one-minute values with average, minimum and maximum, 8.6 million values per day remain: twenty times fewer. Noisy signals compress less well, so build in a margin.

🔐 How do you secure a time series database?

• Authentication and access control - always enable authentication. Some open-source TSDBs, such as InfluxDB 1.x, start without it by default.
• Encryption - use TLS for write and read traffic, also inside the OT network.
• Placement - put the TSDB at level 3 of the Purdue Model or in the DMZ, and shield it with firewalls. Administration goes through a VPN or jump server.
• Integrity - protect data against tampering and loss with immutable backups. In the pharmaceutical industry, GxP applies, with requirements for data integrity and 21 CFR Part 11.
• Compliance - define retention and audit logs in line with compliance requirements. Organisations under NIS2 and the Dutch Cybersecurity Act must also include their OT data platforms in their risk management.

In OT environments it is crucial to guarantee both the integrity and the availability of process data.

❓ Frequently asked questions

What does TSDB stand for?

TSDB stands for time series database, a database built for measurements with a timestamp. A TSDB is optimised for many small writes and for queries over time windows, such as averages per minute or hour.

Is a historian the same as a time series database?

A historian is a specialised time series database for industrial process data. Besides storage, a historian provides built-in collectors for OPC, Modbus and DCS systems, tag management and process-oriented compression. A general-purpose TSDB provides the storage, but you organise data collection yourself.

Why is a TSDB faster than an SQL database for sensor data?

A time series database partitions data by time, stores the values of each series in columns and compresses timestamps and values with techniques such as delta-of-delta and XOR. As a result, a query over a time window reads far less data from disk than a row-based SQL database.

What is cardinality in a time series database?

Cardinality is the number of unique series in a time series database, that is, the number of combinations of measurement and labels. High cardinality arises when unique values such as order numbers are stored as labels. That uses a lot of memory and slows down both writes and reads.

How much storage does a TSDB need?

A time series database uses on average 1 to 2 bytes per value after compression. 2,000 tags with one value per second then cost about 260 MB per day. Downsampling and a retention policy limit growth further.

Which time series databases are open source?

InfluxDB 3 Core, TimescaleDB, QuestDB and Prometheus are well-known open-source time series databases. With TimescaleDB, some features such as compression and continuous aggregates fall under the separate Timescale License. Commercial editions often add extras such as high availability.

📌 In summary

A time series database (TSDB) stores timestamped measurements compactly and makes them quickly retrievable by time window. With time partitioning, columnar storage, compression, downsampling and retention policies, a TSDB forms the storage layer beneath historians, SCADA trends, monitoring and predictive maintenance. Secure it like any OT system: with authentication, TLS, segmentation and reliable backups.