Preventing Data Loss During Industrial Network Outages

A claim that a gateway supports store and forward is not an acceptance criterion. A reliable design defines when data is sampled, where it is stored, how long the buffer lasts, what happens when it fills and how records are reconciled after the network returns. These rules must be tested with the real gateway and platform.

Separate sampling from reporting

Timestamp records when the process value is sampled, not when the server finally receives it. Synchronize the gateway clock and define behavior if time becomes invalid. Store the value, engineering unit, source, timestamp and quality status needed by the receiving system.

A five-second sample transmitted every minute is different from one sample per minute. Capacity calculations must use the stored record rate and record size, including protocol overhead and metadata.

Size retention from a credible outage

Calculate storage for the required outage duration plus margin. Include all tags, alarms and event bursts. Decide whether the buffer stops acquisition, overwrites the oldest records or prioritizes critical points when full. The selected behavior should be visible to operators.

Flash endurance also matters. Frequent writes and uncontrolled power loss can corrupt storage or shorten its life. Use hardware and file systems intended for the required duty and verify recovery after abrupt power interruption.

Preserve ordering and prevent duplicates

Assign a stable record or sequence identifier so the platform can acknowledge and deduplicate backfilled data. Reconnects may repeat the last unacknowledged batch; exactly-once behavior cannot be assumed merely because MQTT QoS is enabled.

Upload historical records in bounded batches while continuing to handle current data. Otherwise a long backlog can saturate the mobile link and delay live alarms. Define priority and bandwidth limits explicitly.

Represent stale and missing data correctly

The platform must distinguish a live zero from an old value, a communication failure and a sensor fault. Display source timestamps and quality flags, and define when trends, totals or alarms may use backfilled values. Safety and control decisions should not rely on stale cloud data.

For counters and totals, document how resets, rollover and duplicate packets are reconciled. For alarms, decide whether a historical event should notify immediately after recovery or be recorded without a misleading current alarm.

Run outage acceptance tests

Disconnect the WAN for a measured period, keep the process simulator changing and then restore service. Compare source records with the platform by count, timestamp, order and value. Repeat with a power interruption and with a buffer near capacity. Save the evidence as part of commissioning.

Repeat the test after a firmware or platform change when the data path is modified. Monitor storage health and backfill failures during operation so a nominally enabled buffer does not become an unobserved single point of failure.

Field checklist

• Define sampling timestamps, quality flags and clock synchronization.
• Calculate buffer capacity from tag count, record size and outage duration.
• Document overflow, acknowledgement and deduplication behavior.
• Limit backfill so live alarms and current data remain responsive.
• Test WAN loss, power loss, near-full storage and platform recovery.

Common questions

Does MQTT QoS 2 guarantee no duplicate business records?

No. Protocol delivery and application storage are different layers. Reconnects and database retries still require stable identifiers and idempotent processing.

How much buffer is enough?

Base it on the longest credible outage, stored record rate and operational risk, then add a documented margin. There is no universal number of days.

Should old alarms be sent after recovery?

That is an operational decision. Preserve the event, but distinguish historical alarms from current conditions to avoid unsafe responses.