Streaming Data Must Be Fast and Count Correctly

An order is placed, but the inventory display still shows the old quantity. A sensor reports a change, but the alert arrives after the person who needed it has moved on. A dashboard updates quickly, then its total changes again. These are the situations that send people looking for “streaming data.” They want new information to be useful while a decision can still change, and they want to know whether the first number they see can be trusted.

Streaming data is a continuing flow of records that a system handles as they arrive, rather than waiting for a complete, fixed set. That sounds like a speed decision. In practice, the harder decision is what the system may say or do before it knows whether more relevant records are coming. Google Cloud describes streaming collections as continuously updated and explains why a key alone cannot close an aggregation: more events for that key may always arrive. Its streaming model uses windows, watermarks, and triggers to decide which events belong together and when a result can be emitted. Google Cloud’s streaming pipeline documentation sets out that distinction.

The useful question is therefore not “How fast is streaming?” It is “How late can this decision be, and what happens if the answer changes?” I would design the action around those two answers before choosing a processing mode. A count on a dashboard can be revised. A customer notification may need a stronger rule, because sending the same message twice cannot be undone by correcting a count.

Start with the decision that cannot wait#

Suppose a retailer wants to show stock changes during a busy sale. An order event, an inventory calculation, the displayed quantity, and an alert to staff are four different things. The calculation might be fresh while the display is stale. The display might be current while the staff alert is delayed. Calling the whole arrangement “real time” hides the point at which the business actually acts.

For each proposed stream, name the consuming decision. If an employee can act on an inventory change only after seeing an alert, measure from the relevant event to that alert, not merely to a pipeline step. If a dashboard is the product, specify how old a displayed count may be and whether a later correction is acceptable. The same incoming records can support both uses, but the two outputs need different rules. One can favor an early, revisable view; the other may wait for a more settled result or require a check before acting.

This is also the test for whether continuous processing is worth its cost. If the decision happens at a fixed daily meeting, updating a number every moment has little value by itself. If the opportunity to act disappears quickly, waiting for a scheduled report may defeat the purpose. Neither conclusion follows from the word “streaming.” It follows from the interval between an event and the last useful moment for a response.

There is a cost to the faster path. A system that shows provisional numbers must explain revisions; a system that fires an external action early must handle the chance that later information changes its context. I would accept that cost for a time-sensitive decision and make the status of the output explicit. For a decision that can wait, I would favor a later, steadier result over speed that nobody uses.

Separate when an event happened from when the system saw it#

Two clocks matter in a stream. Event time describes when the event occurred. Processing time describes when a stage of the system handled it. Google Cloud distinguishes these clocks in its explanation of streaming triggers. Its streaming pipeline guide also notes that data need not arrive in the order in which events occurred.

Consider an illustrative record of an order placed at 10:02, received by the pipeline at 10:05, and reflected on a display at 10:07. A report asking “How many orders were placed between 10:00 and 10:05?” needs the 10:02 event time. A question about how long the display took to update needs the later handling and display times. Substituting one clock for the other can make a delayed order look as though it occurred later, or make a slow display look healthy because the processing step ran quickly once the record arrived.

This distinction changes what “latency” should mean in a project brief. Record arrival is one milestone. Processing is another. The destination making a result visible is a third. The business action may be later still. A figure that stops at processing time cannot answer whether a person saw a useful alert in time. AWS describes Kinesis propagation delay as the interval from a record being written to the stream until a consumer reads it; that definition makes clear that even a named latency measure covers only a specified stretch of the route. AWS’s Kinesis latency guidance is useful when identifying where that stretch begins and ends.

I would write down the start and end of every timing measure before comparing systems. “Event to display” and “stream write to consumer read” are different promises. Neither should be presented as a guarantee for a full workflow. The pipeline and its destination determine how much delay a particular action actually experiences.

Decide what an early number means when records arrive late#

An ongoing flow cannot wait for every possible record before saying anything. Windows give the system a finite group to calculate: for example, events assigned to a defined period according to their timestamps. A trigger determines when the system emits an aggregate. A watermark represents a threshold after which a record for that window is considered late. These are related choices, not a single switch that makes a result both immediate and final. Google Cloud explains windows, watermarks, and triggers in its Dataflow documentation.

Imagine, purely as an illustration, a five-minute order count for 10:00 through 10:05. The display first shows 24 orders. An event timestamped 10:04 arrives after that count was shown. If the pipeline accepts the late record and emits a revised result, the display might show 25. If its late-data rule drops the record, the earlier displayed count may remain. The choice depends on what the number is for. A live operations screen may benefit from an early count that is visibly provisional. A later financial total calls for a rule that makes corrections and omissions understandable.

The mistake is to call the first emitted count “the final number” without saying how late records are treated. An early output is a view of the records available under the chosen timing rule. It is not proof that every relevant event has arrived. Google Cloud notes that late arrivals can change a window’s result after earlier output, and its explanation of exactly-once processing says that accurate handling of records already processed does not itself make a streaming aggregate complete. The distinction between accuracy and completeness appears in Dataflow’s exactly-once documentation.

The policy should be chosen where its consequences are visible. Ask how long a correction is useful, which destinations can replace an earlier value, and whether an earlier action can be recalled. If a dashboard can update a count, permit a clearly labeled revision when that serves the reader. If an alert would cause an irreversible action, use a rule that checks current state before acting or delay that particular output. The price of waiting is later action; the price of acting early is more work to explain and contain change.

Treat processing once and acting once as separate promises#

People often hear “exactly once” and infer that every downstream effect will happen once. That is too broad. Dataflow’s exactly-once mode concerns the processing results committed within its pipeline. Its documentation states that side effects, including writes to an external store, do not inherit that assurance unless the destination has suitable semantics. A transform can be retried even when the pipeline commits its result once. Google Cloud’s exactly-once explanation draws this boundary explicitly.

Return to the illustrative order. A pipeline might produce one committed record saying that stock fell below a threshold. If a step also calls a notification service, a retry can make that call again. The inventory calculation and the customer-facing effect are different outcomes. I would never rely on a processing-mode label as the only protection against a duplicate message, charge, or task. The receiving system needs a rule for repeated requests, such as recognizing a stable event identifier and treating a repeat as the same intended action. That recommendation follows from the documented boundary between pipeline results and external effects; its exact implementation depends on the destination.

Duplicate handling also has to match the meaning of the output. Suppressing a repeated event is one problem. Replacing an earlier window result after a late event is another. If the 10:00–10:05 count legitimately changes from 24 to 25, a destination that refuses every second write for that window would preserve the old number. A destination could instead identify the window and accept a newer version of its result. Conversely, an action such as “send this threshold alert” may need to recognize that a later retry is still the same alert. One universal deduplication rule will not serve both cases.

This is where I would spend design effort first when the output can affect a person or account. Define the identity of the event, the identity of an aggregate result, and the identity of the action. Then specify whether a repeated delivery should be ignored, replace an earlier value, or create a new action. Those decisions turn “correctness” into something the destination can enforce, rather than a hopeful description of the pipeline.

Choose the processing mode for the cost of a duplicate#

Streaming systems offer different compromises between speed, resource use, and duplicate handling. In Dataflow, exactly-once processing is the default streaming mode, while at-least-once mode permits duplicates within the pipeline and can reduce cost and latency for suitable workloads. Google Cloud recommends matching the mode to the workload rather than treating the faster mode as universally better. Its streaming-mode guide states these trade-offs.

For a count, sum, or other result whose meaning changes if one record contributes twice, I would start with exactly-once processing unless the complete design already removes duplicates at the relevant boundary. For a stream that translates each record and writes to a destination that already has an effective duplicate rule, at-least-once processing may be a sound choice. The potential saving comes with a responsibility: the duplicate rule must survive retries and must apply to the output people actually use. Moving that responsibility downstream does not make it disappear.

The decision should include the side effect either way. Exactly-once pipeline processing cannot by itself promise one external notification, and at-least-once processing does not require duplicate business actions if the destination can recognize repeats. I would therefore describe the mode and the destination rule together. “Exactly-once pipeline result, with destination deduplication for alerts” is a meaningful design statement. “Exactly once everywhere” is not.

There is no defensible universal latency target here. A faster internal mode may not shorten a slow destination step, while extra speed can be wasted if an output deliberately waits for late data. The useful comparison is the time to the business action, the quality of the result at that time, and the operational cost of making both acceptable. Google Cloud’s mode descriptions are choices within one platform, not service guarantees for every pipeline or destination.

Measure whether people receive usable information in time#

A running job is only the start of an operating picture. AWS Glue publishes measures for its streaming jobs, including metrics intended to show job health and performance. AWS’s Glue streaming monitoring documentation describes that operational view. It is valuable when a job slows or stops, but a healthy job metric does not answer whether the inventory alert appeared before staff needed it.

I would keep a second view tied to the decision: the age of the newest relevant event when the output became visible, how often an early result was revised, and whether the intended action occurred once. These are proposed measures for the workflow, not claims that a particular service publishes them. They tell different stories. A short processing delay with an old event means the data was already stale on arrival. A prompt calculation with a late dashboard update points to the destination. A timely alert sent twice points to effect handling rather than timing.

Use the same boundaries when setting an expectation with readers of a dashboard or alert. State whether a number is provisional, when it can change, and how old the information may be at the moment someone acts. If those answers differ for a dashboard and a notification, give each its own expectation. One vague “real-time” label cannot explain both.

Build the stream around a clear action contract#

Before implementation, write a short statement for the specific output: which event starts the clock, what the destination must show or do, the latest useful moment, the rule for late records, and the rule for repeats. In the illustrative stock case, that might mean that the display can revise a five-minute count while the staff alert represents one identified threshold event. The times and rules are design choices in the illustration, not performance claims about a service.

Then trace one event through the whole route. Give it an event time and an identity. Decide which window, if any, uses that time. Decide whether the first output is provisional and how a later correction reaches the destination. Finally, ask what a retry would do at the destination. If the answers are ambiguous on paper, faster processing will only deliver ambiguous results sooner.

Streaming data earns its place when a decision benefits from information arriving continuously and the system makes timing, revisions, and repeated effects explicit. My preference is to deliver an early, revisable view where people need awareness, and to require a firm action rule where the output changes something outside the pipeline. That combination respects both reasons the reader searched in the first place: the information must arrive while it matters, and it must remain trustworthy when someone uses it.