Programming & Compilers

How DMA cache coherence depends on buffer handoff and completion

Trace dirty CPU input, stale CPU output and shared-line hazards under a noncoherent device path, then separate synchronization from confirmed device completion.

By Republeq Editorial · 10 min read ·
DMA cache coherence illustrated through outgoing dirty CPU data, returning stale CPU data and shared-cache-line hazards, with ordered buffer ownership and observed device completion.

A CPU and a DMA-capable device can exchange a buffer without necessarily sharing the same automatically coherent view of it. New CPU data can remain in a dirty cache, while new device data can reach memory behind an older CPU copy.

DMA cache coherence therefore needs explicit system assumptions and buffer ownership. A platform-defined synchronization step, an ordered command and confirmed device completion establish different parts of the transfer.

The examples use hypothetical cached and backing values. They do not provide driver instructions, an IOMMU configuration or a storage-durability procedure.

Define device buffer ownership and coherence assumptions

Let CPU C and device D share buffer B. In the noncoherent model, D reads and writes the backing memory path without automatically obtaining current CPU-cache data or invalidating every stale CPU copy.

In a coherent model, the specified device path participates in the required coherence relationship. A device read can obtain current data through that relationship, including data held by a dirty owner under its protocol.

These are stipulated alternatives, not conclusions drawn from the device name or the word DMA. The platform must identify the coherence domain and access path relevant to B.

Ownership names who may access or modify B during each phase. CPU ownership permits the stated preparation work. Device ownership permits the specified operation while restricting conflicting CPU access under the contract.

The pipeline-backpressure guide supplies a retention-boundary comparison for device buffer ownership. Removing a payload from its queue does not release the backing object while a worker or another stage retains it. Likewise, handing B to D must retain the storage until the device-use obligation is settled. The pipeline account limits retained application objects; it supplies no device coherence, cache operation or DMA command-ordering rule.

The buffer must remain valid for the whole device-use interval. Handing over an address without the required lifetime and access conditions does not establish a complete ownership transfer.

Trace a device read behind dirty CPU data

Initially let backing memory contain value zero for B. While it owns the buffer, C prepares value one in its cache. The cached line is dirty, and backing memory still holds zero in this model.

D then reads through the noncoherent memory path without the required handoff synchronization. It can obtain zero even though C's local read sees one. The stipulated access paths explain those different values.

A correct transfer must make the prepared value available under D's access contract before the command depends on it. In this teaching model, that includes propagating the current dirty data to the backing location D will read.

The structured-concurrency guide supplies completion-boundary context for cache maintenance. Its parent distinguishes a cancellation request from observed child termination and required cleanup. That distinction helps keep a requested transition separate from finished work. It is a task-lifetime contract, not a device-cache specification: the required DMA synchronization and hardware completion evidence must still come from the actual platform and device operation.

The exact synchronization operation and ordering rules are platform-defined. The example states the data relationship that must hold, not a generic instruction sequence safe for every architecture.

If the ownership transfer and command are properly ordered under that contract, D's read uses prepared one. A CPU-side cache status alone is not evidence that the device actually consumed that value.

Trace a device write behind an older CPU copy

For the reverse direction, let C retain a cached copy of B containing one. Under the ownership contract, D becomes the permitted writer and writes value two to backing memory.

In the noncoherent model, that write does not automatically make C's cached one unusable. If C consumes the old copy without the required return synchronization, its local read can still produce one.

Confirmed device completion establishes that the specified device work finished under its completion contract. The CPU then needs the platform's required visibility and cache-state transition before consuming the returned buffer.

In our simplified return model, the stale CPU copy is made unusable and the later permitted CPU read obtains two. Invalidation and data acquisition occur under the stated platform contract.

The result should remain distinct from command submission. A request sent to D does not establish that D wrote two, and cache-state maintenance does not independently establish that D finished its operation.

Conflicting CPU modifications during D's ownership would also change the assumptions. The handoff needs the access restrictions that make the two agents' intended effects compatible.

Place synchronization around ordered transfer and completion

Use a complete CPU-to-device-to-CPU history. C prepares B while owning it, performs the platform-defined outgoing synchronization, and transfers the permitted use to D under the required command ordering.

D performs its stated work. The owner observes the relevant completion evidence rather than assuming work ended because the command was accepted. The return synchronization then establishes C's allowed view before C consumes B.

The handoff record should distinguish preparation, visibility, device admission, completed work and CPU consumption. Each event needs the buffer and operation identity it concerns, with the completion event tied to the device effect promised by that operation.

In a coherent device-access model, some explicit maintenance requirements can differ, but ownership and completion do not disappear. The system still needs to know when D may use B and when C may safely resume its own access.

The initial choice of coherent or noncoherent path must remain attached to the trace. Linux's Dynamic DMA mapping Guide provides a documented API example: coherent mappings do not remove the need for appropriate memory ordering, while a reused streaming region needs the specified CPU-side and device-side synchronization around the respective accesses. Its guide also warns about CPU-written fields sharing a cache line with device-written data on DMA-incoherent systems. Those are Linux API contracts; our value traces remain hypothetical and prescribe no architecture-independent maintenance sequence.

How cache maintenance granularity involves neighbouring bytes

Place B and unrelated CPU-owned value N within the same maintenance unit, a hypothetical cache line L. A whole-line operation affects both regions even though the application named only B as the device buffer.

Start this separate history with backing B=1 and N=10, and a matching CPU-cached line. D owns B and updates its backing value to two. C then changes N to eleven through the retained line under an incorrectly narrow ownership assumption.

The resulting dirty CPU line can contain B equal to old one and N equal to new eleven. If a later full-line writeback propagates that whole line, it can overwrite D's two with one while correctly carrying N's update.

N's application-level independence did not isolate its physical maintenance scope from B. The ownership and synchronization model must account for that shared unit.

Discarding the line without preserving a dirty N can create the converse problem: obtaining the device's B value while losing the CPU's required N change. The correct action cannot be inferred from B's byte range alone.

A design needs a platform-valid arrangement or ownership rule that covers the relevant granularity. The example does not prescribe an allocation alignment or a specific maintenance API; it identifies the shared bytes whose effects must be preserved.

Visibility, command ordering, and finished work are separate claims

Buffer visibility asks whether the consuming agent can obtain the intended bytes through its actual path. Command ordering asks whether the operation that permits D to act is ordered after the required preparation and publication.

Finished work asks whether D completed the promised operation and whether its completion evidence covers that effect. A command acknowledgment can have a narrower contract than a work-completion event.

A correct CPU cache maintenance event would not prove that D examined the buffer. Conversely, observing a device completion event would not prove that C discarded every stale noncoherent copy before reading the output.

The report needs the relationship between each synchronization, consuming view and completed operation. The synchronization must concern the right buffer, and the completion must concern the right device operation.

A handoff record can identify B, its covered maintenance unit, the command operation and the completion event consumed by the owner. On return, the CPU view must be associated with that completed use rather than another request to the same address. This makes value visibility and operation identity separately checkable without treating either record as a substitute for the other.

Buffer reuse also depends on those identities. Evidence about one earlier use cannot establish completion of a later operation merely because it targets the same address.

Retain ownership when an outcome is unresolved

If device completion is not established, the buffer's permitted reuse cannot be inferred from a timeout or an issued synchronization request. The contract must state who still owns the operation and what access remains admissible.

For example, a CPU that reuses B while D may still write it has introduced a conflicting-use history. Successfully invalidating an old cache copy would not resolve that uncertainty.

The recovery policy is platform- and operation-specific. This article does not supply a device-reset procedure or claim that an uncertain write is safe to repeat. It keeps the completion obligation separate from cache visibility.

The buffer lifetime retains the same outstanding-use obligation. An unresolved device user cannot be excluded from the retention model simply because a software request returned an error.

Verify the two agents' actual values at the handoff boundaries

Preserve the coherence assumptions, buffer and maintenance-unit identities, allowed owner at each stage, and the cached versus backing values. Inspect the outgoing and returning views under the defined synchronization contract.

The first counterexample distinguishes CPU-cache one from device-read zero. The return counterexample distinguishes device-written two from stale CPU one. A passing trace needs the intended consumer value in each direction.

For the shared-line case, retain B and N together so the check can detect either overwrite of the device result or loss of the neighbouring CPU change. Verifying only B's named range would hide the granularity defect.

Keep CPU-cached values and backing-memory values in separate columns, with buffer B and the device operation identified at each transition. The Lenovo IdeaPad 15.6-inch FHD touch laptop listing pairs 16GB DDR4 memory with a 15.6-inch FHD touchscreen and identifies 512GB SSD storage and a Core i5-1334U processor. The touchscreen can display the separate value columns, while the SSD supplies storage for the retained records. Establishing the handoff still requires the actual device path, platform contract and operation receipt.

Inspect B and neighbouring N together when reviewing a whole-line action, and attach completion status to the exact device-use interval before reusing B. A 14-inch FHD display and 16GB DDR4 memory are listed for the ASUS Vivobook 14 FHD laptop, whose processor is identified as the Core 5 120U. The display supplies a place to inspect the granularity and ownership table. Its processor and memory specifications do not identify the device's coherence path. The table needs the platform's actual maintenance unit and access rules, including the neighbouring bytes affected by the operation.

The outgoing counterexample leaves backing B at zero while C sees cached one. The return counterexample leaves C seeing one after D writes two. In the shared-line failure, writing back cached B=1 with N=11 overwrites D's B=2. A valid handoff must preserve the intended consumer values, the neighbouring required update and the exact operation's completion before admitting return use.

DMA cache coherence: ownership, visibility and finished work