Memory Consistency

In multi-threaded programming, concurrent correctness is one of the harder aspects to achieve. There are multiple contributing factors, which include the following:

• Compiler optimizations may reorder and recombine memory accesses to improve performance, and in some cases, memory accesses may also be duplicated.
• Hardware memory subsystems may reorder and recombine memory accesses to improve performance.
• Hardware may implement out-of-order execution.
• Concurrent threads or thread groups may execute at different rates.

In general, the appearance of program order is maintained for a single thread. That is, given inputs, a single-threaded program computes the outputs as if everything were executed in the source program order. This is sometimes referred to as the “as-if” rule. As long as the program contains no undefined behavior, its output will be computed according to the program.

The behavior of a multi-threaded program is undefined in the presence of data races. Broadly speaking, a data race results from two threads accessing the same memory location, where at least one access is a write, using non-atomic accesses without established happens-before relationships between the accesses.

⚠️ Warning: slangc currently emits incorrect code for Atomic<T> for multiple targets. See GitHub issue #10683.

📝 Remark 1: When feasible, the best way to avoid concurrency issues is to design the program such that different threads never write to memory locations that other threads may read or write concurrently. Such designs are often desirable for performance reasons, too, since concurrent read/write accesses to shared memory locations can become a bottleneck for scaling.

📝 Remark 2: For more formal definitions of memory consistency, refer to the Vulkan Memory Model, the C++23 standard (ISO/IEC 14882:2024), or Multi-threaded executions and data races in the C++ reference.

Data Race

Two memory accesses conflict if they access overlapping memory locations and at least one is a write. A memory access conflict is a data race unless:

• the memory accesses are executed by the same thread, or
• the memory accesses are atomic, or
• one memory access happens before the other.

The happens-before relation is established by an atomic load-acquire observing an atomic store-release, memory barriers, or higher-level constructs provided by the Slang Standard Library.

Memory Order

The following memory orders are defined for operations in the current thread:

• Relaxed memory order — no constraints are imposed on other memory accesses.
• Acquire memory order — loads and stores after a load-acquire operation cannot be reordered before the operation.
• Release memory order — loads or stores before a store-release operation cannot be reordered after the operation.
• Acquire-Release memory order — memory accesses cannot cross an acquire-release operation.
• Sequentially-consistent memory order — acquire-release memory order with a sequential total order across all sequentially consistent operations.

Consequences:

Program sequence	Memory order of A	Consequences
In program order: Memory access M0 Operation A Memory access M1	Relaxed	None
	Acquire	A happens before M1
	Release	M0 happens before A
	Acquire-Release	M0 happens before A A happens before M1
	Sequentially-consistent	M0 happens before A A happens before M1 A is totally ordered with regard to other operations using sequentially-consistent memory order

Atomic Memory Access

All atomic modifications of a single variable occur in a total order. That is, the modifications are serialized. For example, if one thread increments a 0-initialized atomic variable by 1 and another thread increments it by 2, all threads will observe the increments in the same order: either 0 → 1 → 3 or 0 → 2 → 3.

The relaxed memory ordering with atomic memory access does not provide any ordering relationship with other memory accesses. All that is guaranteed is that the modifications are atomic and the modifications occur in a total order specific to the variable.

The release-acquire ordering with atomic memory access provides an ordering relationship with other memory accesses as follows:

• All memory loads and stores by a thread, in the program order, happen before an atomic store-release to a memory location.
• An atomic load-acquire from a memory location by a thread happens before all subsequent memory loads and stores in the program order.

Therefore, if thread A performs an atomic store-release X to memory location M, and thread B performs an atomic load-acquire from M and observes the modification X, then thread B will also observe all side effects by thread A prior to modification X.

The sequentially-consistent ordering implies release-acquire ordering with the addition that all atomic memory accesses using the sequentially-consistent ordering occur in a total order.

The Slang standard library provides atomic memory access primitives as follows:

• Relaxed atomic operations
• Atomic<T> type

📝 Remark: It is advisable to use only relaxed-memory-order atomics for Slang code that is intended for multiple targets. Most targets do not have native support for other memory ordering semantics.

Memory Barriers

A memory barrier imposes reordering constraints for memory accesses. Memory accesses before and after a barrier cannot be reordered across the barrier, matching the acquire-release memory order for the barrier. That is, memory accesses before the barrier happen before memory accesses after the barrier.

There are three address space scopes for a memory barrier:

• All — applies to all memory accesses. That is, device and thread group memory accesses.
• Device — applies to address spaces with all threads instance scope. This includes all storage buffers and images.
• Thread group — applies to thread group memory accesses.

The following memory barrier primitives are provided by the Slang standard library:

• AllMemoryBarrier()
• AllMemoryBarrierWithGroupSync()
• AllMemoryBarrierWithWaveSync() (TODO: link)
• DeviceMemoryBarrier()
• DeviceMemoryBarrierWithGroupSync()
• GroupMemoryBarrier()
• GroupMemoryBarrierWithGroupSync()
• GroupMemoryBarrierWithWaveSync() (TODO: link)

General Programming Guidance

Minimize the Number of Atomic Memory Accesses

Atomic memory accesses by multiple threads in a warp can be expensive. When feasible, a single thread should be elected to perform the atomic memory access.

For example:

RWStructuredBuffer<uint> outputBuffer;
RWStructuredBuffer<Atomic<uint>> syncBuffer;

[numthreads(64,1,1)]
void computeMain(uint3 dispatchThreadID: SV_DispatchThreadID)
{
    // write some output...
    outputBuffer[dispatchThreadID.x] = dispatchThreadID.x;

    // Synchronize the wave and issue a memory barrier.
    //
    // For all threads in the wave, the writes to
    // output happen before the write that signals completion
    AllMemoryBarrierWithWaveSync();

    // signal the completion by the first thread of the wave
    if (WaveIsFirstLane())
    {
        syncBuffer[0].store(1, MemoryOrder.Relaxed);
    }
}

A similar consideration applies to reduce operations, which are typical in neural network computation. It is often better to first collate the results within a wave (e.g., WaveActiveSum()) and then use a single thread to perform the atomic operations to write the results to shared memory.

📝 Remark: For shared memory reduction operations, atomic reductions that do not provide return values may provide better performance than atomic operations that do. See, e.g., Atomic<T>.reduceAdd().