TL;DR

Memory is slow, Disk is faster.

With a fast enough set of disks you can feed data into the L3 cache over PCIe faster than you can stream from DRAM from a single thread.

Why a Part 2.5

In Part 2 I showed disk beat memory, but only on a technicality. If the data was actually in a buffer, at 13 GiB/s RAM was faster than my 6 GiB/s disk array.

However, I followed up with:

On a proper modern server the CPUs will let you do IO directly to the L3 cache, bypassing memory altogether. Because PCIe bandwidth is higher than memory bandwidth, on paper we could even get more max bandwidth than we can get from memory if we carefully pin the buffers into the CPU cache. I haven't confirmed this works in practice...

I upgraded my lab. I confirmed this works in practice.

The new rig

To get this to work I swapped out my AMD with an Intel. I have 8 disks distributed in 4 machines connected by 100Gbit IB, for other testing. It provides full bandwidth to the disk, with a little hit to queue depth and latency.

The counting box (initiator)

The disks (targets)

RAM performance

Here's what this CPU can read with likwid's load kernels:

One core reading from DRAM peaks at 13 GiB/s, but if the data fits in its L3 it can reach 24.8 GiB/s, almost twice the DRAM ceiling. By landing the disk IO, which peaks at 22.6 GiB/s, into L3 instead of DRAM we can exceed the single core bandwidth limit.

The results

Why AVX-512 was necessary

In Part 2 the unrolled loop got the compiler to emit 128-bit SSE2 compares, and that was just faster than the DRAM. On this machine the unrolled loop compiled out slower than the DRAM. So I had to write the count function with AVX-512 intrinsics.

#include <immintrin.h>

// count_avx512: compare 16 ints to a mask, then a masked subtract keeps the
// tally in the vector unit.  4 accumulators, 64 ints per iteration.
static inline __attribute__((target("avx512f")))
size_t count_avx512(const void* data, size_t bytes) {
    const int* p = __builtin_assume_aligned(data, 64);
    const size_t total = bytes / sizeof(int);
    const __m512i ten = _mm512_set1_epi32(10), one = _mm512_set1_epi32(1);
    __m512i a0 = _mm512_setzero_si512(), a1 = a0, a2 = a0, a3 = a0;
    size_t i = 0;
    for (; i + 64 <= total; i += 64) {
        a0 = _mm512_mask_sub_epi32(a0, _mm512_cmpeq_epi32_mask(_mm512_load_si512(p + i), ten), a0, one);
        a1 = _mm512_mask_sub_epi32(a1, _mm512_cmpeq_epi32_mask(_mm512_load_si512(p + i + 16), ten), a1, one);
        a2 = _mm512_mask_sub_epi32(a2, _mm512_cmpeq_epi32_mask(_mm512_load_si512(p + i + 32), ten), a2, one);
        a3 = _mm512_mask_sub_epi32(a3, _mm512_cmpeq_epi32_mask(_mm512_load_si512(p + i + 48), ten), a3, one);
    }
    __m512i sum = _mm512_add_epi32(_mm512_add_epi32(a0, a1), _mm512_add_epi32(a2, a3));
    size_t count = (size_t)(uint32_t)(-_mm512_reduce_add_epi32(sum));
    for (; i < total; i++) count += (unsigned)(p[i] == 10);   // leftovers
    return count;
}

This code tops out at 127 GiB/s, which means it's faster than L3 but slower than L2. And it's 22x faster than the simple loop.

DDIO

With Intel's Data Direct I/O (DDIO), when a PCIe device writes to memory, the write is allocated into the L3 cache instead of going out to DRAM. Here, the ConnectX-4 receives the RDMA data from the targets and writes it into my buffers. Because the buffers fit in L3, those lines are still in L3 when core 0 reads them. The core reads at L3 speed (up to 24.8 GiB/s), not DRAM speed (13.1 GiB/s).

By default DDIO may only use 2 of the 11 L3 ways, which is about 6.5 MiB. I opened it up to all 11. We can show the DDIO feature was working because the benchmark measured the memory controller's own counters (uncore IMC CAS reads) during the disk runs. DRAM reads came to 0.42-0.49% of the bytes counted: 0.08 GiB/s while the core counted at 18.3 GiB/s.

The tuning between Part 2 and here

Same idea as Part 2: an io_uring engine streams blocks into buffers, and the main thread counts each buffer as it lands. What changed:

The big shift is the buffer pool. In Part 2 the buffers just had to exist. Here the whole pool has to stay comfortably inside L3, so DDIO's writes are still in cache when the counting core gets to them. That's why queue depth went from 8192 down to 8: more data in flight just spills to DRAM.

Recap

• SIMD, use it.
• Caches, be aware.
• Memory, slower than you think.
• Disks, faster than you think.

Next Time

See me at P99Conf.io 2026-10-22, register now.

Big O leading you astray? Find out in Part 3.

— Jared Hulbert

Source code can be found here