CPU Architecture
On this page 20
Part I — Foundations · Interview reference
Seniors who understand the memory hierarchy and CPU realities explain performance that Big-O cannot. Interviewers use this topic to separate “I know caches exist” from “I can diagnose false sharing / NUMA / branch mispredicts.”
Mental model: latency hierarchy
Approximate relative costs (order-of-magnitude; exact numbers vary):
| Level | Typical latency | Notes |
|---|---|---|
| Register | ~1 cycle | Working set of the ISA |
| L1 cache | ~4 cycles | Per-core; 32–64KB |
| L2 cache | ~10–20 cycles | Per-core or shared |
| L3 cache | ~40–60+ cycles | Shared; MBs |
| DRAM | ~200+ cycles | 100ns order |
| SSD / network | orders worse | Avoid on hot path |
Rule: Algorithms that thrash memory lose to “worse” Big-O with good locality.
Core concepts
Caches and lines
- Data moves in cache lines (commonly 64 bytes), not single bytes
- Spatial locality: nearby addresses likely used soon → arrays win over pointer chasing
- Temporal locality: reuse recently touched data
- Cache miss types: compulsory, capacity, conflict (associativity)
Prefetching & sequential scans
Sequential array scans are fast because hardware prefetchers pull next lines. Linked lists and pointer-rich graphs defeat this.
Pipelines, superscalar, speculation
- CPUs execute many instructions in flight; branch misprediction flushes work
- Branchless code can help or hurt (dependency chains)
- Speculative execution had security side channels (Spectre) — awareness-level for most app interviews
SIMD / vector units
- One instruction on multiple data lanes (AVX, NEON)
- Relevant for codecs, analytics, crypto, JVM/HotSpot auto-vectorization
- Alignment and remainder loops matter
Multicore: coherence & false sharing
- Cores hold copies of lines; MESI-like protocols keep coherence
- False sharing: unrelated variables on same line → ping-pong invalidations under concurrent writes
- Fix: pad/align to cache line; use thread-local aggregation; striped counters
Production case study (high volume)
Context: Metrics library increments per-tenant counters on every request in a multi-tenant API (~millions RPS fleet-wide).
Why seniors care: False sharing turns “cheap” atomics into a scalability ceiling; seniors must diagnose with hardware counters, not guess “need more pods.”
Failure / symptom: CPU high in kernel/lock paths; throughput falls as cores increase; async-profiler shows CAS on adjacent fields.
Resolution: Stripe counters (LongAdder), @Contended padding, thread-local aggregators flushed periodically; confirm LLC invalidation drop.
Seen at / similar to: Netty/JCTools padded sequences; HdrHistogram / Micrometer high-cardinality pitfalls; LMAX Disruptor cache-line padding.
NUMA
- Memory attached to sockets; remote access slower
- Thread migration + allocator locality matter for large JVMs / databases
- Pins, NUMA-aware alloc, and “don’t cross sockets casually” show up in high-scale design talks
Production case study (high volume)
Context: Multi-socket bare-metal / large EC2 for a Redis-like cache and a 200GB+ JVM serving ad-ranking features at high QPS.
Why seniors care: Remote NUMA access looks like “mysterious” p99; thread migration across sockets defeats local alloc; one global AtomicLong QPS counter false-shares and caps scaling long before CPU is busy.
Failure / symptom: Throughput plateaus from 16→32 cores; perf shows high LLC miss / remote DRAM; flame graphs hot in CAS loops on shared counters.
Resolution: numactl / CPU affinity for latency-critical processes; LongAdder / striped counters; pad hot fields (@Contended); verify with perf stat cache-miss and CPI; size heaps with NUMA in mind.
Seen at / similar to: Redis/KeyDB multi-threaded discussions; JVM services at Twitter/Netflix scale; high-frequency trading colocation boxes (extreme form of the same physics).
Virtual memory & TLB
- Page tables; TLB misses expensive
- Huge pages for large heaps / DB buffer pools
- Page faults on first touch / swapped memory — cold start and overcommit issues
Production case study (high volume)
Context: Postgres / MySQL buffer pool and a HotSpot heap both in the 64–500GB class behind a payments API. Why seniors care: TLB shootdowns and page faults dominate more than “algorithm” once working set leaves cache; transparent huge pages (THP) can cause latency spikes; overcommit → allocator stalls under load. Failure / symptom: Periodic multi-ms stalls; compacting GC + THP interaction; first-traffic-after-deploy latency until pages faulted in. Resolution: Explicit huge pages where ops supports it; disable problematic THP for DBs (common Postgres advice); pre-touch heaps; track major faults and compaction stalls in prod. Seen at / similar to: Meta/Netflix JVM tuning writeups; AWS RDS/Aurora large instances; Elasticsearch/OpenSearch heap+page-cache balancing.
How software maps to hardware
| Software pattern | Hardware effect |
|---|---|
| Row-major vs column-major traversal | Cache miss storm if wrong |
| Object graph (many small objects) | Pointer chasing; GC pressure (JVM) |
| Contiguous structs / SoA | Better SIMD & prefetch |
Contended AtomicLong | Cache line bouncing |
| Hash map resize | Bandwidth + allocation spike |
| Excessive syscalls / context switches | Pipeline + TLB disruption |
Performance diagnosis vocabulary
Be ready to name tools conceptually:
- CPU profiles (perf, VTune, async-profiler): on-CPU vs off-CPU
- Cache miss counters, CPI (cycles per instruction)
- Flame graphs for hot methods
- Microbenchmarks: JMH pitfalls (dead code elimination, warmup, power states)
Java under the hood
| Concept | How it shows up on HotSpot |
|---|---|
| Object layout | Mark word + class pointer (compressed oops common) + fields; alignment to 8 bytes → padding |
| False sharing | Adjacent fields/atomics on one 64-byte line; @jdk.internal.vm.annotation.Contended / LongAdder striped Cell[] |
| Allocation | TLAB bump-pointer alloc for young objects; escape analysis may scalar-replace / stack-allocate |
| Contiguous vs chase | ArrayList / int[] sequential scan prefetches; LinkedList node chasing defeats L1 |
| Direct memory | ByteBuffer.allocateDirect / Netty: off-heap, fewer GC visits, capacity limits, cleanup via Cleaner |
| Compressed oops | 32-bit references on heaps ≲ ~32GB — big memory win; layout math changes |
| Diagnosis | JMH (warmup, DCE traps), JFR, async-profiler, perf counters for cache misses |
Interview line: “O(n) LinkedList walk is slower than ArrayList because of cache lines and prefetch, not Big-O.” Tie contended atomics to thread safety and GC pressure to JVM.
Production case study (high volume)
Context: After switching a hot path from ArrayList<Integer> boxed walks to int[] / primitive lists, a ride-hailing geo service cut p99 by double-digit percent without changing Big-O.
Why seniors care: Interviewers want this story: locality and allocation beat asymptotic bragging at fixed n in the millions per second.
Failure / symptom: High L1/L2 miss rate; allocation rate GB/s; GC contributing to p99.
Resolution: SoA / primitive arrays; JMH + async-profiler + JFR allocation; reject “optimize LinkedList” cargo cult.
Seen at / similar to: High-scale JVM shops (Netflix, Alibaba Dragonwell tuning posts); game/sim backends; HFT gateways.
Decision criteria (interview)
When asked “optimize this”:
- Measure — confirm CPU-bound vs IO / lock / GC
- Fix algorithmic complexity first if n large
- Then locality (layout, batching), then concurrency scaling (false sharing), then micro-arch
What interviewers probe
- Why is iterating a linked list slower than an array of same n?
- Explain false sharing with a counter array example; how to fix.
- What is a cache line? Impact on padding / atomics.
- NUMA effect on a multi-socket DB or JVM.
- Context switch cost vs spinning briefly (link to threads / pools).
- “O(n) but slow” story from experience.
- THP / TLB / NUMA as root causes when “CPU isn’t busy but latency is high.”
Senior-level expectation: Connect code layout and concurrency design to the memory hierarchy without claiming fake cycle counts.
Pitfalls
- Optimizing for L1 before proving the hotspot
- Sharing packed mutable fields across threads
- Ignoring allocation rate (memory bandwidth + GC) as “CPU”
- Assuming single-socket performance scales linearly to 64 cores
- Over-using huge pages without operational understanding