The memory hierarchy: where the data sits decides how the interface feels

A person judges an interface by its tempo. A key that answers at once, a list that scrolls without stutter, a first screen that appears before doubt sets in: these are properties of the machine underneath, which is mostly a memory system. One thing decides the tempo: whether the bytes an interface needs are already close to the processor or must be fetched from somewhere far away.

This first of ten units argues that the tier of the memory and storage hierarchy holding an interface's data sets its felt speed; how much arithmetic runs over that data matters less. The sharpening is that placement is now a policy a system chooses and measures. The next unit stays on the machine and asks who gets the processor.

A hierarchy is an assumption about program behaviour

A hierarchy is worth building only if programs reuse a small set of items. Wilkes (1965) proposed a small fast store holding recently used blocks beside main memory. He argued that a program's locality of reference would keep the hit rate high, so the cache begins as an assumption about behaviour rather than as hardware. Denning (1967) made that assumption measurable, defining the working set as the pages a process references in a window of time and showing that keeping it resident prevents thrashing. If reuse is real, a fast tier earns its cost; if it can be measured, the tier's size is a decision rather than a guess.

The gap is structural, and it has been measured

Wulf and McKee (1995) projected that processor speed would outgrow DRAM bandwidth, argued that memory latency rather than arithmetic would limit performance, and named the memory wall. Patterson (2006) assembled two decades of figures showing the same pattern across memory, disk and network: bandwidth improves far faster than latency, so a design should tolerate latency rather than assume it will fall.

Wulf and McKee assumed DRAM improving by 7 per cent a year against a processor improving by 80 per cent a year, and calculated that processor cycles per memory access would rise from 1.52 in 2000 to 8.25 in 2005 and 98.8 in 2010, putting the wall less than a decade away. Patterson records the same shape and its cause: microprocessors and networks gained as much as 1000 to 2000 times in bandwidth but only 20 to 40 times in latency, because more transistors and pins add bandwidth while longer on-chip distances hold latency back. That is a mechanism and a forecast, not a law, since Patterson adds that caching and prefetching hide part of it.

The two ends of the hierarchy are both sets of trade-offs

Smith (1982) surveyed cache design and consolidated the parameters that govern it, covering mapping, replacement policy and write strategy.

The bottom of the hierarchy is not merely slow, which is why its latency cannot simply be optimised away. Chen, Lee, Gibson, Katz and Patterson (1994) classified disk arrays into levels and compared the trade each makes between redundancy, performance, capacity and cost, showing the storage tier as a compromise in which reliability is a first-class term. Agrawal, Prabhakaran, Wobber, Davis, Manasse and Panigrahy (2008) found that flash performance depends on access pattern, internal parallelism and the flash translation layer rather than raw bandwidth, so a designer must know which tier is addressed.

RAID level 5 shows the trade end to end: striping with distributed parity survives one disk failure but pays a small-write penalty, so reliability is bought with write performance rather than capacity.

A budget is written from ratios, and every latency figure depends on its load

The hierarchy becomes an instrument when it is reduced to numbers a designer can budget against. Gray and Shenoy (2000) tabulated order-of-magnitude latency and capacity figures for memory, disk, tape and network, derived rules from the ratios between them, and offered the table as a budgeting instrument rather than a benchmark. The machine budgeted for is rarely the user's own: on a low-cost device the fast tier is small, so placement is the margin its owner can move, and a remote fetch is billed as well as delayed. A placement decision asks three questions:

1. Which tier holds the bytes this transaction needs, and what does a fetch cost?
2. What would holding them in the tier above cost, in capacity and money?
3. Is that ratio large enough to justify the placement, and for how long?

A latency figure is a measurement under a workload model. Schroeder, Wierman and Harchol-Balter (2006) showed by measurement and simulation that open and closed workloads produce materially different response-time curves, so a benchmark that closes the system can mislead. A statement about placement must name the load under which it was measured, or it is not testable.

Placement has become a software decision, and it is measured

The placement decision has moved from hardware into software, where a system sets it and a measurement reports it. Lagar-Cavilla and colleagues (2019) describe a software-defined far-memory tier deployed across Google's warehouse-scale computers since 2016. It compresses cold pages into a cheaper tier under a service objective. They report at least a 67 per cent reduction in memory cost, access at about six microseconds, and about 20 per cent of infrequently accessed data placed in the far tier. Maruf and colleagues (2023) place hot and cold pages between a fast local tier and a slower CXL tier from the operating system. They report a gap of less than one per cent from an all-local baseline. Li and colleagues (2023) predict how much of each virtual machine's memory must stay local. Their policy cuts DRAM cost by 7 per cent while remaining within 1 to 5 per cent of an all-local allocation across 158 workloads. The CXL tier those systems address now has a measured profile of its own. Liu and colleagues (2025) characterise four real CXL devices across 265 workloads, reporting access latencies of 140 to 410 nanoseconds, the first CXL-induced tail latencies, and a prefetcher that handles the slower tier poorly.

Contested question: is placement a hardware fact or a design variable?

The structural position holds that the machine fixes placement: the cache decides by a policy set at design time (Smith, 1982), and the memory wall is a property of physics and cost (Wulf and McKee, 1995), so the designer budgets around a hierarchy that will not move. The software position holds that the tier boundary is a tunable policy: the operating system promotes and demotes pages (Maruf et al., 2023), the kernel compresses cold pages into the far tier (Lagar-Cavilla et al., 2019), and a predictor decides what each tenant keeps local (Li et al., 2023).

If placement is a hardware fact, gains come from capacity and workloads that fit; if it is a design variable, a fixed machine gains through a better policy. The evidence that would settle the question is a controlled comparison: a hardware-managed hierarchy against an explicitly managed one on the same latency-sensitive workload, with capacity held constant and placement the only difference. Schroeder and colleagues (2006) complicate the test, since response-time curves depend on the workload model, so the comparison must state its load or establish nothing. The software position rests on a few production deployments rather than a broad replication, so the disagreement is live.

The evidence on each side differs in kind. The structural position rests on a projection and a design survey rather than a measurement under load (Smith, 1982; Wulf and McKee, 1995). The software position rests on production deployments and a device-scale measurement (Lagar-Cavilla et al., 2019; Maruf et al., 2023; Li et al., 2023; Liu et al., 2025). While the dispute is open, the designer sizes the tier first and treats placement as a lever to measure rather than assume. The strongest objection is that these cases show placement can be moved by firms with unusual telemetry, not that it is freely movable for every system.

Where the argument is heading

The trajectory points past placement towards the movement of computation itself. Mutlu, Ghose, Gomez-Luna and Ausavarungnirun (2022) argue that modern machines are built to bring data to computation, that off-chip movement costs far more than computation, and they survey architectures that put computation in or near the memory chips. At the other extreme Ousterhout and colleagues (2010) argue for keeping a datacentre's storage in DRAM, projecting 100 to 1000 times the throughput and 100 to 1000 times lower access latency than disk-based storage, the strongest statement here that the fast tier is a choice. Which tier holds the bytes this task needs, and who decides?

The sources this unit rests on

• Wilkes, M. V. (1965). Slave memories and dynamic storage allocation. IEEE Transactions on Electronic Computers. doi:10.1109/pgec.1965.264263 Proposed a small fast store holding recently used blocks beside main memory, arguing that locality of reference would keep the hit rate high. The founding statement of the hierarchy as an assumption about behaviour rather than hardware.
• Denning, P. J. (1967). The working set model for program behaviour. Proceedings of the ACM Symposium on Operating System Principles (SOSP). doi:10.1145/800001.811670 Defined the working set as the pages a process references in a window of time, showing that keeping it resident prevents thrashing. It turns locality into a measurable quantity, which is what lets the tier be sized.
• Smith, A. J. (1982). Cache memories. ACM Computing Surveys. doi:10.1145/356887.356892 Surveyed cache design and consolidated mapping, replacement policy and write strategy as the governing parameters. Used here for the claim that the top tier is a compromise rather than a single best design.
• Chen, P. M., Lee, E. K., Gibson, G. A., Katz, R. H. and Patterson, D. A. (1994). RAID: high-performance, reliable secondary storage. ACM Computing Surveys. doi:10.1145/176979.176981 Classified disk arrays into levels and compared the trade each makes between redundancy, performance, capacity and cost. It shows the slow tier as engineered for reliability as much as speed.
• Wulf, W. A. and McKee, S. A. (1995). Hitting the memory wall: implications of the obvious. ACM SIGARCH Computer Architecture News. doi:10.1145/216585.216588 Projected that processor speed would outgrow DRAM bandwidth and argued that memory latency rather than arithmetic would limit performance, naming the memory wall. The origin of the unit's framing that the gap is structural.
• Gray, J. and Shenoy, P. (2000). Rules of thumb in data engineering. Proceedings of the International Conference on Data Engineering (ICDE). doi:10.1109/icde.2000.839382 Tabulated order-of-magnitude latency and capacity figures for memory, disk, tape and network and derived design rules from the ratios between them. It offers the hierarchy as a budgeting instrument.
• Patterson, D. (2006). Latency lags bandwidth. International Conference on Computer Design (ICCD). doi:10.1109/iccd.2005.67 Assembled two decades of figures showing that bandwidth improves far faster than latency across memory, disk and network, and argued that designs should tolerate latency. The general law behind this unit's claim.
• Schroeder, B., Wierman, A. and Harchol-Balter, M. (2006). Open versus closed: a cautionary tale. USENIX Symposium on Networked Systems Design and Implementation (NSDI). https://www.usenix.org/conference/nsdi-06/open-versus-closed-cautionary-tale Showed by measurement and simulation that open and closed workloads give materially different response-time curves, so a benchmark that closes the system can mislead. The caution that conditions the unit's latency claim on a named model.
• Agrawal, N., Prabhakaran, V., Wobber, T., Davis, J. D., Manasse, M. and Panigrahy, R. (2008). Design tradeoffs for SSD performance. USENIX Annual Technical Conference. https://www.usenix.org/conference/2008-usenix-annual-technical-conference/design-tradeoffs-ssd-performance Found that flash performance depends on access pattern, internal parallelism and the flash translation layer rather than on raw sequential bandwidth. A tier whose behaviour changed with its technology.
• Ousterhout, J., Agrawal, P., Erickson, D., Kozyrakis, C., Leverich, J., Mazieres, D., Mitra, S., Narayanan, A., Parulkar, G., Rosenblum, M., Rumble, S. M., Stratmann, E. and Stutsman, R. (2010). The case for RAMClouds: scalable high-performance storage entirely in DRAM. ACM SIGOPS Operating Systems Review. doi:10.1145/1713254.1713276 Argued for keeping an entire datacentre's storage in DRAM and projected 100 to 1000 times the throughput and 100 to 1000 times lower access latency than disk-based storage. The strongest statement here that the fast tier is a choice.
• Lagar-Cavilla, A., Ahn, J., Souhlal, S., Agarwal, N., Burny, R., Butt, S., Chang, J., Chaugule, A., Deng, N., Shahid, J., Thelen, G., Yurtsever, K. A., Zhao, Y. and Ranganathan, P. (2019). Software-defined far memory in warehouse-scale computers. International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). doi:10.1145/3297858.3304053 Reported a far-memory tier deployed across Google's warehouse-scale computers since 2016, compressing cold pages into a cheaper tier, with at least a 67 per cent memory-cost reduction. Production evidence that placement is a measured decision.
• Maruf, H. A., Wang, H., Dhanotia, A., Weiner, J., Agarwal, N., Bhattacharya, P., Petersen, C., Chowdhury, M., Kanaujia, S. and Chauhan, P. (2023). TPP: transparent page placement for CXL-enabled tiered-memory. International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). doi:10.1145/3582016.3582063 Placed hot and cold pages between a fast local tier and a slower tier from the operating system, reporting a gap of less than one per cent from an all-local baseline. Placement automated and deployed as a kernel policy.
• Li, H., Berger, D. S., Hsu, L., Ernst, D., Zardoshti, P., Novakovic, S., Shah, M., Rajadnya, S., Lee, S., Agarwal, I., Hill, M. D., Fontoura, M. and Bianchini, R. (2023). Pond: CXL-based memory pooling systems for cloud platforms. International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). doi:10.1145/3575693.3578835 Predicted how much memory each virtual machine must keep local in order to pool the rest, cutting DRAM cost by 7 per cent while staying within 1 to 5 per cent of all-local performance. Placement as a priced, shared resource.
• Liu, J., Hadian, H., Wang, Y., Berger, D. S., Nguyen, M., Jian, X., Noh, S. H. and Li, H. (2025). Systematic CXL memory characterization and performance analysis at scale. ASPLOS 2025. doi:10.1145/3676641.3715987 Characterises four real CXL devices across 265 workloads, seven latency levels and five CPU platforms. It reports sub-microsecond access latencies, the first CXL-induced tail latencies and prefetcher inefficiencies, giving the deployed tier a measured profile.
• Mutlu, O., Ghose, S., Gomez-Luna, J. and Ausavarungnirun, R. (2022). A modern primer on processing in memory. In Emerging Computing: From Devices to Systems. Springer. doi:10.1007/978-981-16-7487-7_7 Argued that modern machines bring data to computation, that off-chip movement costs far more than computation, and surveyed architectures that place computation in or near memory. The strongest counter-argument here, making computation's location the variable.