Memory now accounts for nearly two-thirds of AI chip component costs
An analysis published by Epoch AI has quantified a decisive shift in the cost structure of modern AI accelerators: memory now consumes nearly two-thirds of total chip component costs. The finding, which climbed to the top of Hacker News with 315 points, underscores how the insatiable memory bandwidth demands of large language models and deep learning workloads are reshaping semiconductor economics. As AI models grow larger and training clusters scale to tens of thousands of GPUs, the cost of high-bandwidth memory (HBM) and advanced packaging has overtaken the logic compute die as the dominant expense.
Historically, GPU and AI accelerator cost breakdowns were dominated by the compute die—the massive silicon area packed with tensor cores and SIMD units. Memory was a significant but secondary line item. Epoch AI's data indicates that in the latest generation of AI chips, such as NVIDIA's H100 and AMD's MI300X, memory subsystems now represent roughly 65% of total Bill of Materials. This marks a dramatic inversion from just a few years ago, when compute represented 60% or more of costs.
Drivers: HBM3 memory and advanced packaging
The primary culprit is HBM3 and its integration via silicon interposers. HBM stacks are themselves expensive multi-die packages, and the interposer that bridges memory and compute adds non-trivial yield and substrate costs. As chipmakers stack more HBM dies—up to 144GB on some accelerators—the memory cost scales superlinearly. Meanwhile, compute die costs have not increased at the same rate because Moore's Law is slowing, and reticle limits constrain die size. Consequently, memory's share has ballooned.
This trend is visible in cost teardowns of recent products. For example, NVIDIA's H100 80GB HBM2e card had memory costs around 50% of BOM; the H200 with 141GB HBM3e pushes that closer to 60%. AMD's MI300X, which stacks 192GB of HBM3 across chiplets, likely exceeds 65%. Epoch AI's analysis aggregates these data points to confirm the broader trajectory.
Beyond HBM, the analysis also highlights the role of cost in DRAM controllers, memory channels, and on-chip SRAM. While SRAM does not dominate absolute cost, its die area grows with each architecture to meet inference latency targets, further pressuring the memory budget.
Implications for chip architecture and AI economics
The dominance of memory costs has profound implications for chip design choices. Traditional brute-force scaling of compute units becomes less economically attractive if memory cannot feed them. This validates industry moves toward closer memory integration, such as Samsung's memory-centric compute designs and Intel's efforts with HBM and near-memory compute.
For cloud providers and AI companies, the cost shift changes total cost of ownership calculations. Memory is a per-device cost that scales linearly with capacity, whereas compute can be time-shared. As memory takes a larger share, idle memory bandwidth becomes a more expensive waste. This reinforces the push for memory pooling (e.g., CXL-attached memory) and disaggregated architectures where memory can be allocated flexibly across accelerators.
Additionally, the finding puts pressure on DRAM manufacturers. HBM3 prices have remained high due to limited supply and complex packaging. If memory continues to dominate, chip designers may explore alternatives like die-stacked SRAM or embedded DRAM, though these come with their own cost and performance trade-offs.
Impact on next-generation AI accelerators
The Epoch AI analysis arrives as companies prepare next-gen products. NVIDIA's Blackwell architecture is expected to use more HBM3e with likely 192GB per chip, further tilting the cost balance. AMD's next CDNA iteration and Intel's Falcon Shores will face similar pressures. Startups like Cerebras, which uses wafer-scale memory integration, may gain a relative advantage if they can lower memory subsystem costs through different packaging approaches.
Furthermore, the memory cost dominance incentivizes software optimization. Techniques that reduce memory footprint—such as quantization, sparse attention, and model compression—become not just performance improvements but direct cost savings. This could accelerate adoption of lower-precision formats (FP4, INT4) and pruning methods in production deployments.
Outlook: memory cost share may continue to climb
Looking ahead, Epoch AI's data suggests the trend is not yet at its peak. As AI models push beyond 1 trillion parameters, memory capacity and bandwidth requirements will grow faster than compute throughput improvements. Unless radical new memory technologies (e.g., compute-in-memory, photonic memory) reach commercialization, memory's share of AI chip component costs could approach 75-80% within two GPU generations.
For the AI community, this analysis serves as a reality check: the bottleneck in scaling is no longer just compute, but the memory subsystem that feeds it. Chip architects, hyperscalers, and AI startups must adjust their strategies accordingly or risk being priced out of the next wave of model innovation.
Comments