The Hidden Bottleneck Behind AI Scaling

8 juillet 2026 — 12 min read

The next phase of AI infrastructure will be defined by better data efficiency and addressing the limitations of today’s memory architecture

The Shift from Compute to Locality

AI infrastructure has been built around a simple assumption: more compute will unlock more performance, and therefore, more intelligence. For a long time, that assumption worked. Each generation of processor brought more transistors, more parallelism, and more theoretical performance. The industry’s benchmarks followed the same logic, celebrating larger accelerators and increasingly powerful training clusters, with the common story that scaling AI meant scaling compute.

But we have reached a hard physical limit. Classical memory architectures are no longer keeping pace. Expensive, power-hungry High Bandwidth Memory (HBM) still keeps data physically distant from the compute units, while even advanced AI chips can only embed tiny amounts of SRAM, far too little to house modern models.

This has created a new, less visible constraint: the bottleneck is no longer raw arithmetic capacity but the inability of today’s memory architecture to efficiently deliver data to the compute units. As models grow, and with the shift toward Agentic AI, the memory system is placed under even more sustained pressure. Now that systems must deliver intelligence across cloud, edge, and device environments, the defining question is shifting: not how much compute can be placed on a chip, but how efficiently we can feed that compute with data.

This is the “memory wall”, the hidden bottleneck behind AI scaling: a structural issue, not a temporary imbalance. Jensen Huang, the CEO of NVIDIA, once admitted: « GPUs spend 70% of their time waiting for data. » That structural issue sits at the intersection of semiconductor physics, memory architecture, and data center economics.

At Vertical Compute, our view is simple: the future of AI depends on reducing the distance between where data is located and where it is processed. The industry has spent decades optimizing compute. The next era will be defined by memory and data locality.

The Shift: Why Data Movement Defines the AI Era

Modern AI systems are often described through the language of compute. We talk about TFLOPS, number of CPU/GPU/xPU cores, memory capacity, bandwidth, rack density and cluster scale. These metrics matter, but they only describe one part of the system. A processor can only work on data that is available to it. If the data is not available at the right time, the compute units wait. This is the essence of the “memory wall” : a concept identified decades ago as the performance gap between processors and memory began to widen, which AI has now transformed into a critical system-level bottleneck.

In classical computing, many workloads’ performance was limited by instruction execution, instruction per cycle (IPC) efficiency, control flow or software behavior. In AI, and especially with large language models (LLMs), systems repeatedly move enormous volumes of weights and cached context through the memory hierarchy. The arithmetic here is significant, but the data movement surrounding that arithmetic increasingly dominates performance, power and cost: It consumes 100X more energy to move data than to perform math operations.

AI is often described as computation at scale, but a more accurate description may be data movement at scale with computation embedded inside it. Every element of the workload, from storing model parameters to generating each token via language models, creates continuous movement across caches, memory stacks, and interconnects. The physical cost of moving this data has become one of the defining constraints of AI system design.

Generative AI adds long-lived context. Furthermore, inference is where this bottleneck becomes industrial. While the memory bottleneck was visible during training, inference is continuous, distributed, and economically sensitive. Large language model inference is heavily dependent on memory bandwidth and capacity, especially during the token-by-token output (decode) phase, where the system must repeatedly access model weights and cached context. Agentic AI further compounds this effect: a single task can require 5 to 30 times more tokens than a simple prompt-response interaction, involving planning, reasoning, and tool interactions that increase the sustained pressure on memory access.

This leads to the uncomfortable reality: many of today’s most advanced AI systems are not constrained by a lack of arithmetic; they are constrained by the cost and energy required to keep arithmetic units busy. If the majority of active energy in an AI system is consumed moving data, simply adding more compute does not solve the problem, it can make it worse by demanding more data, increasing bandwidth pressure, heat, and infrastructure overhead.

The future of AI infrastructure will therefore be measured differently. Success will be determined by useful compute per watt, per dollar, and per square millimeter of silicon; a shift that places locality, bandwidth density, latency, and memory capacity at the center of AI system design.

The limits of pushing traditional memory further

While HBM provides essential bandwidth, its physical separation from compute logic is a major factor in the memory wall we face today. This delocalization forces data to traverse increasingly complex interconnects, consuming up to 100x more energy for movement than for the actual computation. Furthermore, the heat generated by these high-speed data transfers necessitates sophisticated, costly cooling solutions and often leaves compute units idling, contributing to the reality that GPUs can spend 70% of their time waiting for data. As a result, the distance between memory and compute has become a primary bottleneck that limits both the energy efficiency and the scalability of modern AI systems.

High Bandwidth Memory has been one of the most important enablers of modern AI. It has dramatically increased bandwidth by stacking DRAM dies and placing them close to the processor through advanced packaging. Without HBM, today’s AI acceleration landscape would look very different.

But HBM is not a complete answer to the memory wall.

It is still external to the compute logic. It relies on complex, expensive packaging that essentially restricts its use to the datacenter. It still requires data to move between separate memory and compute structures. And as AI systems scale, the cost and availability of HBM become increasingly important constraints.

This is not a criticism of HBM. It is an acknowledgement of what it was designed to solve. HBM provides more bandwidth than conventional memory approaches. But it does not fundamentally eliminate the physical separation between data storage and computation.

The same is true for many other techniques that extend existing architectures. Larger caches help. Better interconnect helps. Advanced packaging helps. Optical links may help at rack and system scale.

The challenge of locality has led to diverse architectural responses.

Companies like Cerebras have pursued extreme locality by building massive Wafer-Scale Engines (WSE-3) that integrate massive amounts of fast SRAM directly onto a single silicon wafer. This eliminates off-chip data movement entirely for models that fit, offering immense on-chip memory bandwidth. This approach addresses the memory bottleneck by bringing data maximally close to compute, though it results in an extremely expensive, inflexible solution that requires external system racks for larger models. In contrast, NVIDIA’s acquisition of Groq‘s Language Processing Units (LPUs) represents a different SRAM strategy. LPUs are ordinary-sized chips with a small amount of on-chip SRAM. They excel in low-latency inference by quickly processing the memory-bound decode phase, but larger models require connecting hundreds of these LPUs in a complex cluster. NVIDIA leverages its ecosystem by pairing these SRAM LPUs with its GPUs, using HBM for the prompt prefill phase and the SRAM LPUs for the output decode phase. This hybrid model highlights the industry’s mixed efforts to solve the memory bottleneck, either through massive monolithic integration or through complex system-level orchestration of different memory types.

Other innovative approaches focus on eliminating data movement entirely, such as Axelera AI’s Metis AIPU which utilizes Digital In-Memory Computing (D-IMC). This architecture performs matrix operations directly within the memory arrays, achieving high energy efficiency and dramatically reducing the energy cost of moving data. While D-IMC allows energy efficiency and data proximity for smaller networks (such as those used in edge inference workloads), the on-chip memory (SRAM) required for this method is too large to hold the billions of parameters found in the largest AI models.

In a similar vein, Fractile AI addresses the inference memory wall by utilizing its accelerator to minimize high-latency, power-intensive data movement through extensive on-chip SRAM. Fractile’s solution focuses on optimizing the decode phase by integrating massive, high-speed on-chip SRAM, which stores the KV cache and model weights locally. By minimizing the high-latency, power-intensive data movement to off-chip HBM, they achieve low-latency, high-throughput inference.

But each one addresses symptoms of the same underlying issue: data is still too far from the compute that needs it.

At the same time, traditional DRAM scaling is becoming harder. Memory density, cost per bit, energy efficiency and manufacturing complexity are all under pressure. The industry can continue to optimize the existing hierarchy, but those optimizations are becoming more expensive and less linear.

This is the point at which architecture matters again.

For years, semiconductor progress could rely on manufacturing improvements to hide architectural inefficiencies. That era is ending. The next phase will require new ways of organizing memory and compute, not simply better versions of the old separation.

The economics of AI infrastructure are being rewritten

The memory bottleneck is not only a technical constraint. It is reshaping the economics of the AI industry.

AI infrastructure is capital intensive because the system is capital intensive. Advanced accelerators, memory stacks, packaging, networking, power delivery and cooling all contribute to the total cost of deploying intelligence at scale. As more of that cost shifts toward memory and data movement, the economics become harder to ignore. For example, on a leading-edge accelerator like the NVIDIA B200, memory-related silicon accounts for approximately 65% of the total chip area, with HBM stacks now representing one of the fastest-growing and most critical cost components in the bill of materials

This matters for hyperscalers, but it also matters for everyone else.

If AI infrastructure remains dependent on extremely expensive memory subsystems and centralized data center capacity, access to advanced AI will concentrate around the few organizations able to fund and operate that infrastructure. That has implications for competition, innovation, sovereignty and the ability to deploy AI closer to where data is created.

The economics also affect product design. As the cost per token falls, usage does not necessarily become cheaper overall. Lower unit cost can unlock higher demand. If AI becomes embedded into every workflow, application and device, total token volume may grow much faster than efficiency improvements. The result is a paradox: individual inference may become cheaper, while total infrastructure demand rises dramatically.

That is why architectural efficiency matters.

The goal is not simply to reduce the cost of a single operation. It is to change the scaling curve. AI needs infrastructure that can support orders of magnitude more usage without requiring proportional growth in energy, memory bandwidth, cooling and capital expenditure.

This shift requires us to rethink the fundamental architecture of our systems; by moving from compute-first designs to a memory-centric paradigm, we can build infrastructure that scales sustainably without the compounding costs of traditional data movement.

Vertical Compute’s approach, known as Vertically Integrated Memory(™) (VIM), fundamentally rethinks chip architecture by stacking high-density magnetic memory directly on top of compute logic. This shortens the distance data needs to travel to nanometers, moving from data movement to data proximity. The objective is not to replace every element of the existing hierarchy. It is to create new points in the system where high-density memory and compute can interact with much greater locality and efficiency.

Toward memory-centric AI architectures

The next decade of AI hardware will not be defined by a single technology or packaging technique. It will be defined by a broader architectural shift toward memory-centric design.

Memory-centric does not mean compute becomes unimportant. It means compute performance is understood in relation to data access. It means the location, density and movement of memory become first-order design decisions. It means systems are optimized around the physical reality that moving data is expensive.

We can already see that AI architects are paying more attention to memory bandwidth per watt, memory capacity per accelerator, cache behavior, KV-cache management, chiplet architectures and system-level data orchestration. The industry is moving from a compute-first model toward a data efficiency and memory-centric model.

Vertical integration is part of this broader evolution.

That matters because the best architecture is not necessarily the one with the most compute. It is the one that delivers the most useful work under real constraints.

For cloud infrastructure, this means improving accelerator utilization and reducing the energy wasted on data movement. For edge and device environments, it means enabling more AI capability within strict power, thermal and cost limits. For agentic and physical AI, it means supporting continuous inference where latency, locality and energy efficiency are critical.

A new definition of AI scaling

The industry is entering a new phase.

The first phase of AI scaling was dominated by a relentless pursuit of peak compute. The next phase will be dominated by better data efficiency and a fundamental restructuring of memory architecture.

The memory wall is a systemic bottleneck, impacting the economics, energy profile, and accessibility of AI. Solving it requires more than incremental improvements; the industry must rethink the assumption that memory and compute can remain physically separated. The future of AI will be defined by proximity: between data and compute, capacity and bandwidth, and intelligence and the environments where it is deployed.

That is the architectural shift now underway, and it may define the next decade of computing more than any single increase in peak performance.

The compute era is not ending. But the data-locality era has begun.

The hardware revolution to support the next era of AI is happening now. By solving the memory bottleneck today, we are laying the foundation for the persistent, intelligent systems of tomorrow. If you are an architect, developer, or infrastructure lead working on memory-intensive scaling, we invite you to join the conversation on defining the data-locality era.