Idle GPU Hours Are the Cheapest AI Capacity Most Operators Aren't Selling
- Natalie Stipan
- 372 views
Inference servers average about 20% utilization, so most paid GPU capacity sits idle. Pooling those spare cycles into an open market is one way to recover it.


Most of the AI capacity conversation is about buying more. Gartner projects that global data center electricity use will grow 26% in 2026, to 565 terawatt-hours, according to Gartner, with most of that growth coming from AI-optimized servers. New clusters keep running into limits on memory supply and grid access. My read is that the cheapest capacity in the building right now isn't on any purchase order. It's the GPU hours that were already paid for and already powered, and nobody is using them.
The Gap Between Installed and Delivered Compute
Training clusters tend to run hot. Lawrence Berkeley National Laboratory's latest U.S. modeling assumes they average about 80% utilization. Inference fleets are a different story. They get sized for peak traffic, and then they sit underused for long stretches of the day.
That same modeling assumes AI inference servers will average about 20% utilization by 2030. That's half the figure the lab's previous edition used. It also assumes idle machines still draw roughly a fifth of their rated power, and that grid interconnections run at around half their rated capacity because of redundancy requirements and maintenance.
Put those numbers next to a depreciation schedule and the problem comes into focus. At 20% average utilization, about four-fifths of what an inference server costs in depreciation and floor space is paying for capacity that does no work. Teams in this position often focus on the server price, but the idle share is where the money actually leaks.
Why Idle Hours Cost More on Modern Hardware
Large-model inference is often limited by memory before it's limited by compute. The amount and bandwidth of high-bandwidth memory (HBM) on each card can cap throughput first. The International Energy Agency expects the current HBM shortage to last through at least the end of 2027. When a model is split across many GPUs, performance also drops sharply once the interconnects between those GPUs saturate.
Denser hardware makes each idle hour more expensive. The IEA estimates that AI server power density rose elevenfold between 2020 and 2025 and will quadruple again by 2027. At those densities, liquid cooling stops being optional. The source material puts the heat load of a single AI rack at roughly that of 30 natural gas boilers, a comparison worth treating as the author's illustration rather than a measured figure. Liquid-cooled AI halls helped bring the U.S. average power usage effectiveness (PUE) down from 1.55 to 1.45 between 2018 and 2024, according to the Berkeley Lab report.
All of this raises the capital a GPU has to earn back during the hours it actually works. A card that sits at a fifth of its potential utilization carries that cost with almost no return.
Pooling Spare Cycles Into an Open Inference Market
The argument from the author, David Liberman, is that a decentralized compute network can recover some of that idle capacity. The model he describes pools GPUs from data centers, cloud providers, and independent operators into a single inference market. Requests can route to whichever accelerators have spare capacity, so a card that one facility can't keep busy can still earn.
The network he co-created, Gonka, is designed to send nearly all contributed compute to AI workloads and keep consensus overhead low. In his account, hosts prove their capacity through a short, transformer-based compute sprint, and outside that window their hardware serves inference. Other participants re-check only a small random sample of results. That's a vendor's description of its own design, and it hasn't been independently measured in the source. Treat it as a claim to test, not a benchmark.
Two details matter for operators weighing this. First, the network reportedly lets a node mix NVIDIA GPUs from the A100 to the B300, provided the pool holds at least 320 GB of memory. That's useful for owners of older A100 and H100 fleets who would otherwise see those cards depreciate next to newer hardware. Second, data center operators could earn on capacity they built for peak demand, which is the part of the fleet that sits idle most of the day.
Where Open Pools Fit and Where They Don't
Inference is the workload that suits this model today. A shared pool can absorb demand spikes and narrow the gap between peak provisioning and average load. Frontier-scale training is another matter. It still depends on ultra-fast fabrics inside a single facility, and no spare-cycle market replaces that.
Microsoft, for its part, plans to begin deploying its Maia 300 accelerator in 2027 and says it will deliver 40% more AI tokens per megawatt than leading GPU-based systems. That's the vendor's claim, and it points to a parallel strategy. Custom silicon targets better efficiency per watt on new hardware, while pooling targets better utilization on hardware that already exists. Both can be true at once, and neither helps a team that needs capacity this quarter.
What to Check in Your Own Capacity Plan
If you run inference infrastructure, start with a number you may not track: average utilization across the fleet, measured over a full day rather than at peak. If that figure is anywhere near 20%, the spare capacity is a cost line already on your books, and the question becomes whether any of it can earn.
Then look at memory per node, since that's what decides whether mixed-generation cards can pool together. And check how your power and cooling costs break down per idle hour, because idle machines still draw a meaningful share of rated power under the modeling cited above.
Utilization isn't a substitute for new capacity when training demand is growing. But it's cheap to measure and easy to ignore. My bet is that it ends up on AI infrastructure scorecards right alongside megawatts and PUE. Open networks like this one are one way to push that number up, and whether they deliver in practice will show up in utilization figures teams publish, not in the pitch.