What a power cap does to inference

Prefill and decode

Every GPU runs under a power limit, and the limit is adjustable. I wanted to know what lowering it costs a GPU that is serving a language model, in both speed and energy.

Inference is really two workloads. Prefill processes the prompt, all of its tokens in parallel. Decode generates the response, one token per step. They stress the hardware in opposite ways, so a single blended number hides most of what happens. I measured them separately.

I ran vLLM on eight H100 SXM GPUs, each rated for 700 W: four serving Qwen3-8B and four serving Qwen3-14B. I stepped each GPU's power limit from 700 W down to 200 W. At each step I ran prefill-only and decode-only load at 32 concurrent requests, and recorded throughput, core clock and energy, the last from the GPU's own energy counter.

Each GPU is compared against its own uncapped run, which removes chip-to-chip variation. The two models agree within a few points, so the charts show Qwen3-8B. The code and raw data are on GitHub.

Decode shrugs off the cap

At 415 W, about 60% of the rated power, decode keeps 96% of its throughput. Prefill keeps 74%.

The lower limit also makes both phases more efficient. At 415 W, decode produces 23% more tokens per joule and prefill 25% more. For decode, that is a fifth less energy per token for a 4% loss in speed.

The gains have a floor. Below roughly 345 W, throughput drops sharply in both phases, and at 200 W decode runs at half its uncapped speed. Batch size moves this line: at 270 W, decode keeps 85% of its speed with one request in flight and 71% with 32.

The same sweep on the H100 PCIe, a 350 W part, shows the same split without the sharp drop. Decode still keeps 87% of its speed at 200 W.

Compute-bound versus memory-bound

A power limit is enforced through the core clock. When the GPU would exceed its limit, it lowers its clock until it doesn't. The question is which workloads depend on that clock.

Prefill is compute-bound. Each weight loaded from memory is applied to thousands of prompt tokens, so the cores are the bottleneck and every cycle does useful work. At 415 W the clock falls by about 25%, and prefill throughput falls by the same amount.

Decode is memory-bound. Each step reads every weight in the model to produce a single token per request, doing only a few operations per byte loaded. The cores finish their work and wait for memory. At 415 W decode's clock also falls by about 25%, but its throughput falls only 4%, because memory bandwidth does not depend on the core clock.

The efficiency gain follows from how chips use power. Dynamic power scales with frequency times voltage squared, and a lower clock allows a lower voltage. Energy per operation therefore falls faster than speed. Decode, which barely slows, keeps nearly all of that saving.

Batch size matters because it changes the ratio of math to memory traffic. More requests in flight means more operations per weight loaded, so decode becomes more compute-bound and more sensitive to the clock.

The sharp drop below 345 W is likely the voltage floor. Once the voltage reaches its minimum, further power savings can only come from the clock, which falls to 345 MHz at 200 W. I didn't measure voltage directly, so this explanation is an inference rather than a result.

What it means for inference power

If power is what limits inference capacity, and it increasingly is, then a GPU's power limit is a tuning parameter rather than a fixed spec. These results suggest setting it by workload. Decode can run well below full power with almost no loss in speed, and the watts it gives back can run more GPUs. Prefill needs the full budget. Serving systems that already split prefill and decode onto separate GPUs could cap only the decode side.

Most work on inference efficiency goes into faster chips and smaller models. This is a gain on hardware that already exists.