L40S vs H100: The Honest Bandwidth Math

TL;DR

• L40S vs H100 in short: the L40S (48 GB GDDR6, 864 GB/s, $1.09/hr on RunPod) is cheaper per hour than the H100 SXM (80 GB HBM3, 3.35 TB/s, $3.99/hr) — but per hour is the wrong unit.
• LLM decode is memory-bandwidth-bound, and on bandwidth-per-dollar the H100 SXM (840 GB/s per $/hr) edges out the L40S (793). The "cheap H100" framing breaks for inference serving.
• The L40S's real win is compute-per-dollar: it delivers roughly 1.36x more FP8 TFLOPS per dollar, so it wins for vision, fine-tuning and batch throughput that fit in 48 GB.
• No NVLink and no meaningful FP64 on the L40S rule it out for multi-GPU large-model training and HPC.

Use the L40S for cost-sensitive inference of models under ~13B that fit in 48 GB, vision workloads, fine-tuning, and media or graphics. Use the H100 for large-model training, long-context or high-batch LLM serving, multi-GPU jobs needing NVLink, and FP64. The L40S is cheaper per hour; the H100 is often cheaper per unit of work that matters.

What's the difference between the L40S and the H100?

They're built for different jobs. The L40S is an Ada Lovelace card with 48 GB of GDDR6 — a universal data-center GPU tuned for graphics, media and compute-bound AI. The H100 is a Hopper data-center accelerator with 80 GB of HBM3, built for the largest training and inference runs.

The gap that decides most AI decisions is memory. The H100 SXM moves data at 3.35 TB/s (NVIDIA H100 datasheet); the L40S moves it at 864 GB/s (NVIDIA L40S datasheet). That's a roughly 3.9x bandwidth advantage for the H100, and bandwidth — not raw TFLOPS — is what governs token generation.

The second gap is interconnect. The H100 has NVLink (900 GB/s on SXM) for pooling GPUs into one fast memory domain. The L40S has no NVLink — multi-GPU traffic falls back to PCIe. If your model needs more than one GPU to train, that's a hard constraint, not a tuning knob.

L40S vs H100 specs

Load-bearing numbers in bold. Tensor figures are the with-sparsity values NVIDIA headlines; dense throughput is roughly half.

Specs from the NVIDIA L40S and H100 datasheets linked above; prices from RunPod's rate card.

For how the L40S stacks up against the previous-generation workhorse, see our L40S vs A100 comparison; for the H100 against its own predecessor and a workstation-class challenger, see A100 vs H100 and RTX Pro 6000 vs H100.

Which is faster for inference?

It depends on whether you're compute-bound or bandwidth-bound — and most LLM serving is bandwidth-bound.

Token generation (decode) reads the entire model and KV cache from memory for every token. Throughput scales with memory bandwidth, so the H100 SXM's 3.35 TB/s gives it close to its 3.9x raw bandwidth edge in real decode. For interactive LLM serving, the H100 is several times faster, not marginally.

Prefill and compute-heavy workloads tell a different story. For batched vision inference, embeddings, or prompt prefill that saturates the tensor cores, the L40S's 1,466 FP8 TFLOPS keeps it competitive — especially once you weigh price. And if a model doesn't fit in 48 GB, the L40S isn't slower, it simply can't run it without quantization or offload.

L40S vs H100 cost: which is actually cheaper?

Per hour, the L40S is 2.7x cheaper ($1.09 vs $3.99). Per unit of work, it depends entirely on what bounds your workload — and the arithmetic overturns the page-one assumption.

Compute the two ratios that matter from the rate card above:

Read those two rows carefully, because they contain the whole decision. On bandwidth-per-dollar — the metric that governs LLM decode — the H100 SXM is actually slightly cheaper than the L40S. The hourly discount is an illusion the moment your workload is memory-bound; you pay less per hour and get less bandwidth per dollar, plus you're capped at 48 GB. For token serving, the H100 is the cheaper machine.

On compute-per-dollar, the L40S wins by about 36%. For a compute-bound job that fits in 48 GB — batched vision, embeddings, LoRA fine-tuning of a small model, media transcode — the L40S does more FLOPs per dollar and is genuinely the cheaper choice.

This is the arithmetic a GPU-rental vendor won't hand you, because every page-one result for this query has a commercial stake in the rental. The neutral read: "cheaper per hour" and "cheaper per token" are different claims, and for the LLM inference most people are googling this for, they point opposite directions.

What this means for you

Inference of small models (<~13B, fits in 48 GB): L40S. You get 1.36x the compute per dollar, FP8 via the Transformer Engine, and the memory headroom is adequate. The clear budget win.

Large-model training: H100. You need the 80 GB, the HBM3 bandwidth, and above all NVLink to shard across GPUs. The L40S can't pool memory over NVLink, so multi-GPU training is bottlenecked on PCIe.

Long-context / high-batch LLM serving: H100. This is bandwidth-bound, where the H100 wins on both raw speed (~3.9x) and bandwidth-per-dollar. Bigger KV caches also need the 80 GB.

Budget-constrained, throughput over latency: L40S. If you can batch work and tolerate lower per-request speed, the compute-per-dollar advantage compounds.

Media, graphics, transcode: L40S, decisively. It has display-oriented hardware (NVENC/NVDEC, RT cores) the H100 lacks; the H100 is a headless compute part.

Anything needing FP64 (HPC, scientific): H100. The L40S has no FP64 Tensor path; it's the wrong tool.

The verdict

The L40S is a real H100 alternative — for the right workload, not as a blanket swap. If your job is compute-bound and fits in 48 GB, it delivers more compute per dollar and is the smart buy. If your job is bandwidth-bound LLM serving or large-model training, the hourly discount evaporates: the H100 SXM is faster and cheaper per unit of bandwidth, with 80 GB and NVLink the L40S can't match. Match the GPU to the bottleneck, not to the sticker price.

Frequently Asked Questions

Is the L40S a good alternative to the H100?

Yes, for compute-bound workloads that fit in its 48 GB — vision, fine-tuning, embeddings, media. At $1.09/hr versus $3.99/hr it offers about 1.36x more FP8 compute per dollar. It's a poor substitute for large-model training or bandwidth-bound LLM serving, where its 864 GB/s and lack of NVLink are hard limits.

L40S vs H100: which is better for AI inference?

For bandwidth-bound LLM token generation, the H100 is better and, surprisingly, cheaper per unit of bandwidth (840 vs 793 GB/s per $/hr). For compute-bound inference of smaller models that fit in 48 GB — vision or batched throughput — the L40S wins on cost. Decode-heavy serving favors the H100; compute-heavy throughput favors the L40S.

How much slower is the L40S than the H100?

For memory-bandwidth-bound LLM decode, expect the L40S to be roughly 3-4x slower, tracking the 864 GB/s versus 3.35 TB/s gap. For compute-bound work that saturates its tensor cores and fits in 48 GB, the gap narrows to roughly 2x on raw FP8 throughput and disappears on a per-dollar basis.

Does the L40S support NVLink?

No. The L40S has no NVLink; multi-GPU communication falls back to PCIe (128 GB/s on Gen5). The H100 SXM supports NVLink at 900 GB/s. If your workload needs several GPUs pooled into one memory domain — large-model training especially — the L40S is the wrong card.

Is the L40S cheaper than the H100?

Per hour, yes — about 2.7x cheaper ($1.09 vs $3.99 on RunPod). Per unit of work, only sometimes. It's cheaper for compute-bound jobs that fit in 48 GB, but for bandwidth-bound LLM serving the H100 SXM is actually slightly cheaper once you divide bandwidth by price.

Editor's note — sources: NVIDIA L40S datasheet (L40S architecture, 48 GB GDDR6, 864 GB/s, 1,466 FP8 TFLOPS sparse, 350 W); NVIDIA H100 datasheet (80 GB, 3.35 TB/s SXM, tensor throughput, NVLink 900 GB/s, 700 W); RunPod pricing (L40S $1.09/hr, H100 PCIe $2.89/hr, H100 SXM $3.99/hr — the basis for all per-dollar arithmetic, fetched 2026-10-26). All linked inline above. Tensor figures are with-sparsity; dense is ~half. Memory types (HBM3/HBM2e/GDDR6) are NVIDIA's established Hopper and Ada specs.