Fordel Studios
AMD keeps saying ROCm is ready. NVIDIA keeps pretending ROCm does not exist. The truth, as usual, is somewhere in between — and it depends entirely on what you are building.
What is ROCm and why does it matter now?
ROCm (Radeon Open Compute) is AMD's open-source GPU computing platform — their answer to NVIDIA's CUDA. It has existed since 2016, but for most of that decade it was the GPU equivalent of Linux on the desktop: technically possible, practically painful.
2026 changed the math. AMD's MI300X chips are shipping in volume. Hyperscalers are buying them. PyTorch 2.6 treats ROCm as a first-class target. And with NVIDIA H100s still on 16-week lead times at list price, teams that ignored AMD are suddenly running cost comparisons.
The catalyst this week: AMD's AI director publicly criticised Claude Code's performance on ROCm — not to trash Anthropic, but to pressure the ecosystem into treating AMD GPUs as equal citizens. That kind of public pressure only happens when a company believes its hardware is ready and the software is the bottleneck.
What does CUDA still do better?
CUDA is not just a compiler. It is an ecosystem. cuDNN, cuBLAS, TensorRT, Triton (NVIDIA's, not the inference server), NCCL — these libraries have a decade of optimisation behind them. When you pip install torch and it works on your A100, you are standing on thousands of person-years of integration work.
The practical impact: if you are using a framework or model architecture that is well-trodden — transformer training, standard fine-tuning, inference with vLLM or TensorRT-LLM — CUDA just works. No porting. No debugging HIP translation layers. No wondering if that random CUDA kernel your dependency uses has a ROCm equivalent.
How far has ROCm actually come?
Genuinely far. Here is what works well in ROCm 6.x as of April 2026:
- PyTorch training and inference — near-parity performance on MI300X vs H100 for standard workloads
- JAX support — functional, though less battle-tested than PyTorch path
- Flash Attention 2 — native ROCm implementation, no longer requires the composable kernel workaround
- vLLM inference — AMD contributed directly, works on MI300X out of the box
- Docker and container tooling — ROCm containers are stable, no more driver version roulette
- Multi-GPU training — RCCL (AMD's NCCL equivalent) handles standard distributed workloads
The MI300X's 192GB HBM3 is its killer feature. When you are serving a 70B parameter model, not splitting across GPUs matters. That memory advantage alone justifies the switch for specific inference workloads.
How do they compare head to head?
Where does ROCm still fall short?
The gaps are specific but they matter.
Custom CUDA kernels are the biggest pain point. If your ML pipeline uses a library that ships hand-written CUDA kernels — and many do — you are dependent on either HIP translation (which handles maybe 90% of cases automatically) or the library maintainer caring about AMD. Many do not.
Debugging tooling is thinner. NVIDIA's Nsight suite is genuinely excellent. AMD's rocprof works, but when you are tracking down a subtle numerical divergence in a distributed training run, the tooling gap costs real hours.
The long tail of frameworks is the other issue. If you are using anything beyond PyTorch and JAX — say, PaddlePaddle, or a niche research framework — ROCm support ranges from "community effort" to "nonexistent."
“ROCm is ready for the 80% of workloads that run on standard PyTorch. The question is whether your workload is in that 80%.”
Who should pick ROCm?
- You are running standard PyTorch training or fine-tuning and want 25-35% lower GPU costs
- You need to serve large models (70B+) and the 192GB HBM3 on MI300X eliminates multi-GPU complexity
- You are building on vLLM for inference and want the cost advantage without the ecosystem risk
- You are a startup that cannot get H100 allocation at reasonable prices
- Your team has the engineering depth to debug occasional HIP translation issues
- You use custom CUDA kernels or depend on libraries that ship them
- You need TensorRT-LLM for latency-critical inference (no ROCm equivalent)
- Your team is small and cannot afford time debugging platform issues
- You are using TensorFlow as your primary framework
- You need the broadest possible cloud provider selection and spot instance availability
- You are in a regulated environment where GPU vendor switching requires revalidation
What is the real verdict for production teams?
The honest answer: ROCm in 2026 is where Linux was around 2008. It works. It is cheaper. It is improving fast. And it still requires more engineering effort than the incumbent for anything beyond the happy path.
If you are starting a new AI infrastructure build today and your workloads are standard transformer training plus vLLM inference, ROCm on MI300X is a legitimate choice that will save you money. The 192GB memory alone changes the serving economics for large models.
If you are running production workloads that depend on the CUDA ecosystem's depth — TensorRT optimisation, Triton Inference Server with custom backends, complex multi-node training with NCCL tuning — switching to ROCm is a project, not a configuration change. Budget three to six months of engineering time for the migration, and make sure the cost savings justify it.
The smart play for most teams: run your training experiments on ROCm to validate compatibility, keep CUDA as your production path, and watch the MI350 launch later this year. AMD is closing the gap faster than NVIDIA is cutting prices. The crossover point is approaching — it is just not here for everyone yet.
