AMD Instinct MI300X: what it costs to rent and whether your code runs

The memory in AMD Instinct MI300X computing device is 192 GB per GPU. On the other hand, NVIDIA H100 has 80 GB.

AMD Instinct MI300X is available to lease on Microsoft Azure and Oracle Cloud. On the contrary, AWS is not offering it for rent.

Oracle has a set price of $6.00 per GPU hour. Other competitive prices exist, with the lowest price requiring a four-year contract.

Your PyTorch and vLLM applications would most likely work. However, this article explains all of these things in detail.

Key takeaways

• The memory of an AMD Instinct MI300X is 192 GB of HBM3, which is 2.4 times the memory of an H100 SXM.
• It is being used by Oracle Cloud and Azure presently. However, AWS publishes no AMD Instinct instance type at all.
• The price of Oracle for this instance is $6.00. The price of Vultr is $2.290 but it is only for 48 months contract.
• The MI300X is supported by ROCm version 10.1.0 and all install paths are available for JAX, PyTorch, vLLM and SGLang.
• While AMD does not publish independent benchmarks, the information about benchmarks has been gathered from the vendors.

What the AMD Instinct MI300X actually is

AMD’s data sheet describes a single OAM module. Eight of them sit on a UBB 2.0 board to make a platform.

Here is what one accelerator holds.

Above it lie two updated components. The MI325X offers a maximum capacity of 256GB of HBM3E at 6 TB/s with power consumption of 1000W. The MI355X offers a maximum capacity of 288GB of HBM3E at 8 TB/s and a power rating of 1400W.

All three components make use of the same eight-GPU system. Our article explaining why servers are made in packs of eight also provides explanation of how baseboard (UBB 2.0). are related to that issue.

A report is incomplete, however. AMD provides performance numbers for AI with FP16, FP8, FP6 and FP4, but it becomes difficult to follow them in all three AMD reports, as they mention different figures without naming them clearly.

As far as FP4 and FP6 figures are concerned, we failed to find any correct information regarding them in the article.

Why the memory number matters more than the FLOPS

Memory is what determines whether the model will run or not. Computing capacity determines the speed of the model after loading.

The way in which you decide on your purchase matters because buying a faster graphics processing unit but not being able to run your model will be useless.

A model with 70 billion parameters will require around 140 GB just for weights at FP16. This can fit on one MI300X while still leaving some space but will be impossible to fit on one 80 GB H100.

This leads to less graphics processing units to operate with. The fewer graphics processing units the less cross-GPU communication needed and the easier the job becomes. Our guide to how many GPUs you need works through the sizing.

Having a smaller number of graphics processing units will also lead to lower costs for processing. IBM claimed the same thing when it showed its own offering from AMD. It pointed out to the potential of needing less graphics processing units for large models which would consequently lower inference costs.

This is the actual argument for AMD in claiming the issue of capacity rather than speed.

Which clouds actually carry it

This is the question nobody answers, so here is the map.

Azure has revealed all specifications. A Standard_ND96isr_MI300X_v5 comes equipped with 8 MI300X GPUs of 96 vCPUs and 1,850 GiB of RAM. The GPUs are supplied with the capability of separate InfiniBand of 400 Gbps providing 3.2 Tb/s for the VM.

Oracle has provided BM.GPU.MI300X.8, which is equipped with 8 GPUs summing up to 1.5 TB of HBM3. All cluster networks are powered by 8 connections of 400G.

IBM has made a bold announcement, but the actual release is yet to occur. Check its status before making any plans.

AWS and Google Cloud sadly do not include AMD Instinct in their instance types. Let this fact speak for itself rather than attribute meaning.

If the hyperscaler choice hasn’t been made, then our comparison of hyperscalers in the field of AI makes sense.

What it costs, and what the cheap rate hides

Two published numbers tell the whole story.

Oracle’s charge is 2.6 times that of Vultr, but it is not about the type of equipment used, but about one’s commitment.

In order to get the lower price Vultr offers, you have to sign a contract for four years. Having in mind the 730 hours per month for 48 months, this is a lot of expenses before you start to earn anything.

Vultr states: “Reserve AMD MI355X GPUs now,” meaning that they are not generally available in stock. TensorWave charges $2.95 per GPU-hour for MI355X without specifying the contract duration.

To do the math regarding Vultr’s contract, we need to do the following calculations: at $2.290 per hour for 730 hours a month, one single GPU would cost $1,671.70 per month.

Over four years, such an amount totals to $80,241.60 for one GPU. The 8-GPU cluster would charge about $641,900 within this period.

With Oracle’s $6.00 per hour, it equals to $4,380 a month without any required contracts being signed. The cheaper option seems to be more favorable only if you are going to use the service for the entire time specified in the contract.

This is the area of specialty of Vultr. For low prices, the company requires you to make a commitment. When embarking on a project, many begin with uncertainty.

There are around 10 different websites that propose comparisons of different brands in terms of AMD GPU pricing. The prices they present cannot be deemed fake. The terms behind those rates mostly sit elsewhere.

A headline number should never end your research. Our piece on what a GPU-hour measures covers why rates mislead in general.

For context on the other side, see current H100 rates by provider.

AMD Instinct MI300X published rates against the contract terms behind them.

Will your code run on ROCm

The software system developed by AMD is called ROCm. It performs the same tasks as CUDA does on NVIDIA chips.

Most people think that ROCm is not ready for use. But that is not what AMD says in its latest publications.

Most of the work involved in both training and inference is handled via PyTorch and vLLM. Both methods have an installed version from AMD instead of a community port.

Having said that, a few caveats come into play.

That matrix does not give any version number of implemented frameworks, so before applying any changes, make sure to cross-reference the installed versions with AMD’s pages.

Both TensorFlow and Triton are absent from the Instinct compatibility table, so you must verify their compatibility.

Instinct is Linux only, with the listing of Windows versions nowhere in sight. Only Radeon and Ryzen have the information on Windows or WSL2.

Custom CUDA kernels are what poses real danger in this situation. While changes in code written in that programming language are easily made, it has to be re-written from scratch usually if done manually.

What to ask before you commit

Five questions turn a published rate into a decision.

1. What is the price in on demand mode, without any contracts involved?
2. What is the shortest duration of the discounted rate, and what are the implications of cancelling it before the completion of that duration?
3. Is it possible to acquire any hardware currently, without preorders?
4. What is the ROCm version equipped in the image, and is it compatible with the version we use?
5. Is it possible to benchmark our project workload prior to signing any contracts?

Ask the last one first. A provider willing to let you test is telling you something the price list cannot.

Capacity may matter more than price anyway, which our guide to getting GPU capacity covers.

MI300X against H100 and B200

Compare on memory, where the data is clean.

The figure B200 is calculated based on dividing NVIDIAs state 1.4 TB of a single eight-GPU node.

While capacity MI300X is 2.4 times greater than that of H100, MI355X in terms of capacity is 1.6 times higher than B200.

We refrained from making compute comparisons between AMD and NVIDIA. The reason for this is our inability to validate, precision-wise, AMDs table of AI performance. By mixing an unverified value into a verified one, we would be misleading all of you.

For the NVIDIA side on its own terms, see B200 against H100.

When AMD makes sense, and when it does not

Consider AMD when your problem is memory. A model that needs two H100s may fit on one MI300X. Memory-bound inference with long context benefits for the same reason.

Consider AMD when supply is tight. Demand for NVIDIA parts stays high. An alternative with available capacity carries value beyond its specification.

Consider AMD when you run framework-level code. PyTorch and vLLM users carry far less porting risk than teams with custom kernels.

Skip AMD when your stack is CUDA-locked. Hand-written kernels, CUDA-only libraries and a tuned NVIDIA pipeline all make the move expensive.

Skip AMD when AWS already holds your commitment. No AMD Instinct instance type exists there, so the question does not arise.

Skip AMD when you cannot test first. Every performance claim in this article comes from a vendor. No independent benchmark exists to lean on, so benchmark your own workload before signing anything.

AMD’s own results show the demand is real, as our coverage of AMD’s data centre growth describes. Demand for the company’s shares is not evidence that the hardware suits your job.

FAQs

Conclusion

Today, it is possible to rent the AMD Instinct MI300X, which has 2.4 times more memory in comparison to the H100 and supports the popular Python AI software stack utilizing ROCm.

However, What it lacks is an independent benchmark, an AWS instance type, and a cheap rate without a long contract.

To conduct your analysis, consider pricing a single node on Oracle at $6.00 per GPU-hour while running your own workload on it. In doing so, evaluate and compare the cost of the execution of identical tasks using NVIDIA hardware.

This test would cost much less than the four-year commitment to the technology.