The short answer: the cheapest hourly rate is not always the cheapest way to finish a job. What matters is the cost of each useful GPU-hour, meaning an hour in which the GPU is actually doing the work you are paying for. If a card spends part of every hour waiting for data, a slower step or a stalled process, you pay the full rate for the whole hour. This guide shows how to compare providers on effective cost per useful hour, using hypothetical numbers only. It does not describe any tracked provider's rates or performance.
Billed hours versus useful hours
A GPU-hour is billed whenever the instance is running, whether the GPU is busy or not. Utilisation is the share of that time the GPU spends computing. A simple relationship follows:
- Effective cost per useful hour = hourly rate divided by utilisation.
Hypothetical example, for illustration only. A rate of INR 100 per GPU-hour at 50 percent utilisation has an effective cost of INR 200 per useful hour. A rate of INR 140 at 90 percent utilisation works out to about INR 156. The higher listed rate is cheaper in practice in this scenario. With different utilisation figures the answer reverses, which is why it is worth estimating your own.
Common reasons a GPU sits idle while billing
- Data loading. If training data sits on slow or remote storage, the GPU waits for the next batch. Storage placement and throughput therefore affect cost. Charges for moving data are covered in the egress and data-transfer guide.
- CPU or memory limits. A small number of CPU cores or little system memory can slow preprocessing so the GPU is starved. Check the CPU and RAM that come with each GPU, not just the GPU itself.
- Setup and warm-up. Driver installs, image pulls and model downloads are billed time with no useful output. See the setup time guide.
- Interconnect waits on multi-GPU jobs. When GPUs exchange data between steps, a slower link leaves cards waiting. The multi-GPU networking guide covers what a node rate does and does not include.
- Idle gaps between runs. An instance left running overnight between experiments is billed, as explained in the idle and stopped instance guide.
How to estimate utilisation without guessing
- Run a short pilot. A trial of a few minutes on your real workload gives a better figure than a vendor benchmark, because your data pipeline and batch size are part of the result.
- Watch the GPU, not the job log. Standard monitoring tools show how busy the GPU is over time. Persistent low readings usually point to the input pipeline rather than the card.
- Record the time to finish, not just the speed. Measure how long the same job takes on each candidate, then multiply by the rate. Total cost to completion is the number to compare.
- Repeat on each memory tier. A larger card may allow bigger batches and higher utilisation, as discussed in the VRAM sizing guide.
A worked comparison
Hypothetical example, for illustration only. A job needs 100 useful GPU-hours. Provider A lists INR 90 per hour and your pilot shows 60 percent utilisation because of slower storage. Provider B lists INR 120 per hour and the pilot shows 85 percent.
- Provider A: 100 divided by 0.60 is about 167 billed hours, so about INR 15,000.
- Provider B: 100 divided by 0.85 is about 118 billed hours, so about INR 14,118.
Here the higher listed rate costs slightly less overall and also finishes sooner. Change the utilisation figures and the order flips. Billing granularity matters as well: with hourly rounding, short runs can lose more time than the formula shows, as explained in the billing granularity guide. Any GST is added on top for both options and does not change the ranking, since the GST invoicing guide applies to either.
Frequently asked questions
Is a higher utilisation always better? It lowers the cost per useful hour, but pushing it requires engineering time. If tuning takes longer than the savings justify, a plain run on a cheaper rate may still win.
Can I compare providers from listings alone? Listings give a rate, not utilisation. Use the compare tool to shortlist by live INR rate, then use a pilot to estimate effective cost. The cost calculator can take the billed hours you calculate.
All figures in the examples are hypothetical. Utilisation depends on your workload, data layout and configuration, so measure it before deciding.