>This largely depends on whether you own or rent your hardware. At $40,000 per B200, your lifetime cost per user is 40_000/num_users.
In the 100% duty cycle case (worst for cost), that's 6k$ per user. Realistically, serving 300 users per GPU you'll spend a lifetime cost of about $133 per user, plus the datacenter/upkeep bill.
If you rent the GPU, the cost is more straightforward. At an hourly rate of $43, your hourly cost per user is 4/num_users. For num_users=300 you get an hourly rate of about $0.013 per user, or $9.36 per month.
This leads me to believe you can buy a GPU but leave it at a data center?
Do people do this? I don't understand. Or are you equating upkeep bill to electricity on premises?
> Realistically, serving 300 users per GPU you'll spend a lifetime cost of about $133 per user, plus the datacenter/upkeep bill.
What is the operational cost and when does it become more expensive than the upfront capex?
The B200 tops out at 1000W and idles around 140W. It averages around 600W. https://www.lightly.ai/blog/nvidia-b200-vs-h100 U.S. average electricity cost is $.14 per kWh in March. https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...
600/1000 *.14 =$0.084 per hour $2.01 per day $60.30 per month With 300 users, $.20 per user per month. Seems fairly cheap for the electricity.
Does anyone know how to estimate colo/data center rent costs? Where did I screw up my estimates?
>This largely depends on whether you own or rent your hardware. At $40,000 per B200, your lifetime cost per user is 40_000/num_users. In the 100% duty cycle case (worst for cost), that's 6k$ per user. Realistically, serving 300 users per GPU you'll spend a lifetime cost of about $133 per user, plus the datacenter/upkeep bill. If you rent the GPU, the cost is more straightforward. At an hourly rate of $43, your hourly cost per user is 4/num_users. For num_users=300 you get an hourly rate of about $0.013 per user, or $9.36 per month.
This leads me to believe you can buy a GPU but leave it at a data center?
Do people do this? I don't understand. Or are you equating upkeep bill to electricity on premises?
You can, people do. https://www.linkedin.com/posts/activity-7409593739138060288-...
So what's the cost separating them from placing this box at their premise?
Network throughout?
Plus power and cooling.
I'd like to see a bit of the running costs inside the napkin math. Power, cooling, maintenance, rent, etc. are probably significant factors as well.