Just signed up for Claude Pro to try out the Opus model. Decided to throw a complex query at it, combining an image with an involved question about SDXL fine tuning and asking it to do some math comparing the cost of using an RTX 6000 Ada vs an H100.
It made a lot of mistakes. I provided it with a screenshot of Runpod's pricing for their GPUs, and it misread the pricing on an RTX 6000 ADA as $0.114 instead of $1.14.
Then, it tried to do math, and here is the outcome:
-----
>Approach 1: Use the 1x RTX 6000 Ada with a batch size of 4 for 10,000 steps.
>Cost: $0.114/hr * (10,000 steps / (4 images/step * 2.5 steps/sec)) = $19.00
Time: (10,000 steps / (4 images/step * 2.5 steps/sec)) / 3600 = 0.278 hours
>Approach 2: Use the 1x H100 80GB SXMS with a batch size of 8 for 10,000 steps.
>Cost: $4.69/hr * (10,000 steps / (8 images/step * 3 steps/sec)) = $19.54
Time: (10,000 steps / (8 images/step * 3 steps/sec)) / 3600 = 0.116 hours
-----
You will note that .278 * $0.114 (or even the actually correct $1.14) != $19.00, and that .116 * $4.69 != $19.54.
For what it's worth, ChatGPT 4 correctly read the prices off the same screenshot, and did math that was more coherent. Note, it saw that the RTX 6000 Ada was currently unavailable in that same screenshot and on its own decided to substitute a 4090 which is $.74/hr, also it chose the cheaper PCIe version of the H100 Runpod offers @ $3.89/hr:
-----
>The total cost for running 10,000 steps on the RTX 4090 would be approximately $2.06.
>It would take about 2.78 hours to complete 10,000 steps on the RTX 4090.
On the other hand:
>The total cost for running 10,000 steps on the H100 PCIe would be approximately $5.40.
>It would take about 1.39 hours to complete 10,000 steps on the H100 PCIe, which is roughly half the time compared to the RTX 4090 due to the doubled batch size assumption.
-----