491–492 of 492 posts
Earlier quoted context omitted.
I’m not a fan of the 20 tool limit unless you use the Max option which costs you 1 credit for each and every tool call + message. Seems like an artificial limit and it always rips me out
You can just click continue and keep burning tokens. It's not such a big deal.
Earlier quoted context omitted.
That works in systems that exhibit economy of scale. The problem with generative ai workloads: The costs rise linerly with the number of requests, because you need to compute every query.
GPU type and utilization mean that the costs likely rise only logarithmically or sub-linear. If you commit to buying enough inference over long enough, someone can buy a rack of the newest custom inference chips and run them at 100% for you, which may be a lot cheaper per request than doing them on a cpu somewhere.