Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You should write an agent

fly.io

Post reply on HN
Print view View original

171–180 of 409 posts

Re: You should write an agent

#171

Post by roadside_picnic »

furyofantares wrote: post #134
roadside_picnic wrote: post #124

Earlier quoted context omitted.

So you're assuming there's a world where these companies exist solely by providing inference? The first obvious limitation of this would be that all models would be frozen in time. These companies are operating at an insane loss and a major part of that loss is required to continue existing. It's not realistic to imagine that there is an "inference" only future for these large AI companies. And again, there are many…

> So you're assuming there's a world where these companies exist solely by providing inference? Yes, obviously? There is no world where the models and hardware just vanish.

> and hardware just vanish.

Okay, this tells me you really don't understand model serving or any of the details of infrastructure. The hardware is incredibly ephemeral. Your home GPU might last a few years (and I'm starting to doubt that you've even trained a model at home), but these GPUs have incredibly short lifespans under load for production use.

Even if you're not working on the back end of these models, you should be well aware that one of the biggest concerns about all this investment is how limited the lifetime of GPUs is. It's not just about being "outdated" by superior technology, GPUs are relatively fragile hardware and don't last too long under constant load.

As far as models go, I have a hard time imagining a world in 2030 where the model replies "sorry, my cutoff date was 2026" and people have no problem with this.

Also, you still didn't address my point that startups doing inference only model serving are burning cash. Production inference is not the same as running inference locally where you can wait a few minutes for the result. I'm starting to wonder if you've ever even deployed a model of any size to production.

Re: You should write an agent

#172

Post by khimaros »

simonw wrote: post #143
hoppp wrote: post #105

Earlier quoted context omitted.

When comparing the cost of an H100 GPU per hour and calculating cost of tokens, it seems the OpenAI offering for the latest model is 5 times cheaper than renting the hardware. OpenAI balance sheet also shows an $11 billion loss . I can't see any profit on anything they create. The product is good but it relies on investors fueling the AI bubble.

> When comparing the cost of an H100 GPU per hour and calculating cost of tokens, it seems the OpenAI offering for the latest model is 5 times cheaper than renting the hardware. How did you come to that conclusion? That would be a very notable result if it did turn out OpenAI were selling tokens for 5x the cost it took to serve them.

it seems to me they are saying the opposite

Re: You should write an agent

#173

Post by awayto »

simonw wrote: post #139
hoppp wrote: post #122

Earlier quoted context omitted.

Agents use Apis that I will need to pay for and generally software dev is a job for me that needs to generate income. If the Apis I call are not profitable for the provider then they won't be for me either. This post is a fly.io advertisement

"Agents use Apis that I will need to pay for" Not if you run them against local models, which are free to download and free to run. The Qwen 3 4B models only need a couple of GBs of available RAM and will run happily on CPU as opposed to GPU. Cost isn't a reason not to explore this stuff.

Google has what I would call a generous free tier, even including Gemini 2.5 Pro (https://ai.google.dev/gemini-api/docs/rate-limits). Just get an API key from AiStudio. Also very easy to just make a switch in your agent so that if you hit up against a rate limit for one model, re-request the query with the next model. With Pro/Flash/Flash-Lite and their previews, you've got 2500+ free requests per day.

Re: You should write an agent

#174

Post by necovek »

simonw wrote: post #143
hoppp wrote: post #105

Earlier quoted context omitted.

When comparing the cost of an H100 GPU per hour and calculating cost of tokens, it seems the OpenAI offering for the latest model is 5 times cheaper than renting the hardware. OpenAI balance sheet also shows an $11 billion loss . I can't see any profit on anything they create. The product is good but it relies on investors fueling the AI bubble.

> When comparing the cost of an H100 GPU per hour and calculating cost of tokens, it seems the OpenAI offering for the latest model is 5 times cheaper than renting the hardware. How did you come to that conclusion? That would be a very notable result if it did turn out OpenAI were selling tokens for 5x the cost it took to serve them.

I am reading it as OpenAI selling them for 20% of the cost to serve them (serving at the equivalent token/s with cloud pay-per-use GPUs).

Re: You should write an agent

#175

Post by roadside_picnic »

vel0city wrote: post #145
roadside_picnic wrote: post #124

Earlier quoted context omitted.

So you're assuming there's a world where these companies exist solely by providing inference? The first obvious limitation of this would be that all models would be frozen in time. These companies are operating at an insane loss and a major part of that loss is required to continue existing. It's not realistic to imagine that there is an "inference" only future for these large AI companies. And again, there are many…

The models may be somewhat frozen in time but with the right tools available to it they don't need all information innately coded into it. If they're able to query for reliable information to drag in they can talk about things that are well outside their original training data.

For a few months of news this works, but over the span of years even the statistical nature of language drifts a bit. Have you shipped natural language models to production? Even simple classifiers need to be updated periodically because of drift. There is no world where you lead the industry serving LLMs and don't train them as well.

Re: You should write an agent

#176

Post by hoppp »

johnfn wrote: post #168
hoppp wrote: post #157

Earlier quoted context omitted.

The submission is an advertisement for fly.io and OpenAI , both are paid services. We are commenting on an ad. The person who wrote it did it for money. Fly.io operates for money, OpenAi charges for their API. They posted it here expecting to find customers. This is a sales pitch. At this point why is it an issue to expect a developer to make money on it? As a dev, If the chain of monetization ends with me then there…

Why are you repeatedly stating that the post is an ad as if it is some sort of dunk? Companies have blogs. Tech blogs often produce useful content. It is possible that an ad can both successfully promote the company and be useful to engineers. I find the Fly blog to be particularly well-written and thoughtful; it's taught me a good deal about Wireguard, for instance.

And that sounds fine, but Wireguard is not an overhyped industry promising huge gains in the future to investors and to developers jumping on a bandwagon who can find problems for this solution.

I actually have built agents already in the past and this is my opinion. If you read the article the author says they want to hear the reasoning for disliking it, so this is mine, the only way to create a business is raising money and hoping somebody strikes gold with the shovel Im paying for.

Re: You should write an agent

#178

Post by khimaros »

ilikehurdles wrote: post #127
paulcole wrote: post #117

Earlier quoted context omitted.

I love how programmers generally tout themselves as these tinkerers who love learning about and exploring technology… until it comes to AI and then it’s like “show me the profitable use case.” Just say you don’t like AI!

I don't think hn is reflective of where programmers are today, culturally. 10 years ago, sure, it probably was.

what place is more reflective today?

Re: You should write an agent

#179

Post by yeasku »

DANmode wrote: post #177
hoppp wrote: post #93

I should? what problems can I solve, that can be only done with an agent? As long as every AI provider is operating at a loss starting a sustainably monetizable project doesn't feel that realistic.

[flagged]

What is the point of your comment?

Lonely developer?

Re: You should write an agent

#180

Post by simonw »

hoppp wrote: post #176
johnfn wrote: post #168

Earlier quoted context omitted.

Why are you repeatedly stating that the post is an ad as if it is some sort of dunk? Companies have blogs. Tech blogs often produce useful content. It is possible that an ad can both successfully promote the company and be useful to engineers. I find the Fly blog to be particularly well-written and thoughtful; it's taught me a good deal about Wireguard, for instance.

And that sounds fine, but Wireguard is not an overhyped industry promising huge gains in the future to investors and to developers jumping on a bandwagon who can find problems for this solution. I actually have built agents already in the past and this is my opinion. If you read the article the author says they want to hear the reasoning for disliking it, so this is mine, the only way to create a business is raising…

How would you feel about this post if the exact same content was posted on a developer's personal blog instead?

I ask because it's rare for a post on a corporate blog to also make sense outside of the context of that company, but this one does.

Post reply on HN