361–370 of 495 posts
Earlier quoted context omitted.
I have the same experience. I literally cannot fathom how people burn the number of tokens they claim to.
Working on multiple things at the same time. If you've got 4-5 instances running simultaneously you'll burn 4-5 times as many tokens.
Earlier quoted context omitted.
It really depends on what you're doing, but most LLM usage and agentic runs are pretty interchangeable in my experience, and it's usually trivial to switch. If anything, you're better off supporting multiple LLMs as backup because most model providers have been so inconsistent with working all the time
Dude it’s not trivial to switch because the behaviors are different! You’re clearly not building a product based on an LLM. I’m still using various old Anthropic and OpenAI models for products I’ve built and released because I can’t risk the behavior changing in unpredictable ways and the users being pissed. It’s much easier to switch out some deterministic software than an LLM which you’ve spent a ton of time on tes…
I'm not convinced raw costs matter: 1. Compute costs collapsed since the advent of Cloud and yet hyperscalers still have fat margins. 2. Many open source office suites exist yet none compete with the ubiquity of gsuite or office. GitHub, Slack are similar examples. 3. Both Windows and macOS dominate the home desktop space despite free alternatives existing for a long time. 4. Many formerly open source infrastructure…
There's a huge case of survivorship bias when trying to recall historical analogues, because in every instance where margins collapsed and competition made the industry a commodity business, the big proprietary names are no longer with us. Here's a selection of examples, though: 1. Memory chip margins collapsed so much in the 80s that Intel exited the memory chip business entirely. At the time, they were known much m…
We are seeing the later start to unravel.
I also have my fork of metamcp that replaces firebase MCP spec with my own that tells the model to use crawl4ai and SearXNG instead.
I've been using this wia Librechat with every commercial and open weight model I tested.
The search is way better than OpenAI and what ClaudeCode uses, but Gemini is way faster. That will change soon as I'm planning to put these instances in a DC with gigabit pipe.
Firebase is not cheap, but it retrieves everything, bypasses captchas and so on.... As long as one uses it for 1% of Web queries the cost is manageable.
Meanwhile: > China’s Ministry of Commerce has led meetings over the past month with major AI companies, including Alibaba, ByteDance, and http://z.ai/ , to discuss measures that would restrict overseas access to cutting-edge AI models, including models that have not yet been released. > The discussions reportedly include not only closed-source models but also open-weight models. > Future regulations could take the fo…
Yep https://news.ycombinator.com/item?id=48816025 And EU leadership completely destroyed Europe's future by betting on depending on US and Chinese models. https://pleias.ai/blog/fable-eu
You could just as well read the european approach as a bet that frontier models will be unable to keep a significant edge over open competition (and thus not worth throwing subsidies at, because any economic advantage is fleeting at best).
Looking at the data and related past experience, this looks like a pretty solid bet (despite the "risk" being hard to quantify).
> the least understood upcoming shift in AI economics. Then proceeds to talk about something in the AI news every day. Hey, did you guys hear? Open source models are cheaper and their quality is increasing! So, first, by no measure is GLM5.2 as good as Opus. Second, yes, open source models will put pressure on margins...eventually. Everyone knows that. But do you think today's AI business model is the same as tomorro…
re: "So, first, by no measure is GLM5.2 as good as Opus." I accept that for you and your work this is true. I have a different experience: for a month I paid big money for Opus and got a lot done. Now I am gorging on GLM 5.2 running on Fireworks.ai and I am also getting a lot done for about 15% of the money. Everyone should do their own evals on their own work.
I have Max x5 for 120Eur a month. I use it a lot (but usually I don't multitask). I almost never hit the limits.
With GLM5.2 paying $4 per mln tokens I would be burning at least $20-$30 a day.
Earlier quoted context omitted.
The current top comment in https://lobste.rs/s/ua1gxl/glm_5_2_coming_ai_margin_collapse correctly zoomed into cached input tokens, but landed on the opposite conclusion: > That is, for your $100/month fee, you get $3600 equivalent of API usage. This is presumably because Anthropic has figured out some clever things to do with model routing and input caching, and also can subsidize with investor money and take a hit o…
While we are all speculating, Boris kindly provided some guidance in https://news.ycombinator.com/item?id=47880089 > The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for users. In an extreme case, if you had 900k tokens in your context window, then idled for…
So it seems they do care.
I'm not convinced raw costs matter: 1. Compute costs collapsed since the advent of Cloud and yet hyperscalers still have fat margins. 2. Many open source office suites exist yet none compete with the ubiquity of gsuite or office. GitHub, Slack are similar examples. 3. Both Windows and macOS dominate the home desktop space despite free alternatives existing for a long time. 4. Many formerly open source infrastructure…
There's a huge case of survivorship bias when trying to recall historical analogues, because in every instance where margins collapsed and competition made the industry a commodity business, the big proprietary names are no longer with us. Here's a selection of examples, though: 1. Memory chip margins collapsed so much in the 80s that Intel exited the memory chip business entirely. At the time, they were known much m…