Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is the industry ready for tokens-constrained work?

blog.alaindichiappari.dev

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Is the industry ready for tokens-constrained work?

#41

Post by jtfrench »

deadbabe wrote: post #21
alainrk wrote: post #16

Earlier quoted context omitted.

Right, every technology is suboptimal in the beginning, but constraints will ever exists, energy and hardware in primis. Aldo you have to factor entropy in regardless, complexity remains a cost

Imagine if people had said one day computer programs had to be more memory constrained because everything uses too much RAM. Madness. We'd never see 128GB of RAM.

This is actually how Jeff Dean, Chief Scientist at Google recommends doing thought exercises to challenge assumptions (watch is recent YX Startup School talk). The end goal isn’t “128GB of RAM”, the goal is “useful effective software”. If for whatever reason RAM was an immovable constraint, a different tech tree would emerge.

Re: Is the industry ready for tokens-constrained work?

#43

Post by verbify »

sokoloff wrote: post #9

Rationing tokens on a spend level or count makes no sense to me. Why would I pay $10-20K/mo to employ a software engineer and then balk at a $500, $1000, or even $2000 monthly AI bill, assuming they were even vaguely trying to use the tokens productively? I’m not an AI-maximalist, but “work a few days with AI and the rest of the month without because of cost” sounds literally crazy to me. (If you think AI is low/zero…

> Why would I pay $10-20K/mo to employ a software engineer and then balk at a $500, $1000, or even $2000 monthly AI bill, assuming they were even vaguely trying to use the tokens productively?

Spending $10k and then balking at an additional $2k can make sense to me based on the circumstances. It's an additional 20% that hasn't been budgeted for.

The question is partly about what that 20% gives the employer - for some businesses, you might see diminishing returns from additional productivity of a software developer. E.g. I did some work creating BI reports and warehousing for a manufacturing company. Being more productive just meant faster reports or more reports. But faster/more reports didn't necessarily equate to more money for the business.

Sometimes it's a question of a business being penny wise and pound foolish, but sometimes it does make sense to me to not spend this additional money.

Re: Is the industry ready for tokens-constrained work?

#44

Post by [deleted] »

deadbabe wrote: post #12

I don't think people understand, tokens must never be constrained. Right now, LLMs are the worst they'll ever be, but imagine what they will be like at the peak: Anything you want to code, coded instantly. Not waiting for code to stream in or wait hours for some agents to crunch through loops and planned: it just appears on the screen instantly like the way a webpage loads. Then imagine it can be done locally, on you…

[deleted]

Re: Is the industry ready for tokens-constrained work?

#45

Post by gibolt »

abuani wrote: post #27
sokoloff wrote: post #9

Rationing tokens on a spend level or count makes no sense to me. Why would I pay $10-20K/mo to employ a software engineer and then balk at a $500, $1000, or even $2000 monthly AI bill, assuming they were even vaguely trying to use the tokens productively? I’m not an AI-maximalist, but “work a few days with AI and the rest of the month without because of cost” sounds literally crazy to me. (If you think AI is low/zero…

What about a $10k to $20k bill? Or better, 100%-200% of the employees salary? At a certain point in time, finance is going to care greatly about the ability to predict costs. Pre AI, predicting the cost an engineer was quite easy. Salary + benefits + licenses for software. CI systems that were unbounded were rolled up into opex and treated as a separate line item. Where does token spend land in this now? Finance team…

Set a limit, train people to use tokens efficiently. For anyone getting high value from it, raise their limit.

Re: Is the industry ready for tokens-constrained work?

#46

Post by ygjb »

the__alchemist wrote: post #17
cebert wrote: post #2

I have similar concerns about this. The past year or so, software engineers have been encouraged by employers to adopt AI tooling and agentic coding. Now, many enterprises, including mine, are starting to crack down on token spend. I’ve learned how to fully embrace agentic tooling to do tasks like keeping up with vulnerability reports, initially triaging defects that come in, etc. It would be hard for me to do things…

Maybe a dumb question, but what are you doing/seeing others do that uses that many tokens? I find myself pressing the weekly caps on the $20/month plan only when I'm really having the LLMs go wild with the Xtra high effort on the newest/biggest models on abstract problems or where I don't really understand the problem well. (E.g. improving ML models, planning creative molecule synthesis pipelines, identifying subtle…

I routinely hit the limits on a Claude pro max 20 account for personal projects. For context I have 26 years of professional experience and have been programming for around 40 years.

In addition I have virtually unconstrained usage at work (for now).

Tasks that I do on my personal account: - building stuff for my wife's business - analyzing and experimentation and building harnesses to test other models/services - processing interesting research papers that publish without code or sufficient data to replicate the work

And that's besides simple stuff like exploring new topics I am interested in, and building learning assistants and personal tooling, and eliminating technical chores.

Re: Is the industry ready for tokens-constrained work?

#48

Post by delusional »

sokoloff wrote: post #9

Rationing tokens on a spend level or count makes no sense to me. Why would I pay $10-20K/mo to employ a software engineer and then balk at a $500, $1000, or even $2000 monthly AI bill, assuming they were even vaguely trying to use the tokens productively? I’m not an AI-maximalist, but “work a few days with AI and the rest of the month without because of cost” sounds literally crazy to me. (If you think AI is low/zero…

If I had to pay $10k salary for a certain job previously, but I now have to pay $10k + $2k for the same job. Why would I not care about that? You were doing your job perfectly well previously, why do you want more budget?

Re: Is the industry ready for tokens-constrained work?

#49

Post by delusional »

gibolt wrote: post #45
abuani wrote: post #27

Earlier quoted context omitted.

What about a $10k to $20k bill? Or better, 100%-200% of the employees salary? At a certain point in time, finance is going to care greatly about the ability to predict costs. Pre AI, predicting the cost an engineer was quite easy. Salary + benefits + licenses for software. CI systems that were unbounded were rolled up into opex and treated as a separate line item. Where does token spend land in this now? Finance team…

Set a limit, train people to use tokens efficiently. For anyone getting high value from it, raise their limit.

How do you figure out they are "getting high value from it"

What if they feel like they are getting value from it, but are actually not?

Re: Is the industry ready for tokens-constrained work?

#50

Post by ironqcold »

I think we're still in the early days of figuring out how to use AI as a daily work tool. It's going to take time to develop the right practices. I do see the point in having token limits, they force you to be intentional. My guess is that eventually we'll end up with small, local LLMs that are unlimited for routine stuff, and we'll save the big expensive models (with quotas) for the heavy lifting.
Post reply on HN