Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is the industry ready for tokens-constrained work?

blog.alaindichiappari.dev

Post reply on HN
Print view View original

11–20 of 78 posts

Re: Is the industry ready for tokens-constrained work?

#11

Post by andriy_koval »

neaden wrote: post #6
cebert wrote: post #2

I have similar concerns about this. The past year or so, software engineers have been encouraged by employers to adopt AI tooling and agentic coding. Now, many enterprises, including mine, are starting to crack down on token spend. I’ve learned how to fully embrace agentic tooling to do tasks like keeping up with vulnerability reports, initially triaging defects that come in, etc. It would be hard for me to do things…

It's also kind of odd because even a junior engineer has to cost a company around 10 grand a month in salary, taxes, and other costs. So if the company thinks it makes you 10% more effective it seems like it should be an easy choice. So either the companies are shooting themselves in the foot limiting spending or they aren't seeing the productivity boost.

there is likely curve of diminishing productivity return

Re: Is the industry ready for tokens-constrained work?

#12

Post by deadbabe »

I don't think people understand, tokens must never be constrained.

Right now, LLMs are the worst they'll ever be, but imagine what they will be like at the peak: Anything you want to code, coded instantly. Not waiting for code to stream in or wait hours for some agents to crunch through loops and planned: it just appears on the screen instantly like the way a webpage loads. Then imagine it can be done locally, on your device. Need an entire new custom operating system from scratch for some obscure hardware? Done. Here it is.

That's going to be like pure crack to anyone who needs to do absolutely anything. That's going to be like our generation's version of "today's supercomputers will someday be in everyone's pocket".

Re: Is the industry ready for tokens-constrained work?

#13

Post by pdhborges »

neaden wrote: post #6
cebert wrote: post #2

I have similar concerns about this. The past year or so, software engineers have been encouraged by employers to adopt AI tooling and agentic coding. Now, many enterprises, including mine, are starting to crack down on token spend. I’ve learned how to fully embrace agentic tooling to do tasks like keeping up with vulnerability reports, initially triaging defects that come in, etc. It would be hard for me to do things…

It's also kind of odd because even a junior engineer has to cost a company around 10 grand a month in salary, taxes, and other costs. So if the company thinks it makes you 10% more effective it seems like it should be an easy choice. So either the companies are shooting themselves in the foot limiting spending or they aren't seeing the productivity boost.

10 grand a month for a junior is absolutely ridiculous in most of the world including large swats of the western world.

Re: Is the industry ready for tokens-constrained work?

#14

Post by deadbabe »

alainrk wrote: post #10
deadbabe wrote: post #8

Earlier quoted context omitted.

I think it will be a good thing for other reasons: there will be a massive push to get hardware good enough to run local LLMs burning infinite tokens.

Yeah but energy is still a cost, and local inference without batch and multiplexing for many users (like an office would be) is even less optimal. I'm not sure I think it's just pushing the problem forward

Then there will also be a push for cheaper, freer energy as well.

Re: Is the industry ready for tokens-constrained work?

#15

Post by toofy »

this will mirror what happened in the past.

if i’m not mistaken, in the past, people had limited time on “the mainframe”. so eventually they brought in what would be the equivalence of a local model… small computers that could do smaller tasks locally so they didn’t have to keep shelling out money to the mainframe gods and be handcuffed for usable time.

i could absolutely be mistaken that this is how it worked, it was before my time. but this is how people explain it worked for them.

i don’t think most work gives a shit about soa. smaller repeatable tasks can absolutely be run just fine on smaller local models. sure, we’ll upgrade models occasionally just like we went from suitcase sized laptops to whatever we use today.

this idea the hypedorks are pushing that soa is the only way is hilarious. hobbyists spend stupid money on classic cars, tools for woodworking, or whatever. spending money to do a hobby at home has never stopped wonks and their hobbies and businesses will do the same, spend to run models locally.

the sooner the hypeTrash does what hype always does, fades to irrelevance, the sooner we can get back to work.

Re: Is the industry ready for tokens-constrained work?

#16

Post by alainrk »

deadbabe wrote: post #12

I don't think people understand, tokens must never be constrained. Right now, LLMs are the worst they'll ever be, but imagine what they will be like at the peak: Anything you want to code, coded instantly. Not waiting for code to stream in or wait hours for some agents to crunch through loops and planned: it just appears on the screen instantly like the way a webpage loads. Then imagine it can be done locally, on you…

Right, every technology is suboptimal in the beginning, but constraints will ever exists, energy and hardware in primis. Aldo you have to factor entropy in regardless, complexity remains a cost

Re: Is the industry ready for tokens-constrained work?

#17

Post by the__alchemist »

cebert wrote: post #2

I have similar concerns about this. The past year or so, software engineers have been encouraged by employers to adopt AI tooling and agentic coding. Now, many enterprises, including mine, are starting to crack down on token spend. I’ve learned how to fully embrace agentic tooling to do tasks like keeping up with vulnerability reports, initially triaging defects that come in, etc. It would be hard for me to do things…

Maybe a dumb question, but what are you doing/seeing others do that uses that many tokens? I find myself pressing the weekly caps on the $20/month plan only when I'm really having the LLMs go wild with the Xtra high effort on the newest/biggest models on abstract problems or where I don't really understand the problem well. (E.g. improving ML models, planning creative molecule synthesis pipelines, identifying subtle performance problems etc)

Generic CRUD/Config stuff I do for work (and assume most software jobs entail? Maybe not correct) doesn't use a significant amount of tokens.

Re: Is the industry ready for tokens-constrained work?

#19

Post by Avicebron »

sokoloff wrote: post #9

Rationing tokens on a spend level or count makes no sense to me. Why would I pay $10-20K/mo to employ a software engineer and then balk at a $500, $1000, or even $2000 monthly AI bill, assuming they were even vaguely trying to use the tokens productively? I’m not an AI-maximalist, but “work a few days with AI and the rest of the month without because of cost” sounds literally crazy to me. (If you think AI is low/zero…

I think in this future they don't think a software engineer is making that much, maybe a nice 50k a year, 65K after 10+ years on the job
Post reply on HN