Live data from Hacker News

Uber’s COO says it’s getting harder to justify money spent on tokenmaxxing

businessinsider.com

91–100 of 360 posts

Re: Uber’s COO says it’s getting harder to justify money spent on tokenmaxxing

#91
post #68

I always used to wonder this about software stacks even prior to LLMs, but it seems more relevant now somehow: When will Uber (or your favourite company) be 'done'? They've been writing software for 16 years. They match drivers to passengers. More software isn't going to increase the chance that I seek them out instead of taking a bus or train. Will their software be finished in 20 years? 80?

Weren’t they trying to do their own self-driving thing?

I think this is partly a problem with companies that have had heavy investment. Uber’s value isn’t based on what they are doing, it is based on the idea that they are going to render ideas like owning your own car or taking public transit obsolete (I mean that’s an exaggeration but less of one than it ought to be).

Re: Uber’s COO says it’s getting harder to justify money spent on tokenmaxxing

#92
post #29

What if... we stop for a moment, and then, after thinking for a moment, we stop hammering nails with a microscope, and stop using token usage as a metric of productivity? I know it's sounds stupid, but what if

You're now in the last frame of the comic, getting thrown out the window.

Re: Uber’s COO says it’s getting harder to justify money spent on tokenmaxxing

#94

Tokenmaxxing makes no sense, it is akin to write extremely inefficient SQL / Spark Jobs, full of cartesian joins, ultra skewed datasets, etc, just for the sake of using as much compute / memory / IO as possible. This always happens when the metric becomes the goal, companies should nurture and foster an environment where AI is used in the most efficient way possible, first asking "do we really need an agent for this"…

It's toddler-level logic. "You can achieve positive outcomes by using X. Therefore, we need to use as much X as possible to maximize positive outcomes."

It's like trying to win a race by setting a gas station on fire.

Re: Uber’s COO says it’s getting harder to justify money spent on tokenmaxxing

#95

Earlier quoted context omitted.

The crazy thing is their salary does not actually benefit from riding these trends. Unless it's equally/even more clueless board level pressure with ulterior motives (i.e., lifting their other AI investments or the sector as a whole).

Every c suite in the country is panicking about being left behind, from their perspective it’s either token max or fade into obscurity, or at least that’s what they were sold

Please. These are the same people that force their employees to use Microsoft teams because slack is $5 an employee a month. They're not going to sit idly by while employees burn thousands a month in tokens.

Re: Uber’s COO says it’s getting harder to justify money spent on tokenmaxxing

#96

Tokenmaxxing makes no sense, it is akin to write extremely inefficient SQL / Spark Jobs, full of cartesian joins, ultra skewed datasets, etc, just for the sake of using as much compute / memory / IO as possible. This always happens when the metric becomes the goal, companies should nurture and foster an environment where AI is used in the most efficient way possible, first asking "do we really need an agent for this"…

The argument in favor of "tokenmaxxing" has always been that it's creating space for employees to freely explore the broad and novel space of AI-enabled workflows. I've seen a number of use cases where I'm skeptical any value is being produced, but a number of others where some team or another has finally solved a long-standing problem of theirs with an agentic workflow that would have been hard to justify to a cost review committee.

> They should also promote projects that aim at saving tokens, increasing cache hits, codifying the information in ways such they use as less context as possible (graphs of knowledge are pretty good for this!)

My understanding is that most big "tokenmaxxing" companies do have teams who are working on this in the background.

Re: Uber’s COO says it’s getting harder to justify money spent on tokenmaxxing

#97

It's amazing that it took months to figure this out. "Well we thought that if engineers are told to maximize costs through AI use, to consume as much as possible of a resource that costs us money, then obviously good things will happen. Imagine my surprise when it didn't turn out that way." Imagine if engineers were ranked based on their AWS spend. People allocate VMs and fill databases with terabytes of random bits,…

I think unfortunately it's not about what seems obvious, or even what seems more likely, but about what seems retrospectively justifiable regardless of outcome.

The incentive structure of this type of decision is 'absolutely under no circumstances existentially mess up'. Ostensibly with respect to the organisation, but in actual reality much more so with respect to the individual(s) involved in the decision.

If everyone else is doing something that kind of obviously makes no sense, and you decide to break from the crowd by instead doing what does make sense, then there's a pretty solid chance of gaining a temporary edge while reality resolves the truth. But those gains probably won't matter all that much for the organisation, or indeed your position within it. It's a solid chance of an unimportant gain.

However on the other hand, there's a tail risk that something very unexpected happens and the thing everyone's doing that makes no sense actually turns out to make sense - sometimes even for entirely unpredictable incidental reasons - and then, well, you're in trouble. Not necessarily 'you' the organisation.. they'll likely be able to catch up and it won't matter that much. But for 'you' personally, the decision maker, it's very much not good.

As a bonus, in the much more likely scenario that the thing that makes no sense turns out to indeed make no sense, you're in the same boat as everyone else, there's no relative loss, and most importantly you don't stick out as someone who did something as risky as to go against the prevailing, albeit pretty clearly nonsensical, sentiment.

So basically, game theory tells you pretty quickly to just go with the thing that makes no sense if you're optimising for some (weighted) cross of what's best for the organisation and yourself as the decision maker.

Re: Uber’s COO says it’s getting harder to justify money spent on tokenmaxxing

#98
post #68

I always used to wonder this about software stacks even prior to LLMs, but it seems more relevant now somehow: When will Uber (or your favourite company) be 'done'? They've been writing software for 16 years. They match drivers to passengers. More software isn't going to increase the chance that I seek them out instead of taking a bus or train. Will their software be finished in 20 years? 80?

I think you’re missing how complex international operations and optimization are.

Each country has their own laws around what uber is and isn’t allowed to do. This needs to be formalized in code. For example you actually call a taxi, though the uber app, and the amount you pay is per mile, not a fixed fare decided ahead of time. To add to this complexity, some cities will have their own laws. What happens if you take an uber from town a to b, where each one has different laws ? A lawyer probably has an answer but the app needs to adhere to that. On top of that laws change all the time.

Optimization, well you can always optimize something. speed, costs, paths etc. In a way this never ends.

I think the part we interact with as consumers is a tiny sliver of the complexity those services have to build and operate.

Re: Uber’s COO says it’s getting harder to justify money spent on tokenmaxxing

#100
post #54

I'd be interested to know if this is about individual employee AI usage, or use of AI tokens in production features, or both - and assuming both, what the split is. I can see how Uber could burn unbelievable amounts of tokens if they start running internal features that run a bunch of prompts against every completed ride, or every customer profile, for example. Or maybe this is about employee usage, but they introduc…

IMO, it's undoubtedly both.

The number of product teams who have shipped expensive-to-operate AI features is wayyyy up there, and for many of the scenarios I've seen, customers simply don't care or are unwilling to pay significant premium for access to it.

At the same time I'm starting to see some direction from people in leadership that I should "use the right model for the job" and things along those lines, which is a very, very different line from what I was hearing 12 months ago.

My continued prediction is that we are going to see a tweak on the SaaS model where the sweet spot moves to metered usage pricing of really fine-grained API-based access for apps which traditionally have been operated solely via the UI. Long term the trend is going to be "we'll house the data, enrich it, maintain it, provide fine-grained API access over it tailored to model usage, and you bring the model" with some services opting to give you the model interaction layer/harness. IOW I don't think SaaS is dead. Far from it. However, I do think that a lot of people are going to be looking to interact with SaaS apps via their own models with APIs that support those use cases better than a lot of those APIs do today.

Post reply on HN