Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Not everything should cost a token: the case for deterministic AI

vybe.build

Post reply on HN
Print view View original

11–20 of 20 posts

Re: Not everything should cost a token: the case for deterministic AI

#11

Post by marwann »

t-writescode wrote: post #5

"wire up an agent to do": * pull a metrics API every morning * reshape JSON * drop results into a table This means they: * know there's an API that they need to call, likely a specific endpoint * know there's a database * know the schema of the database * know where the data should go into the database -- Or perhaps they re-derive this every single day. Every day, they search for an enpdoint and call it. Every day, t…

I believe many non-technical people are spinning up AI agents nowadays without having a basic understanding of what happens behind the scenes, and usually, with no interest in learning. They prompt, watch the agent work and see the magic, without questioning the structure even a little.

I don't think we should blame them, as they are non-technical, but either educate them or put guardrails at the system prompt level.

Re: Not everything should cost a token: the case for deterministic AI

#12

Post by marwann »

eulgro wrote: post #2

> loading a few thousand tokens of raw JSON into a context window, asking a language model to reformat data it did not need to reason about... > That trap catches almost everyone right now. Has everyone really become so stupid now? I wouldn't even have thought of doing such a horrible thing as using AI to format JSON... Needless to say I stopped reading there.

> Has everyone really become so stupid now?

I wouldn't mistake stupidity for ignorance. A lot of "tech-sensitive", yet non-technical people are using agents/vibe coding these days. Their level of understanding is very basic ("JSON is structured data", "I need to reformat this to fit my need"), but they wouldn't get that the best way of doing things does not necessitate burning tokens.

Re: Not everything should cost a token: the case for deterministic AI

#13

Post by xg15 »

chmod775 wrote: post #10

> A team I talked to recently wired up an agent to do something simple: pull a metrics API every morning, reshape the JSON, and drop the result into a table. Clean idea. It worked on day one. So a team of people with access to "intelligent" help did not immediately figure out that they could just have the model write a small script that does the job perfectly every time? That's "call a doctor" level of stupid that sh…

Yes, from a programmer's perspective, this is insane. But the team may have been made up of people without coding experience who maybe didn't even know what a script was. I imagine there are a lot of people right now who "know AI" and not much else, for whom this would be a genuinely new insight...

Note that even before AI, there were a lot of manual data munging jobs that could famously be replaced by "a very small shell script".

I worry, this might become worse if we see the emergence of "100% nontechnical tech startups" where no one in the entire org knows how to code - because why would you, we have AI for that...

Re: Not everything should cost a token: the case for deterministic AI

#14

Post by doruktt »

We are building an infrastructure coop for NGOs with a wide range of open-source tooling. What we are working on is an agentic tool to build tooling on NixOS. So, for best performance, we split up the agentic interface into a deterministic MCP to generate flakes from templates, build contracts to connect different tools, etc. But there are some tasks where AI is perfect for the job, so it goes down to the model flow to extract tooling from Docker Compose files, read errors to come up with solutions, etc. I believe this approach is both economical and logical to follow. But as I mentioned, we are in the alpha stage, so we will see how it comes out.

Re: Not everything should cost a token: the case for deterministic AI

#15

Post by umpalumpaaa »

doruktt wrote: post #14

We are building an infrastructure coop for NGOs with a wide range of open-source tooling. What we are working on is an agentic tool to build tooling on NixOS. So, for best performance, we split up the agentic interface into a deterministic MCP to generate flakes from templates, build contracts to connect different tools, etc. But there are some tasks where AI is perfect for the job, so it goes down to the model flow…

Reads like an ad

Re: Not everything should cost a token: the case for deterministic AI

#16

Post by _superposition_ »

I thought this was all common sense. Coming from a background of workflow automation I've simply used AI to solve steps that could not be solved before. The example may have made sense if the input was freeform text like an email etc, but surely not json.

Re: Not everything should cost a token: the case for deterministic AI

#17

Post by htrp »

didgeoridoo wrote: post #9

I built a game for a workshop at the AI Engineer conference last week. The idea was to try to optimize a CLI that your agent could use to call a remote service to process and fulfill complex natural language coffee orders. Within 3 minutes, everyone’s agents had reverse engineered the Markov chain I used on the server to generate the order text, and they all began to write deterministic parsers to churn through them.…

is this written up anywhere?

sounds like a fascinating case study!

Re: Not everything should cost a token: the case for deterministic AI

#18

Post by unparagoned »

chmod775 wrote: post #10

> A team I talked to recently wired up an agent to do something simple: pull a metrics API every morning, reshape the JSON, and drop the result into a table. Clean idea. It worked on day one. So a team of people with access to "intelligent" help did not immediately figure out that they could just have the model write a small script that does the job perfectly every time? That's "call a doctor" level of stupid that sh…

People and teams absolutely are doing really dumb and stupid stuff. And they could learn a lot from this article

Re: Not everything should cost a token: the case for deterministic AI

#19

Post by clickety_clack »

chmod775 wrote: post #10

> A team I talked to recently wired up an agent to do something simple: pull a metrics API every morning, reshape the JSON, and drop the result into a table. Clean idea. It worked on day one. So a team of people with access to "intelligent" help did not immediately figure out that they could just have the model write a small script that does the job perfectly every time? That's "call a doctor" level of stupid that sh…

There’s a lot of teams out there who had to find a way to deploy agents to keep their jobs while working on problems like this that didn’t need them.

Re: Not everything should cost a token: the case for deterministic AI

#20

Post by jarodrh »

We've all been there where we choose the easy least efficient route just cuz (laziness...can afford it), but we've mostly done this consciously. The example in the article however does take this to another level and understandably so, as it's so easy now to take the easier route when you have no idea what you've requested is actually doing under the hood. I'm sure if you explained it to them, they'd immediately go...oooh I'm building my car every time I need to drive to work. As many comments have already stated, it's programming 101.

It's almost no different to using an expensive frontier model to do a general scouting task. Use the right tool for the job. A hammer can do most things, but should it be used for most things...

I think one of the most important non-technical things everyone should strive for is a mental awareness for figuring out what the right question is.

> "Exactly!" said Deep Thought. "So once you do know what the question actually is, you'll know what the answer means."

Post reply on HN