Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What I'm Finding About LLM Code Style and Token Costs

jimmont.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: What I'm Finding About LLM Code Style and Token Costs

#11

Post by linzhangrun »

defytonofficial wrote: post #4

This matches my experience. I've been using OpenRouter with GPT-4o for an image verification service, and the prompt engineering choices have a measurable impact on cost. One thing I found: asking the model to respond in structured JSON (with a strict schema) vs free-form text cuts token output by ~40% on average. The model stops "explaining itself" and just gives you the answer. Also noticed that including a referen…

why still use gpt-4o?

Re: What I'm Finding About LLM Code Style and Token Costs

#12

Post by jimmont »

defytonofficial wrote: post #4

This matches my experience. I've been using OpenRouter with GPT-4o for an image verification service, and the prompt engineering choices have a measurable impact on cost. One thing I found: asking the model to respond in structured JSON (with a strict schema) vs free-form text cuts token output by ~40% on average. The model stops "explaining itself" and just gives you the answer. Also noticed that including a referen…

That's an excellent idea I plan to try, thanks—re using structured JSON with schema. The most success I've had is saying "be brief" or an explicit size, like one line, or do not explain, etc. I haven't measured other instructions so extensively. They do work but the more specific the better. Other strategies around outputs that are more natural language seem to be hands-down the direction to take, and get away from the machine language habits we've used in the past. It's super interesting seeing this new practice emerging and more or less inventing parts of it along the way. Right now I'm at the place where my brute force and elaborate explanations were reaching their limit and in the frustration just realized I need to take a few days and try to figure out the tool. Across all these the pattern seems entirely that the constraints bound the probability space, whether it's the format like you suggested, or the instruction we give, including the space we point it toward (Web APIs, runtime, schema, etc). In all instances where it's not working the solution seems to be what does the pattern reduce to, and what specifics are the do/don't to go with that, and most of the time the results improve immediately. Your tip seems excellent for this. An easy-button.

Re: What I'm Finding About LLM Code Style and Token Costs

#13

Post by Balinares »

anttiharju wrote: post #8
bombcar wrote: post #5

I just had Claude try to process an RSS feed and it was about to ZALGΌ IS TOƝȳ THË PO NY itself and I pointed that out and it immediately said "Wordpress has a json interface, I'll use that". You need to know the shape of the solution ...

Context about tony the pony https://stackoverflow.com/questions/1732348/regex-match-open...

Thank you for helpfully de-darmoking that for the less pony-attuned of us.

Re: What I'm Finding About LLM Code Style and Token Costs

#15

Post by allanmacgregor »

lelanthran wrote: post #14

It's really hard for me to parse LLM-generated prose in blog posts - the reasoning is disjointed, logic is split everywhere. Is it already too late to have humans just write down what they are thinking instead of passing it through a lossy expander?

Human writting will come at a premium. I don't think LLM-generated prose goes away at all; the scary part is that I'm starting to catch people adopt the same tropes and language patterns in their speech/writing, manufactured contrarianism for example seems rampant in social media and blog posts ... but I guess nobody is talking about that :D.

Re: What I'm Finding About LLM Code Style and Token Costs

#17

Post by jimbokun »

anttiharju wrote: post #8
bombcar wrote: post #5

I just had Claude try to process an RSS feed and it was about to ZALGΌ IS TOƝȳ THË PO NY itself and I pointed that out and it immediately said "Wordpress has a json interface, I'll use that". You need to know the shape of the solution ...

Context about tony the pony https://stackoverflow.com/questions/1732348/regex-match-open...

> Have you tried using an XML parser instead?

Re: What I'm Finding About LLM Code Style and Token Costs

#18

Post by joshka »

username135 wrote: post #16

More than half my battle using things like codex comes from removing unnecessary code checks and verbose logic. Even when prompted, it just can't help itself. It's a willful beast.

Click the thumbs down report on the output (to better train the model in future to produce better code / provide signal to the people doing training that a common behavior is problematic) and then stick some guidance in your AGENTS.md. It can be fairly whack-a-mole at times on this sort of thing, but sharpening your personal saw with AGENTS.md is an effective approach to generally doing better.

Re: What I'm Finding About LLM Code Style and Token Costs

#19

Post by joshka »

I think the reasonable thing that needs to happen here is frontier labs need to look at ways to incorporate better up to date gilded chunks of gold data into their post training. I'd love to be able to throw a chunk of scenarios and expected outputs for using various tooling and have this applied to gpt-next / opus-next. I recently did a test with having Claude do this for generating a skill for jujutsu vcs that was based on eval-ing and ablating many many instructions over hundreds of scenarios. This is good, but skill based post training costs tokens for every user rather than at the model level.

Maybe there should be a submit your post training corpus here thing somewhere.

E.g. this really comes down to advice changing in short time frames that aren't represented in data that satisfies knowledge cutoff which can be as long as 14 months in some things or more for older models. It's not just the problem of knowledge, but the grading of seen output. The models have been trained to produce the older style code because it compiles and solves the problems.

Here, the reframe that likely makes sense noting is that "Deno and runtimes like Cloudflare Workers implement the Web API surface natively", that's the strongest single thing that would help steer an agent to correctly write code for the code in question (assuming the Web API surfaces that are key are in distribution). Add something like - "Where there may be reasonable obvious updates that can be used in 2026, use them" ...

Post reply on HN