I'm liking where LLMs are headed: They can do the difficult small level optimization, the boring but tedious code but cannot be tasteful. That means I'm more valuable and more productive. Good stuff
This guy gets it.
Hy4 preview
131–140 of 258 posts
Re: Hy4 preview
#132Re: Hy4 preview
#133> [...] Let's maybe add a helmet? It could improve riding theme, but may obscure head. Maybe a small cycling cap or helmet? The user didn't ask; can add red helmet? Might be cute. But pelican with big beak; a helmet might obscure. Better maybe no. > Maybe add sunglasses? no. > Maybe add water? no. https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
If someone can look at that reasoning trace and see a stochastic parrot next word prediction machine, we don't understand those words in the same way.
The stochastic parrot epithet is so 4 months ago
Re: Hy4 preview
#134Genuine Q about word optimization/token density: If we create a stripped-down vocabulary with greater token density to use less resources and to resolve ambiguities earlier in the semantic process, aren't we creating NEWSPEAK and dragging along the worst aspects of it? The ambiguity and multi-valence of words is what creates more connections between words, increases the directionality of associations, and expands the…
I don't think so. It's pretty clear that LLMs use the higher level layers for reasoning, so a bit of logorrhea very possibly enriches the result quality.
Re: Hy4 preview
#135> [...] Let's maybe add a helmet? It could improve riding theme, but may obscure head. Maybe a small cycling cap or helmet? The user didn't ask; can add red helmet? Might be cute. But pelican with big beak; a helmet might obscure. Better maybe no. > Maybe add sunglasses? no. > Maybe add water? no. https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
Is the broken English an optimization or a byproduct of the model being developed in China?
Likely something that was first made especially obvious by Chinese models and then became something worth optimizing for in English too.
Chinese can be extremely information-dense in token terms, though it depends on the tokenizer. Roughly speaking, you can pack more "meaning" into a short sequence than English often allows for. That's why "caveman" reasoning is a pretty good fit.
There's a difference between bolting caveman speak onto an existing model and training a model to reason that way, though. If you just force an existing model to be concise in outputs, you're artificially reducing its available reasoning steps and can possibly prevent useful exploration or verification. If it's trained specifically to use compressed reasoning, it can learn to represent the same intermediate ideas in fewer generated tokens, cutting the number of sequential inference steps without necessarily sacrificing the useful reasoning itself.
It's not so much inherently a Chinese-model trait, but Chinese models could definitely have helped demonstrate how effective very compressed reasoning traces can be.
There are few tests of this, but one example I thought was interesting was here: https://github.com/PastaPastaPasta/llm-chinese-english
I wouldn't say it was Chinese specifically that was emulated, but it got people thinking about tokenizers and representation efficiency, and how natural English is rather inefficient.
Re: Hy4 preview
#136Genuine Q about word optimization/token density: If we create a stripped-down vocabulary with greater token density to use less resources and to resolve ambiguities earlier in the semantic process, aren't we creating NEWSPEAK and dragging along the worst aspects of it? The ambiguity and multi-valence of words is what creates more connections between words, increases the directionality of associations, and expands the…
This is one of the dangers. AI boosters would say that humans already do this compression and it was accelerated by mass media and then the internet, and that model memory + context can be broad enough that compared to human capabilities the opportunities for depth and variability are even greater. But I think we know which way this optimization usually goes. Even the notion of a "fine-tune for subtlety" is a contradiction.
Re: Hy4 preview
#137I experimented with Hy3 for a project and was surprised with how good it was. I don't know if it's good for coding, but as a general purpose agentic model, it was only beaten by deepseek4-flash in our tests. It was so close to deepseek behaviour I kept thinking it must have been forked from it.
For the last few days I've been experimenting with the _free_ version of Hy3 offered by Opencode Go and I was also surprised to see how (relatively) good it is on coding tasks too.
The free quota from Opencode Go is also surprisingly generous, I perhaps hit limits one or two times and I've been using it _a lot_ for implementation tasks (using e.g. GLM-5.3-flash for working on specs and planning next steps).
Re: Hy4 preview
#138Re: Hy4 preview
#139Earlier quoted context omitted.
If someone can look at that reasoning trace and see a stochastic parrot next word prediction machine, we don't understand those words in the same way.
Yeah, it has been clear for a long time that there is reasoning and mental modeling going on here. The other option is that you do understand those words the same way, and the people making these (now nonsensical) anti-AI claims simply aren’t talking about the same programs/models we are. Their idea of SOTA is when chatgpt.com launched. If you took a point sample pre-Opus, and didn’t write a good prompt, of course yo…
[deleted]
Re: Hy4 preview
#140Genuine Q about word optimization/token density: If we create a stripped-down vocabulary with greater token density to use less resources and to resolve ambiguities earlier in the semantic process, aren't we creating NEWSPEAK and dragging along the worst aspects of it? The ambiguity and multi-valence of words is what creates more connections between words, increases the directionality of associations, and expands the…
Nah reducing token length means that we're just reducing English down towards a programming language like a nice demi-glace