Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Is Not a Compiler

blog.exe.dev

Post reply on HN
Print view View original

131–140 of 174 posts

Re: Claude Is Not a Compiler

#131

Post by reinitctxoffset »

gritzko wrote: post #55

LLM-is-a-compiler is indeed a simplistic approach. I wrote a rebuttal to the yesterday's Cursor post, may reuse it here https://replicated.live/blog/follow-up The idea that a 835-page spec "just exists" and we run an LLM to implement it is completely flawed. Specs do not appear out of nowhere, they co-evolve with the code. If you have the code, why do you want to generate it again? Good software is made as a product…

Well, since we started training them on RLHF-style rating of the last generation they've totally collapsed back onto predictable suffix generators, which is why there are desperate, low-AUC BERT-inspired clasufiers bolted to them now.

They have like, Chomsky grammars now. And you can't fix it, it's structural to the process. You have to rewind almost to the pre-train.

Re: Claude Is Not a Compiler

#132

Post by encyclopedism »

keleftheriou wrote: post #96
encyclopedism wrote: post #67

Earlier quoted context omitted.

Because Claude isn't a human the perception of it is different. Humans can fly planes, computers too can fly planes. I'm not going to give the same leeway to a 'computer' flying a plane that I would to a human. Because we understand that humans are just that, humans. The demands placed on a computer are so very different.

What about driving a car? Computers are already driving cars autonomously, possibly better than the average human already, and quite plausibly will eventually be better than the best human driver.

Precisely my point. Say for examples planes, trains or automobiles, humans may make way more errors than a computer does but you can bet your bottom dollar the threshold for allowing computers to autonomously control the above is way higher. Because the expectation is that if it is a computer it ought do it far better/safer.

Re: Claude Is Not a Compiler

#133

Post by lilbigdoot »

Xmd5a wrote: post #123
atn34 wrote: post #112

Earlier quoted context omitted.

One of my professors used an example like this to encourage you to pause if you ever find yourself doing something like "maybe if I just flip this the tests will pass". The space of incorrect programs (even ones that pass your tests) is enormous compared to the space of correct programs. If you like thinking about this and haven't seen this before, check out the busy beaver problem[1]. [1]: https://en.wikipedia.org/w…

flipping > to >= works quite often though!

I was about to say a lot of algorithms or invariant bugs for me are slight mistakes at an edge of something

Re: Claude Is Not a Compiler

#134

Post by lilbigdoot »

atomicnumber3 wrote: post #70
gritzko wrote: post #55

LLM-is-a-compiler is indeed a simplistic approach. I wrote a rebuttal to the yesterday's Cursor post, may reuse it here https://replicated.live/blog/follow-up The idea that a 835-page spec "just exists" and we run an LLM to implement it is completely flawed. Specs do not appear out of nowhere, they co-evolve with the code. If you have the code, why do you want to generate it again? Good software is made as a product…

Something I have always (even pre-LLMs) found funny to think about is - all code possible to run on a computer already exists. It's some permutation of all the bits of available memory. It's in there somewhere. So, suppose you want a specific program. 1. Some huge % of those possible programs are obviously not the one you want (most don't even compile). 2. Remaining programs might look similar to the one you want, bu…

I think this is a great abstraction of the process I've never considered in those terms and I like it. It explains a bit of how software development differs from some other building processes where the desired outcome is being discovered as you figure out how to make something that produces that outcome

Re: Claude Is Not a Compiler

#135

Post by kloud »

kimjune01 wrote: post #130
kloud wrote: post #129

Earlier quoted context omitted.

You can reference other specs in a prompt. Video codecs are well specified with hundreds of pages of specs, they are probably in some form in the weights already, but you could also reference them and hash them as inputs directly to create a functional alternative to ffmpeg.

i meant to make the distinction between 'prompt' and 'spec' in that prompts are something that a human types in by hand. Interesting programs are a combination of canon and novel input

Gotcha, yeah if a prompt does not produce the result you want, you need to add more details to it, depending on the scope it can be a lot. When it includes all sufficient details it becomes a spec.

Re: Claude Is Not a Compiler

#136

Post by [deleted] »

maweki wrote: post #92
chubot wrote: post #74

Earlier quoted context omitted.

all code possible to run on a computer already exists. It's some permutation of all the bits of available memory. It's in there somewhere This seems either mathematically impossible or vacuous, depending on what you mean ...

I think the intent is clear. We could just enumerate all strings in size and lexicographical order. Then the task is only to filter, which strings are programs we desire. It's the same sentiment as "the typing itself is not the issue, it's the order of the button presses that's difficult." Of course, by Rice's theorem, all interesting questions are undecidable anyway...

[deleted]

Re: Claude Is Not a Compiler

#139

Post by hombre_fatal »

kimjune01 wrote: post #107
kloud wrote: post #62

Specs might become one solution for coping with the need to review increased volume of code. A spec is a higher level of abstraction than code, which is a higher level of abstraction than machine code. The industry made the transition to higher-level once, paradigm is changing so it might happen again. The workflow I imagine is either deriving specs from the conversation or reverse engineering the code to spec, revie…

prompts are not specs because most often, prompts are underspecified. You can prompt fizzbuzz well enough for it to be functional, but not ffmpeg. So the distinction is less about Claude's capability, more about the quality of the input. Garbage in garbage out

I think something can be considered a spec to some degree if outcome was derived from it.

It might underspecify, but that is just a quality of the spec. And as long as there is any discretion at implementation time, then the spec necessarily underspecifies to some degree. The goal of a spec is often to be clear on key directional concerns. Consider RFCs.

There's a trade-off between specification and implementation discretion. But if you think about starting at one end ("an internet forum") and then incrementally expanding the specifications ("it has topics with many posts in a linear paginated list"), there's some point where everything you care about is in the spec, and the rest is left to implementor discretion.

That's what I'd consider a good spec and a good LLM prompt.

Re: Claude Is Not a Compiler

#140

Post by r2_pilot »

dfee wrote: post #124
agentultra wrote: post #120

Earlier quoted context omitted.

You should check out Borges’, The Library of Babel . Basically if you have an infinite string of characters in some language then you have every possible book, tweet, script, scribble, and scrap of everything ever written. You just have to find it. Thats already a large universe to cover. Programs? Much bigger set.

the infinite string must have more refined properties than infinite length. an infinite string of a repeating char doesn't encode much of interest.

I believe it's more that the library contains the finite set of all books which have a (unique) permutation of the language's characters to size N. The challenge to finding the cure to cancer in such a collection of sufficiently large N is indeed finding which book it is in.
Post reply on HN