Sibling post here: https://news.ycombinator.com/item?id=46193412
I failed to recreate the 1996 Space Jam website with Claude
441–450 of 483 posts
Re: I failed to recreate the 1996 Space Jam website with Claude
#442Earlier quoted context omitted.
No, the thing needing proof is the novel idea: that LLMs can produce original code.
What's your proof that the average college student can produce original code? I'm reasonably certain I can get an LLM to write something that will pass any test that the average college student can, as far as that goes.
I'm not asking about averages. I'm asking about any. There is no need to perform an academic research study to prove that humans are capable of writing original code because the existence of our conversation right now is the counter-example to disprove the negation.
Yes, it is true that a lot of humans remix existing code. But not all. It has yet to be proven that any LLM is doing something more than remixing code.
I would submit as evidence to this idea (LLMs are not capable of writing original code) the fact that not a single company using LLM-based AI coding has developed a novel product that has outpaced its competition. In any category. If AI really makes people "10x" more productive, then companies that adopted AI a year ago should be 10 years ahead of their competition. Substitute any value N > 1 you want and you won't see it. Indeed, given the stories we're seeing of the massive amounts of waste that is occurring within AI startups and companies adopting AI, it would suggest that N < 1.
Re: I failed to recreate the 1996 Space Jam website with Claude
#443Them: AI will take jobs.
The AI: https://chatgpt.com/share/6923df03-7304-8010-bd08-cd335f0ee9...
Re: I failed to recreate the 1996 Space Jam website with Claude
#444I wouldn't call it entirely defeated, it got maybe 90% of the way there. Before LLMs you couldn't get 50% of the way there in an automated way. > What he produces I feel like personifying LLMs more than they currently are is a mistake people make (though humans always do this), they're not entities, they don't know anything. If you treat them too human you might eventually fool yourself a little too much.
As a couple other comments pointed out, it's also not fair to judge Claude based on a one shot like this. I sort of assume these limitations will remain even if we went back and forth but to be fair, I didn't try that more than a few times in this investigation. Maybe on try three it totally nails it.
Very true, I would also caution this with test projects with real humans in the hiring process. Comparing one-shots from actual people is unfair too, and often the most valid assessment comes with giving them feedback and seeing how they respond to it.
Aside from that point: if you are reading this and making people do a project as part of the hiring process, you should absolutely be paying them for their time (even a token amount).
Re: I failed to recreate the 1996 Space Jam website with Claude
#445Earlier quoted context omitted.
"want to automate our jobs away", "Asmodai and team want you working in line at McDonald's", "You're a serf" all obviously fit that description. Low-information, high-indignation rhetoric is exactly what we don't want here, so please don't post like this.
Fair enough, I'll temper the rhetoric. It still seems you can make the front page posting these words as long as they're externally hosted. Or maybe it's the fact Anil is a bit of a celebrity: https://news.ycombinator.com/item?id=46128548
Hey man, there's a world of difference between what's essentially a tweet (HN comment) and an article.
If you wanna make charged statements, at the very least substantiate them. (To be clear: I don't necessarily disagree with you, but you gotta admit your language is emotionally laden and lacks substance.)
Re: I failed to recreate the 1996 Space Jam website with Claude
#446Earlier quoted context omitted.
Ah, those days, where you would slice your designs and export them to tables.
It was relatively OK to deal with when the pages were created by coders themselves. But then DreamWeaver came out, where you basically drew the entire page in 2D and it spat out some HTML tables that stitched it all back together again, and the freedom it gave our artists in drawing in 2D and not worrying about the output meant they went completely overboard with it and you'd get lots of tiny little slices everywhere…
wasn't it Fireworks that sliced the image originally. you'd then be able to open that export into Dreamworks for additional work. I didn't do that kind of design very long. Did Dreamworks get updated to allow the slicing directly bypassing Fireworks?
Re: I failed to recreate the 1996 Space Jam website with Claude
#447There was a response to this post on the front page earlier this morning that was able to get Claude to succeed simply by giving it access to Playwright so it could see what it was doing and telling it in the prompt that it needed to be pixel perfect:
https://news.ycombinator.com/item?id=46193412
As of right now, it seems to have been flagged into oblivion by the anti-AI crowd. I found both posts to be interesting, and it's unfortunate that one of them is missing from the conversation.
Re: I failed to recreate the 1996 Space Jam website with Claude
#448There was a response to this post on the front page earlier this morning that was able to get Claude to succeed simply by giving it access to Playwright so it could see what it was doing and telling it in the prompt that it needed to be pixel perfect: https://news.ycombinator.com/item?id=46193412 As of right now, it seems to have been flagged into oblivion by the anti-AI crowd. I found both posts to be interesting, a…
yeah that sucks. I also linked it to the top of my post so folks can take a look.
Re: I failed to recreate the 1996 Space Jam website with Claude
#449Earlier quoted context omitted.
Certainly if a human wrote code that solved this problem, and a second human copied and tweaked it slightly for their use case, we would have a word for them. Would we use the same word if two different humans wrote code that solved two different problems, but one part of each problem was somewhat analogous to a different aspect of a third human's problem, and the third human took inspiration from those parts of both…
> What if it were ten different humans writing ten different-but-related pieces of code, and an eleventh human piecing them together? What if it were 1,000 different humans? What if it was just a single person? I take it you didn't read any of the code in the ocaml vibe pr that was posted a bit ago? The one where Claude copied non just implementation specifics, but even the copyright headers from a named, specific pe…
> So your comment boils down to; plagiarism is fine as long as I don't have to think about it.
It is actually worse: plagiarism is fine if I'm shielded from such claims by using a digital mixer. When criminals use crypto tumblers to hide their involvement we tend to see that as proof of intent, not as absolution.
LLMs are copyright tumblers.
Re: I failed to recreate the 1996 Space Jam website with Claude
#450I personally don't understand why asking these things to do things we know they can't do is supposed to be productive. Maybe for getting around restrictions or fuzzing... I don't see it as an effective benchmark unless it can link directly to the ways the models are being improved, but, to look at random results that sometimes are valid and think more iterations of randomness will eventually give way to control is a…
I think this is a reasonable take. I think for me, I like to investigate limitations like this in order to understand where the boundaries are. Claude isn't impossibly bad at analyzing images. It's just pixel perfect corrections that seem to be a limitation. Maybe for some folks it's enough to just read that but for me, I like to feel like I have some good experiential knowledge about the limitations that I can keep…
Yes but follow this forward, what about current models would be informative about future models. We've seen waves of "insight" come and go, to the point where there are endless waves of people at different points in the journey, there's a cohort of people that would be upset at the statement that prompt engineering is useless, and others that would support that as exactly right, and still more that would have a redefinition of the word prompt to include many other things. This is my exact complaint. You would want it work like how you want it to work, that our collective discoveries will turn into education and learning, but the content in the models and the subsequent inference based on that information have all not behaved like the physical sciences with regards to discoveries providing universal and reliable knowledge.