Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The last six months in LLMs in five minutes

simonwillison.net

Post reply on HN
Print view View original

521–530 of 631 posts

Re: The last six months in LLMs in five minutes

#521

Post by embedding-shape »

kenjackson wrote: post #503
freedomben wrote: post #460

Earlier quoted context omitted.

I've noticed that the bar for "quality" when people judge AI is often significantly higher than what they'd hold a human to. I'm not saying GP et al are doing this (I haven't looked myself), but it is a widespread pattern I've noticed both professionally and personally. I don't know why it is.

I have seen it too. The answer is easy - they don’t like AI. I've seen similar things with some people that don’t like women in tech or certain minorities - they suddenly critique at an extremely high level. I also haven’t looked at this particular case, but it wouldn’t surprise me to be the same thing here.

> I also haven’t looked at this particular case, but it wouldn’t surprise me to be the same thing here.

Be surprised then, because me, who left the critique, probably exclusively programmed with agents for the last year or so, so unlikely I think the code is bad because I "don't like AI". I don't love it either, but wouldn't call myself a AI-hater by any measurements, would be weird to write articles like this if so: https://emsh.cat/en/one-human-one-agent-one-browser/

Re: The last six months in LLMs in five minutes

#522

Post by embedding-shape »

malfist wrote: post #518
embedding-shape wrote: post #496

Earlier quoted context omitted.

Hmm, ok, I think the penis in case is a bit distracting, can you de-analogize this to their real terms and tell me what this is supposed to mean and be related to developing with LLMs?

Just because you _can_ do something with a tool, doesn't mean it's the right tool for the job. Just because someone has contorted their entire process to adapt to a misshapen tool, and gotten good results, doesn't mean that's the right thing to do. It is reasonable to both use the right tool for the right job, and demand better tools than you currently have. Success with the wrong tool in the wrong job doesn't mean i…

> Just because you _can_ do something with a tool, doesn't mean it's the right tool for the job. Just because someone has contorted their entire process to adapt to a misshapen tool, and gotten good results, doesn't mean that's the right thing to do.

Ok, I agree with this, don't use the wrong tool for the wrong job.

> It is reasonable to both use the right tool for the right job, and demand better tools than you currently have. Success with the wrong tool in the wrong job doesn't mean it's the right tool for the right job.

Yes, I agree with this too.

I'm still not sure how this relates to LLMs and particular this specific context. I claimed that the output of your agents depend on the developer driving it. You're saying "not every tool is right for every job", I agree with this too, but is that against/for what I said?

Could you just clearly write out exactly what you're arguing for here, no analogies or metaphors, just plain and simple, because I still feel like we're having two different conversations.

Re: The last six months in LLMs in five minutes

#523

Post by embedding-shape »

knollimar wrote: post #504
embedding-shape wrote: post #493

Earlier quoted context omitted.

That's just poor engineering, product building and testing, same can happen with/without LLMs, no doubt.

If the company making hammers can't hold it right, it suggests something about the hammers, no?

Yeah, they have bad engineers, product people and testers.

Microsoft is pretty shit at launching products, does that mean "products" as a concept is wrong? No, it just means Microsoft is bad at products, not more than that. Not sure why you have to extrapolate over an entire ecosystem just because one actor is bad at something.

Re: The last six months in LLMs in five minutes

#524

Post by arcanemachiner »

knollimar wrote: post #504
embedding-shape wrote: post #493

Earlier quoted context omitted.

That's just poor engineering, product building and testing, same can happen with/without LLMs, no doubt.

If the company making hammers can't hold it right, it suggests something about the hammers, no?

In the case of Claude Code, it suggest a lot about the company making the hammers.

Re: The last six months in LLMs in five minutes

#525

Post by verdverm »

LZ_Khan wrote: post #57

I'm curious how the 6 months have looked from a non-programmer's perspective. What kind of co-working tools and similar optimizations have people from other fields experienced?

They lag behind because we build for ourselves first. We are rolling out Claude to the biz team this week and they will get access to Cowork, which is still preview aiui.

Sales will be another big user of agent automations, for better or worse. Poor usage by Google to craft emails and slides for us is why the suits are getting an Anthropic sub. Stay human in the loop my friends!

Re: The last six months in LLMs in five minutes

#526

Post by djeastm »

IdiotSavage wrote: post #350
jimbobthemighty wrote: post #161

I asked Gemini for a video of 'pelican riding a unicycle in hyde park' - I was blown away by the output: https://gemini.google.com/share/55e250c99693

Graphically perfect, but content-wise nonsense. The pelican's center of gravity is clearly behind the wheel. It needs to be above or very slightly ahead of the wheel.

Maybe the pelican has something heavy in its mouth.

Re: The last six months in LLMs in five minutes

#527

Post by ionwake »

falcor84 wrote: post #239
ionwake wrote: post #218

Earlier quoted context omitted.

only SVG counts tho, dont know why

Willison chose this task because (unlike actual images of pelicans) is was clearly not in training data, but could be reasoned about and composed from what's there. But just like those "how many golf balls can you fit in a 747?" interview questions, it should now be retired.

Thank you for the reply. Would something like a Squirrel flying a hangglider as an SVG be a good new test? Or would that be indirectly in the training data too?

Re: The last six months in LLMs in five minutes

#528

Post by randusername »

hollowturtle wrote: post #459
Falimonda wrote: post #410

Earlier quoted context omitted.

Which languages and subject matter do you work with?

c/c++, java, kotlin, go, some perl scripting, some javascript. Gaming industry

We'll there's your problem.

F1 mechanic pops the hood of a mass-market Toyota Corolla and doesn't understand why everyone says it's really good.

A lot of us are out here building websites or phone apps.

Not to say that these things can't also be taken very seriously from first-principles, but I think that's rare.

Re: The last six months in LLMs in five minutes

#529

Post by albedoa »

walthamstow wrote: post #303
Philip-J-Fry wrote: post #292

Earlier quoted context omitted.

I have tried Claude code. It doesn't look like that! I don't know what the project is. All I see is a TUI that looks completely broken. Go and use Claude Code right now. Does it look like that? Random underscores all over the page. No it doesn't.

It can look like that in certain conditions. The question is why are you so eager to give critique on unrelated work, appearing in a demo screencap, to someone who didn't produce it?

> The question is why are you so eager to give critique on unrelated work

That is not the question. The topic of discussion had been defined multiple times before you commented!

Re: The last six months in LLMs in five minutes

#530

Post by godelski »

romaniv wrote: post #515

> there’s zero chance any AI lab would train a model for such a ridiculous task. A lot of people here stated that this is a ridiculous metric, but no one seems to remember that it was introduced in the initial GPT report ("Sparks of Artificial General Intelligence: Early experiments with GPT-4" [1]) by Microsoft about 3 years ago. Shortly after that it was parroted by a network of booster accounts and became a thing…

For those curious, Simon's first public usage of it is Oct 25th, 2024[0]. While I'm not aware of any specific "pelican riding a bicycle" prompts being tested in a paper[1], the GPT paper did several SVG and tikz tests and the actual image is rather arbitrary. You wouldn't want to optimize for a singular image but also if you're doing halfway decent training a pelican riding a bicycle shouldn't be too hard to draw, and well... you can see several good examples if you look through different pages on [0].

[0] https://simonwillison.net/tags/pelican-riding-a-bicycle/?pag...

[1] I'm sure there is because of Simon's fame

Post reply on HN