31–40 of 245 posts
> This is expensive—the pelican cost 25 cents! Engineers get unbelievably silly about evaluating costs of things. "The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.
Would anyone pay a human to create an SVG of a pelican riding a bike?
It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website
Pelicans and bikes can be in the training set without them training for this specific benchmark.
It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website
"That artist saw a pelican at the beach once!" [cue the outrage] "He's not a real artist, he's a cheater and produces nothing original!"
Earlier quoted context omitted.
You are thinking too hard on this. This entire "benchmark" is a performative joke for attention that only works on HN. > What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? We will just have more of the same.
You say it's performative joke, but it all depends what you're using model for. So far the rule has been quite straightforward, better models consistently renders pelican in higher quality, I've yet to see an exception. It is also a good enough (for me at least) test for "taste" the model has.
It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website
Imagine if we applied this train of logic to humans. "That artist saw a pelican at the beach once!" [cue the outrage] "He's not a real artist, he's a cheater and produces nothing original!"
Plus obviously humans can still overfit to a specific style of test.
It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website
You still need an OpenRouter API Key and be careful this can burn quite a bit of money.
Imagine shilling some CLI tools no one uses in this post.
You’re reading a personal blog and complaining about an open source personal project he runs and distributes for free. He’s allowed to talk about his personal work on his personal blog. Especially considering the cli utility he talks about is directly related to the post.
Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.