Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

mastodon.world

Post reply on HN
Print view View original

231–240 of 998 posts

Re: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

#231

Post by BoredomIsFun »

chrisandchris wrote: post #181
BoredomIsFun wrote: post #155

Ok folks, here is a different perspective. I used local model, GLM-4-0414-32b, a trashy IQ4_XS quant, and here what I got: prompt #1: > the car wash only 50 meters from my home. I want to get my car washed, should I drive or walk? Walking is probably the better option! Here's why: Convenience: 50 meters is extremely short – only about 160 feet. You can likely walk there in less than a minute. Efficiency: Driving invo…

What I relly dislike about these LLM is how verbose they get even for such a short, simple question. Is it really necessary to have such a lobg answer and who's going to read that one anyway? Maybe it's me and may character but when human gets that verbose for a question that can be answered with "drive, you need the car" I would like to just walk away halfway through the answer to not having to hear all the universe…

Well, when I asked for a very long answer (prompt #2), the quality had dramatically improved. So yes, longer answer produces better result. At least with small LLMs I can run on my GPU locally.

Re: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

#232

Post by londons_explore »

j_maffe wrote: post #179
clktmr wrote: post #153

Earlier quoted context omitted.

At least try a different question with similar logic, to ensure this isn't patched into the context since it's going viral.

You can't "patch" LLM's in 4 hours and this is not the kind of question to trigger a web search

A tiny bit of fine-tuning would take minutes...

Re: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

#233

Post by thorio »

I challenged Gemini to answer this too, but also got the correct answer.

What came to my mind was: couldn't all LLM vendors easily fund teams that only track these interesting edge cases and quickly deploy filters for these questions, selectively routing to more expensive models?

Isn't that how they probably game benchmarks too?

Re: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

#234

Post by ninjagoo »

I wonder if the providers are doing everyone, themselves included, a huge disservice by providing free versions of their models that are so incompetent compared to the SOTA models that these types of q&a go viral because the ai hype doesn't match the reality for unpaid users.

And it's not just the viral questions that are an issue. I've seen people getting sub-optimal results for $1000+ PC comparisons from the free reasoning version while the paid versions get it right; a senior scientist at a national lab thinking ai isn't really useful because the free reasoning version couldn't generate working code from a scientific paper and then being surprised when the paid version 1-shotted working code, and other similar examples over the last year or so.

How many policy and other quality of life choices are going to go wrong because people used the free versions of these models that got the answers subtly wrong and the users couldn't tell the difference? What will be the collective damage to the world because of this?

Which department or person within the provider orgs made the decision to put thinking/reasoning in the name when clearly the paid versions have far better performance? Thinking about the scope of the damage they are doing makes me shudder.

Re: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

#235

Post by nearbuy »

jaccola wrote: post #156
lkeskull wrote: post #113

Earlier quoted context omitted.

this example worked in 2021, it's 2026. wake up. these models are not just "finding the most likely next word based on what they've seen on the internet".

The person above was being a bit pedantic, and zealous in their anti-anthropomorphism. But they are literally predicting the next token. They do nothing else. Also if you think they were just predicting the next token in 2021, there has been no fundamental architecture change since then. All gains have been via scale and efficiency optimisations (not to discount that, an awful lot of complexity in both of these)

That's not what they said. They said:

> It's evaluation function simply returned the word "Most" as being the most likely first word in similar sentences it was trained on.

Which is false under any reasonable interpretation. They do not just return the word most similar to what they would find in their training data. They apply reasoning and can choose words that are totally unlike anything in their training data.

If you prompt it:

> Complete this sentence in an unexpected way: Mary had a little...

It won't say lamb. Any if you think whatever it says was in the training data, just change the constraints until you're confident it's original. (E.g. tell it every word must start with a vowel and it should mention almonds.)

"Predicting the next token" is also true but misleading. It's predicting tokens in the same sense that your brain is just minimizing prediction error under predictive coding theory.

Re: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

#236

Post by TZubiri »

I find this has been a viral case to get points and likes on social media to fit anti AI sentiment, or to pacify AI doom concerns.

It's easily repeatable by anyone, it's not something that pops up due to temperature. Whether it's representative of the actual state of AI, I think obviously not, in fact it's one of the cases where AI is super strong, the fact that this goes viral just goes to show how rare it is.

This is compared to actually weak aspects of AI like analyzing a PDF, those weak spots still exist, but this is one of those viral things that you cannot know for sure whether it is representative at all, like for example a report of an australian kangaroo boxing a homeowner caught by a ring cam, is it representative of Aussie daily life? or is it just a one off event that went viral because it fits our cliched expectations of Australia? Can't tell from the other part of the world.

Re: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

#237

Post by jstummbillig »

ssl-3 wrote: post #199
jstummbillig wrote: post #147

Earlier quoted context omitted.

> so you need to tell them the specifics That is the entire point, right? Us having to specify things that we would never specify when talking to a human. You would not start with "The car is functional. The tank is filled with gas. I have my keys." As soon as we are required to do that for the model to any extend that is a problem and not a detail (regardless that those of us, who are familiar with the matter, do bu…

The question is so outlandish that it is something that nobody would ever ask another human. But if someone did, then they'd reasonably expect to get a response consisting 100% of snark. But the specificity required for a machine to deliver an apt and snark-free answer is -- somehow -- even more outlandish? I'm not sure that I see it quite that way.

I'd be entirely fine with a humorous response. The Gemini flash answer that was posted somewhere in this thread is delightful.

Re: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

#238

Post by yipbub »

ninjagoo wrote: post #234

I wonder if the providers are doing everyone, themselves included, a huge disservice by providing free versions of their models that are so incompetent compared to the SOTA models that these types of q&a go viral because the ai hype doesn't match the reality for unpaid users. And it's not just the viral questions that are an issue. I've seen people getting sub-optimal results for $1000+ PC comparisons from the free r…

I used a paid model to try this. Same deal.

Re: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

#239

Post by TZubiri »

ninjagoo wrote: post #234

I wonder if the providers are doing everyone, themselves included, a huge disservice by providing free versions of their models that are so incompetent compared to the SOTA models that these types of q&a go viral because the ai hype doesn't match the reality for unpaid users. And it's not just the viral questions that are an issue. I've seen people getting sub-optimal results for $1000+ PC comparisons from the free r…

I don't think 100% adoption is necessarily the ideal strategy anyways. Maybe 50% of the population seeing AI as all powerful and buying the subscription vs 50% of the population still being skeptics, is a reasonable stable configuration. 50% get the advantage of the AI whereas if everybody is super intelligent, no one is super intelligent.

Their loss

Re: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

#240

Post by moffkalast »

yipbub wrote: post #238
ninjagoo wrote: post #234

I wonder if the providers are doing everyone, themselves included, a huge disservice by providing free versions of their models that are so incompetent compared to the SOTA models that these types of q&a go viral because the ai hype doesn't match the reality for unpaid users. And it's not just the viral questions that are an issue. I've seen people getting sub-optimal results for $1000+ PC comparisons from the free r…

I used a paid model to try this. Same deal.

I think the real misleading thing is marketing propping up paid models being somehow infinitely better when most of the time it's the same exact shit.
Post reply on HN