Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Honda: 2 years of ml vs 1 month of prompting - heres what we learned

levs.fyi

Post reply on HN
Print view View original

41–50 of 115 posts

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#41

Post by StefanBatory »

Aniket-N wrote: post #36

Once you start to recognize AI written, rewritten or even edited articles, it’s hard to stop. It’s not X it’s Y. We didn’t just do A we did B. There’s definitely a lot of hard work that has gone in here. It’s gotten hard to read because of these sentence patterns popping up everywhere.

True now.

At the same time, as a nonnative speaker of English, this is literally how we were taught to write eye-catching articles and phrases. :P

A lot of formulaic writing is what we were taught to do, especially with more formal things. (This is more of a sidenote to this example)

So in a hunt for LLMs, we also get hit.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#42

Post by cestith »

embedding-shape wrote: post #17
killerstorm wrote: post #14

Hmm, why was their starting point not something like BERT: * already known as SotA for text classification and similarity back in 2023 * natively multi-lingual

People generally sleep when you start talking about fine-tuned BERT and CLIP, although they do a fairly decent job as long as you have good data and know what you're doing. But no, they want to pay $0.1 per request to recognize if a photo has a person in it by asking a multimodal LLM deployed across 8x GPUs, for some reason, instead of just spending some hours with CLIP and run it effectively even on CPU.

I’m no ML engineer and far from an LLM expert. Just reading the article though it seemed to me that leveraging an SQL database here was a bigger issue than using traditional ML on the data, rather than the LLM being a win specifically. Just finding anything that was better suited than string matching on a RDBMS to the type of inputs seems like the natural conclusion when the complaint in the article itself was literally about SQL.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#43

Post by chanux »

Aniket-N wrote: post #36

Once you start to recognize AI written, rewritten or even edited articles, it’s hard to stop. It’s not X it’s Y. We didn’t just do A we did B. There’s definitely a lot of hard work that has gone in here. It’s gotten hard to read because of these sentence patterns popping up everywhere.

That example sentence read more like a part of a LinkedIn post.

... Wait a minute!

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#44

Post by cestith »

suriya-ganesh wrote: post #27
yahoozoo wrote: post #2

I wonder if text embeddings and semantic similarity would be effective here?

I have. For a similar-ish task. LLMs still beat a clarifier, because they're able to extract more signals than a text embedding. It's very difficult to beat an LLM + prompt in terms of semantic extraction.

Is it hard to beat on performance for the cost, though?

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#45

Post by tengbretson »

Aniket-N wrote: post #36

Once you start to recognize AI written, rewritten or even edited articles, it’s hard to stop. It’s not X it’s Y. We didn’t just do A we did B. There’s definitely a lot of hard work that has gone in here. It’s gotten hard to read because of these sentence patterns popping up everywhere.

You're absolutely right

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#46

Post by notanastronaut »

prasoonds wrote: post #24
jwong_ wrote: post #23

Wish there was a bit more technical details in how the prompt iterations looked like. > We didn’t just replace a model. We replaced a process. That line sticks out so much now, and I can't unsee it.

Right? This one is also very clear ChatGPTese > That’s not a marginal improvement; it’s a different way of building classifiers. They've replaced an em-dash with a semi-colon.

One of the benefits of being immersed in model usage is being able to spot it in the wild from a mile away. People really hate when you catch them doing it and call them out for it.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#47

Post by Ancapistani »

StefanBatory wrote: post #41
Aniket-N wrote: post #36

Once you start to recognize AI written, rewritten or even edited articles, it’s hard to stop. It’s not X it’s Y. We didn’t just do A we did B. There’s definitely a lot of hard work that has gone in here. It’s gotten hard to read because of these sentence patterns popping up everywhere.

True now. At the same time, as a nonnative speaker of English, this is literally how we were taught to write eye-catching articles and phrases. :P A lot of formulaic writing is what we were taught to do, especially with more formal things. (This is more of a sidenote to this example) So in a hunt for LLMs, we also get hit.

I believe this is likely a consequence of how RLHF is done. I’ve not verified it, but I suspect the frontier model labs are outsourcing it to companies employing primarily non-native English speakers.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#50

Post by PaulHoule »

esafak wrote: post #48

The old model was capable of running on a CPU. The new one requires a GPU. This might be a consideration for some.

Honda probably spends $100-$10,000 on a warranty claim in terms of technician time and parts. [1] Even at the low end they can afford to spend 10 cents on an LLM to analyze a claim.

[1] specifically https://www.warrantyweek.com/archive/ww20230817.html claims the expectation value of warranty claims for a car is around $650.

Post reply on HN