Live data from Hacker News

Honda: 2 years of ml vs 1 month of prompting - heres what we learned

levs.fyi

11–20 of 115 posts

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#11
post #2

I wonder if text embeddings and semantic similarity would be effective here?

> We tried multiple vectorization and classification approaches. Our data was heavily imbalanced and skewed towards negative cases. We found that TF-IDF with 1-gram features paired with XGBoost consistently emerged as the winner.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#12
post #10

This was fun to read “ Fun fact: Translating French and Spanish claims into German first improved technical accuracy—an unexpected perk of Germany’s automotive dominance.”

I wonder how they came up with that. Was it a human idea, or did the AI stumble upon it?

Given that it was inside a 9-step text preprocessing pipeline, it would be surprising if the AI had that much autonomy.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#15
post #2

I wonder if text embeddings and semantic similarity would be effective here?

Yeah I’m curious if they tried training a Bert or similar classifier… intuitively this seems better than tfidf which is throwing away a ton of information.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#16
post #2

I wonder if text embeddings and semantic similarity would be effective here?

> We tried multiple vectorization and classification approaches. Our data was heavily imbalanced and skewed towards negative cases. We found that TF-IDF with 1-gram features paired with XGBoost consistently emerged as the winner.

Well, "vectorization" can be anything. BERT is in same capability class as GPT, very different from LSA people did in 1980s...

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#17

Hmm, why was their starting point not something like BERT: * already known as SotA for text classification and similarity back in 2023 * natively multi-lingual

People generally sleep when you start talking about fine-tuned BERT and CLIP, although they do a fairly decent job as long as you have good data and know what you're doing.

But no, they want to pay $0.1 per request to recognize if a photo has a person in it by asking a multimodal LLM deployed across 8x GPUs, for some reason, instead of just spending some hours with CLIP and run it effectively even on CPU.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#18

Did the author exactly define "Nova Lite" somewhere in there?

It's the Amazon own model. I'm baffled someone would pick it, even more that someone would test Llama 4 for a task in an age where Sonnet 4.5 is already out, so in the last 45 days.

Looks like they were limited by AWS Bedrock options.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#19
post #10

This was fun to read “ Fun fact: Translating French and Spanish claims into German first improved technical accuracy—an unexpected perk of Germany’s automotive dominance.”

I wonder how they came up with that. Was it a human idea, or did the AI stumble upon it? Given that it was inside a 9-step text preprocessing pipeline, it would be surprising if the AI had that much autonomy.

I think it's fairly known among "LLM practitioners" (or what to call it), that some languages are better at solving specific tasks. Generally if you find yourself in a domain dominated by research in language X, shifting your prompts to that language will give you better results.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#20
Warranty data is a great example of where LLMs have evolved bureaucratic data overhead. What most people do not know is because of US federal TREAD regulation Automotive companies (If they want to land and look at warranty data) need to review all warranty claims, document, and detect any safety related issues and issue recalls all with an strong auditability requirement. This problem generates huge data and operations overhead, Companies need to either hire 10's if not hundreds of individuals to inspect claims or come up with automation to make this process easier.

Over the past couple of years people have made attempts with NLP (lets say standard ML workflows) but NLP and word temperature scores are hard to integrate into a reliable data pipeline much less a operational review workflow.

Enter LLM's, the world is a data gurus oyster for building an detection system on warranty claims. Passing data to Prompted LLM's means capturing and classifying records becomes significantly easier, and these data applications can flow into more normal analytic work streams.

Post reply on HN