Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Honda: 2 years of ml vs 1 month of prompting - heres what we learned

levs.fyi

Post reply on HN
Print view View original

21–30 of 115 posts

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#21

Post by HotHotLava »

Veliladon wrote: post #4
stego-tech wrote: post #3

And this is where the strengths of LLMs really lie: making performant ML available to a wider audience, without requiring PHDs in Computer Science or Mathematics to build. It’s consistently where I spend my time tinkering with these, albeit in a local-only environment. If all the bullshit hype and marketing would evaporate already (“LLMs will replace all jobs!”), stuff like this would float to the top more and compan…

^ This. I'm waiting for an LLM where I can just point it to a repo, slurp it up, and let me ask questions about it.

Basically every AI agent released in the last 6 months can do this pretty well out of the box? What feature exactly are you missing from these?

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#22

Post by ryandvm »

cpursley wrote: post #5
Veliladon wrote: post #4

Earlier quoted context omitted.

^ This. I'm waiting for an LLM where I can just point it to a repo, slurp it up, and let me ask questions about it.

github copilot somewhat does this.

Copilot is too stingy with context. In my experience Claude Code is much better at seeing the big picture.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#24

Post by prasoonds »

jwong_ wrote: post #23

Wish there was a bit more technical details in how the prompt iterations looked like. > We didn’t just replace a model. We replaced a process. That line sticks out so much now, and I can't unsee it.

Right? This one is also very clear ChatGPTese

> That’s not a marginal improvement; it’s a different way of building classifiers.

They've replaced an em-dash with a semi-colon.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#25

Post by efavdb »

embedding-shape wrote: post #17
killerstorm wrote: post #14

Hmm, why was their starting point not something like BERT: * already known as SotA for text classification and similarity back in 2023 * natively multi-lingual

People generally sleep when you start talking about fine-tuned BERT and CLIP, although they do a fairly decent job as long as you have good data and know what you're doing. But no, they want to pay $0.1 per request to recognize if a photo has a person in it by asking a multimodal LLM deployed across 8x GPUs, for some reason, instead of just spending some hours with CLIP and run it effectively even on CPU.

Are you suggesting use the clip embedding for the text as a feature to train a standard Ml model on?

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#26

Post by klabb3 »

prasoonds wrote: post #24
jwong_ wrote: post #23

Wish there was a bit more technical details in how the prompt iterations looked like. > We didn’t just replace a model. We replaced a process. That line sticks out so much now, and I can't unsee it.

Right? This one is also very clear ChatGPTese > That’s not a marginal improvement; it’s a different way of building classifiers. They've replaced an em-dash with a semi-colon.

They are really getting to the heart of the problem!

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#27

Post by suriya-ganesh »

yahoozoo wrote: post #2

I wonder if text embeddings and semantic similarity would be effective here?

I have. For a similar-ish task.

LLMs still beat a clarifier, because they're able to extract more signals than a text embedding.

It's very difficult to beat an LLM + prompt in terms of semantic extraction.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#28

Post by PaulHoule »

efavdb wrote: post #25
embedding-shape wrote: post #17

Earlier quoted context omitted.

People generally sleep when you start talking about fine-tuned BERT and CLIP, although they do a fairly decent job as long as you have good data and know what you're doing. But no, they want to pay $0.1 per request to recognize if a photo has a person in it by asking a multimodal LLM deployed across 8x GPUs, for some reason, instead of just spending some hours with CLIP and run it effectively even on CPU.

Are you suggesting use the clip embedding for the text as a feature to train a standard Ml model on?

I think he is. I do things like that plenty.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#29

Post by PaulHoule »

I'll note that they had a large annotated data set already that they were using to train and evaluate their own models. Once they decided to start testing LLMs it was straightforward for them to say "LLM 1 outperforms LLM 2" or "Prompt 3 outperforms Prompt 4".

I'm afraid that people will draw the wrong conclusion from "We didn’t just replace a model. We replaced a process." and see it as an endorsement of the zero-shot-uber-alles "Prompt and Pray" approach that is dominant in the industry right now and the reason why an overwhelming faction of AI projects fail.

If you can get good enough performance out of zero shot then yeah, zero shot is fine. Thing is that to know it is good enough you still have to collect and annotate more data than most people and organizations want to do.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#30

Post by frikk »

nmfisher wrote: post #9
Veliladon wrote: post #4

Earlier quoted context omitted.

^ This. I'm waiting for an LLM where I can just point it to a repo, slurp it up, and let me ask questions about it.

$ git clone repo && cd repo $ claude Ask away. Best method I’ve found so far for this.

This technique is surprisingly powerful. Yesterday I built an experimental cellular automata classifier system based on some research papers I found and was curious about. Aside from the sheer magic of the entire build process with Cursor + GPT5-Codex, one big breakthrough was simply cloning the original repo's source code and copy/pasting the paper into a .txt file.

Now when I ask questions about design decisions, the LLM refers to the original paper and cites the decisions without googling or hallucinating.

With just these two things in my local repo, the LLM created test scripts to compare our results versus the paper and fixed bugs automatically, helped me make decisions based on the paper's findings, helped me tune parameters based on the empirical outcomes, and even discovered a critical bug in our code that was caused by our training data being random generated versus the paper's training data being a permutation over the whole solution space.

All of this work was done in one evening and I'm still blown away by it. We even ported our code to golang, parallelized it, and saw a 10x speedup in the processing. Right before heading to bed, I had the LLM spin up a novel simulator using a quirky set of tests that I invented using hypothetical sensors and data that have not yet been implemented, and it nailed it first try - using smart abstractions and not touching the original engine implementation at all. This tech is getting freaky.

Post reply on HN