Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Honda: 2 years of ml vs 1 month of prompting - heres what we learned

levs.fyi

Post reply on HN
Print view View original

101–110 of 115 posts

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#101

Post by throwaway314155 »

keeda wrote: post #87
embedding-shape wrote: post #17

Earlier quoted context omitted.

People generally sleep when you start talking about fine-tuned BERT and CLIP, although they do a fairly decent job as long as you have good data and know what you're doing. But no, they want to pay $0.1 per request to recognize if a photo has a person in it by asking a multimodal LLM deployed across 8x GPUs, for some reason, instead of just spending some hours with CLIP and run it effectively even on CPU.

>... as long as you have good data and know what you're doing. I think you've just identified, in a set-theoretic complementary manner, the TAM for GenAI.

What's TAM?

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#102

Post by greazy »

I found this fun fact really fascinating:

> Translating French and Spanish claims into German first improved technical accuracy—an unexpected perk of Germany’s automotive dominance.

It brings up an interesting idea that some languages are better suited for different domains.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#104

Post by keeda »

throwaway314155 wrote: post #101
keeda wrote: post #87

Earlier quoted context omitted.

>... as long as you have good data and know what you're doing. I think you've just identified, in a set-theoretic complementary manner, the TAM for GenAI.

What's TAM?

https://en.wikipedia.org/wiki/Total_addressable_market

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#105

Post by deepsquirrelnet »

killerstorm wrote: post #14

Hmm, why was their starting point not something like BERT: * already known as SotA for text classification and similarity back in 2023 * natively multi-lingual

I love using encoder models, and they are generally a better technology for this kind of application. But the price of GPU instances is too damn high.

I won’t lie that I’ve been unreasonably annoyed that I have to use a lot more compute than I need, for no other reason than an LLM API exists and it’s good enough in a relatively small throughput application.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#106

Post by zahlman »

beepbooptheory wrote: post #80
richardlblair wrote: post #76

Earlier quoted context omitted.

There was a post on here recently about how you should build your own agent, and I completely agree. I'd say most competent developers should be building even more complex projects than an agent. Once you do you quickly realize how it's a constant uphill battle, and it quickly becomes apparent that the data you're working with is the primary issue.

I don't know if that is what gp and above is talking about. "Agents" are the kind of thing/word that helps to paper over the very fact that these things only work because of huge amount of humans in-the-loop in the outset (that is, you know, labor). Agents help us believe that LLM's can do everything for us, even bootstrap themselves, but, what the above thread is about is that, really, what you get out correlates on…

> Agents help us believe that LLM's can do everything for us, even bootstrap themselves

Having the agent, and treating it carelessly, helps one believe this.

Making it is another story.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#107

Post by jacquesm »

spwa4 wrote: post #79
isoprophlex wrote: post #66

Earlier quoted context omitted.

Amen brother. Working on a computer vision project right now, it's a wild success. This isn't my first CV project, but it's the most successful one. And that chiefly because my client pulled out their wallets and let an army of annotators create all the train data I asked for, and more.

This has been the huge problem in AI research since at least 1998 (and that was just when I was first exposed to it). With data, everything is so much easier, and much simpler machine learning methods. Supervised learning. Took a while to make that work well. And then every few years someone comes up with a way to distill data out of unsupervised examples. GPT is these days the big example of that, but there was "Ima…

> Now LLMs use that pretty well (even though stuffing everything into an LLM is getting old, and as this article points out, in any specific application they tend to get bested by something like XGBoost with very simple models)

You can't blame the users for that though, for instance, OpenAI's ChatGPT uses 'Ask Anything' as their home page prompt. Zero specialization, expert at anything. And people totally believe it.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#108

Post by jacquesm »

ghm2180 wrote: post #96
PaulHoule wrote: post #29

I'll note that they had a large annotated data set already that they were using to train and evaluate their own models. Once they decided to start testing LLMs it was straightforward for them to say "LLM 1 outperforms LLM 2" or "Prompt 3 outperforms Prompt 4". I'm afraid that people will draw the wrong conclusion from "We didn’t just replace a model. We replaced a process." and see it as an endorsement of the zero-sh…

I would offer a stronger more pointed observation, ofen the problem in building a good classifier is having good negative examples. More generally how a classifier identify good negatives is a function of: 1. Data collection technique. 2. Data annotation(labelling). 3. Classfier can learn on your "good" negatives — quantitaively depending on the machine residuals/margin/contrastive/triplet losses — i.e. learn the dif…

I think the biggest problem with such classifiers is to actually know what is good data and what is bad data. To take a sample of the data and to recognize whether or not this dataset is a general enough representation of both true and false examples (for a binary classifier) to be able to use it to train a model. Because it isn't rare at all to have data sets that are biased 100 to 1 or more for one of the classes, which contain hints about what class the object is in that isn't in the object itself and so on. You can train until the cows come home on such data but it will never lead to satisfactory results.

Re: Honda: 2 years of ml vs 1 month of prompting - heres what we learned

#109

Post by scrame »

PaulHoule wrote: post #29

I'll note that they had a large annotated data set already that they were using to train and evaluate their own models. Once they decided to start testing LLMs it was straightforward for them to say "LLM 1 outperforms LLM 2" or "Prompt 3 outperforms Prompt 4". I'm afraid that people will draw the wrong conclusion from "We didn’t just replace a model. We replaced a process." and see it as an endorsement of the zero-sh…

> an overwhelming faction of AI projects fail.

An overwhelming amount of software projects fail, AI just helps them get there faster.

Post reply on HN