71–80 of 115 posts
Earlier quoted context omitted.
Right? This one is also very clear ChatGPTese > That’s not a marginal improvement; it’s a different way of building classifiers. They've replaced an em-dash with a semi-colon.
They are really getting to the heart of the problem!
/s if it wasn't obvious
“cut-chip” usually describes a fault where the engine cuts out briefly—as if someone flicked the ignition off for a split second—and the driver hears or feels a little “chip” or sharp interruption in power.i'm curious about some kind of notion of "prompt overfitting." it's good to see the plots of improvement as the prompts change (although error bars probably would make sense here), but there's not much mention of hold out sets or other approaches to mitigate those concerns.
And yet, the source problem still remains. The company has a shitty way of reporting quality issues in relation to parts and assemblies. Being an automaker, I can almost smell the silos where data resides, the rigidly defended lines between manufactures, sales and post-sales, the intra-departmental political fights. Then you have all the legacy of enterprise software. And the result is this shitty warranty claims dat…
I'll note that they had a large annotated data set already that they were using to train and evaluate their own models. Once they decided to start testing LLMs it was straightforward for them to say "LLM 1 outperforms LLM 2" or "Prompt 3 outperforms Prompt 4". I'm afraid that people will draw the wrong conclusion from "We didn’t just replace a model. We replaced a process." and see it as an endorsement of the zero-sh…
> Thing is that to know it is good enough you still have to collect and annotate more data than most people and organizations want to do. This has been the bottleneck in every ML (not just text/LLM) project I’ve been part of. Not finding the right AI engineers. Not getting the MLops textbook perfect using the latest trends. It’s the collecting enough high quality data and getting it properly annotated and verified. T…
In fact there are companies such as Medallia which specialize in CX and have really strong classification solutions for specifically these use cases (plus all the generative AI stuff for closing the loop).
And yet, the source problem still remains. The company has a shitty way of reporting quality issues in relation to parts and assemblies. Being an automaker, I can almost smell the silos where data resides, the rigidly defended lines between manufactures, sales and post-sales, the intra-departmental political fights. Then you have all the legacy of enterprise software. And the result is this shitty warranty claims dat…
Warranty data flows up from the technicians - good luck getting any auto technician to properly tag data. Their job is to fix a specific customer’s problem, not identify systematic issues.
There’s a million things that make the data inherently messy. For example, a technician might replace 5 parts before they finally identify the root cause.
Therefore, you need some sort of department to sit between millions of raw claims and engineering. I would be curious what kind of alternative you have in mind?
Earlier quoted context omitted.
> Thing is that to know it is good enough you still have to collect and annotate more data than most people and organizations want to do. This has been the bottleneck in every ML (not just text/LLM) project I’ve been part of. Not finding the right AI engineers. Not getting the MLops textbook perfect using the latest trends. It’s the collecting enough high quality data and getting it properly annotated and verified. T…
Amen brother. Working on a computer vision project right now, it's a wild success. This isn't my first CV project, but it's the most successful one. And that chiefly because my client pulled out their wallets and let an army of annotators create all the train data I asked for, and more.
Supervised learning. Took a while to make that work well.
And then every few years someone comes up with a way to distill data out of unsupervised examples. GPT is these days the big example of that, but there was "ImageNet (unlabeled)" and LAION before that too. The issue is that there is just so much unsupervised data.
Now LLMs use that pretty well (even though stuffing everything into an LLM is getting old, and as this article points out, in any specific application they tend to get bested by something like XGBoost with very simple models)
The next frontier is probably "world models", where you first train unsupervised, not to train your model but to predict the world. THEN you train the model in this simulated, predicted world. That's the reason Yann Lecun really really wants to go down this direction.
Earlier quoted context omitted.
> Thing is that to know it is good enough you still have to collect and annotate more data than most people and organizations want to do. This has been the bottleneck in every ML (not just text/LLM) project I’ve been part of. Not finding the right AI engineers. Not getting the MLops textbook perfect using the latest trends. It’s the collecting enough high quality data and getting it properly annotated and verified. T…
There was a post on here recently about how you should build your own agent, and I completely agree. I'd say most competent developers should be building even more complex projects than an agent. Once you do you quickly realize how it's a constant uphill battle, and it quickly becomes apparent that the data you're working with is the primary issue.