61–70 of 115 posts
Wish there was a bit more technical details in how the prompt iterations looked like. > We didn’t just replace a model. We replaced a process. That line sticks out so much now, and I can't unsee it.
[flagged]
Hmm, why was their starting point not something like BERT: * already known as SotA for text classification and similarity back in 2023 * natively multi-lingual
People generally sleep when you start talking about fine-tuned BERT and CLIP, although they do a fairly decent job as long as you have good data and know what you're doing. But no, they want to pay $0.1 per request to recognize if a photo has a person in it by asking a multimodal LLM deployed across 8x GPUs, for some reason, instead of just spending some hours with CLIP and run it effectively even on CPU.
This is the bottleneck in my experience. Going for the expensive per-request LLM gets something shipped now that you can wow the execs with. Setting up a whole process to gather and annotate data, train models, run evals, and iterate takes time. The execs who hired those expensive AI engineers want their results right now, not after a process of hiring more people to collect and annotate the data.
TLDR: "old" ml techniques like XGBoost can beat LLMs and neural networks for some tasks.
The old model was capable of running on a CPU. The new one requires a GPU. This might be a consideration for some.
Even if it took $10 to run everything to handle each request, that’s far cheaper than even a minimum wage employee when you consider all of the employment overhead.
And this is where the strengths of LLMs really lie: making performant ML available to a wider audience, without requiring PHDs in Computer Science or Mathematics to build. It’s consistently where I spend my time tinkering with these, albeit in a local-only environment. If all the bullshit hype and marketing would evaporate already (“LLMs will replace all jobs!”), stuff like this would float to the top more and compan…
^ This. I'm waiting for an LLM where I can just point it to a repo, slurp it up, and let me ask questions about it.
It helps to give it a little context and suggest where to look in the repo. The tools also have mechanisms where you can leave directions and notes in the context for the project. Updating that over time as you discover where the LLM stumbles helps a lot.
I'll note that they had a large annotated data set already that they were using to train and evaluate their own models. Once they decided to start testing LLMs it was straightforward for them to say "LLM 1 outperforms LLM 2" or "Prompt 3 outperforms Prompt 4". I'm afraid that people will draw the wrong conclusion from "We didn’t just replace a model. We replaced a process." and see it as an endorsement of the zero-sh…
> Thing is that to know it is good enough you still have to collect and annotate more data than most people and organizations want to do. This has been the bottleneck in every ML (not just text/LLM) project I’ve been part of. Not finding the right AI engineers. Not getting the MLops textbook perfect using the latest trends. It’s the collecting enough high quality data and getting it properly annotated and verified. T…
This isn't my first CV project, but it's the most successful one. And that chiefly because my client pulled out their wallets and let an army of annotators create all the train data I asked for, and more.
* "2 years vs 1 month" is a bit misleading because the work that enabled testing the 1 month of prompting was part of the 2 years of ML work.
* xgboost is an ensemble method... add the llm outputs as inputs to xgboost and probably enjoy better results.
* vectorize all the text data points using an embedding model and add those as inputs to xgboost for probably better results.
Being an automaker, I can almost smell the silos where data resides, the rigidly defended lines between manufactures, sales and post-sales, the intra-departmental political fights.
Then you have all the legacy of enterprise software.
And the result is this shitty warranty claims data.
Hmm, why was their starting point not something like BERT: * already known as SotA for text classification and similarity back in 2023 * natively multi-lingual