81–90 of 115 posts
Wish there was a bit more technical details in how the prompt iterations looked like. > We didn’t just replace a model. We replaced a process. That line sticks out so much now, and I can't unsee it.
[flagged]
It had high quality info about a large ML effort inside an old school auto company, which is very interesting. I was just a bit disappointed no one thought to edit those out.
Crucially, this is: - text classification, not text generation - operating on existing unstructured input - existing solution was extremely limited (string matching) - comparing LLM to similar but older methods of using neural networks to match - seemingly no negative consequences to warranty customers themselves of mis-classification (the data is used to improve process, not to make decisions)
But I think it's still an interesting result, because related and similar tasks are everywhere in our modern world, and they tend to have high importance in both business and the public sector, and the older generation of machine learning techniques for handling these tasks we're both sophisticated and to the point where very capable and experienced practitioners might need an R&D cycle just to conclude if the problem was solvable with the available data up to the desired standard.
LLM's represent a tremendous advancement in our ability as a society to deal with these kinds of tasks. So yes, it's a limited range of specific tasks, and success is found within a limited set of criteria, but it's a very important tasks and enough of those criteria are met in practice that I think this result is interesting and generalizable.
That doesn't mean we should fire all of our data scientists and let junior programmers just have at it with the LLM, because you still need to put together a good day to say, makes sense of the results, and iterate intelligently, especially given that these models tend to be expensive to run. It does however mean that existing data teams must be open to adopting LLMs instead of traditional model fitting.
Wish there was a bit more technical details in how the prompt iterations looked like. > We didn’t just replace a model. We replaced a process. That line sticks out so much now, and I can't unsee it.
Wish there was a bit more technical details in how the prompt iterations looked like. > We didn’t just replace a model. We replaced a process. That line sticks out so much now, and I can't unsee it.
[flagged]
Wish there was a bit more technical details in how the prompt iterations looked like. > We didn’t just replace a model. We replaced a process. That line sticks out so much now, and I can't unsee it.
I think we're getting into reverse slop discrimination territory now. LLMs have been trained on so much of what we consider "good writing", that actual good writing is now attributed by default to LLMs.
Hmm, why was their starting point not something like BERT: * already known as SotA for text classification and similarity back in 2023 * natively multi-lingual
People generally sleep when you start talking about fine-tuned BERT and CLIP, although they do a fairly decent job as long as you have good data and know what you're doing. But no, they want to pay $0.1 per request to recognize if a photo has a person in it by asking a multimodal LLM deployed across 8x GPUs, for some reason, instead of just spending some hours with CLIP and run it effectively even on CPU.
I think you've just identified, in a set-theoretic complementary manner, the TAM for GenAI.
Earlier quoted context omitted.
> We tried multiple vectorization and classification approaches. Our data was heavily imbalanced and skewed towards negative cases. We found that TF-IDF with 1-gram features paired with XGBoost consistently emerged as the winner.
Anthropic found a similar result for retrieval: embeddings + BM25 keyword search (variant of TF-IDF) produced significantly better results. https://www.anthropic.com/engineering/contextual-retrieval They also found improvements from augmenting the chunks with Haiku by having it add a summary based on extra context. That seems to benefit both the keyword search and the embeddings by acting as keyword expansion. (Thoug…
And, agreed, each individual technique seems marginal but they really add up. What seems to be missing is some automated layer that determines the best way to chunk documents into embeddings. My use case is mostly normalized mostly technical documents so I have a pretty clear idea of how to chunk to preserve semantics. But I imagine that for generalized documents it is a lot trickier.
Earlier quoted context omitted.
True now. At the same time, as a nonnative speaker of English, this is literally how we were taught to write eye-catching articles and phrases. :P A lot of formulaic writing is what we were taught to do, especially with more formal things. (This is more of a sidenote to this example) So in a hunt for LLMs, we also get hit.
I believe this is likely a consequence of how RLHF is done. I’ve not verified it, but I suspect the frontier model labs are outsourcing it to companies employing primarily non-native English speakers.
I learned it from MtG and I do believe it's a very cool word and I hate that I can't use it without people raising their eyebrows.