Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we've learned from a year of building with LLMs

eugeneyan.com

Post reply on HN
Print view View original

41–50 of 148 posts

Re: What we've learned from a year of building with LLMs

#41

Post by fnordpiglet »

dbs wrote: post #17

Show me the use cases you have supported in production. Then I might read all the 30 pages praising the dozens (soon to be hundreds?) of “best practices” to build LLMs.

We use LLMs in dozens of different production applications for critical business flows. They allow for a lot of dynamism in our flows that aren’t amenable to direct quantitative reasoning or structured workflows. Double digit percents of our growth in the last year are entirely due to them. The biggest challenge is tool chain, limits on inference capacity, and developer understanding of the abilities, limits, and techniques for using LLMs effectively.

I often see these messages from the community doubting the reality, but LLMs are a powerful tool in the tool chest. But I think most companies are not staffed with skilled enough engineers with a creative enough bent to really take advantage of them yet or be willing to fund basic research and from first principles toolchain creation. That’s ok. But it’s foolish to assume this is all hype like crypto was. The parallels are obvious but the foundations are different.

Re: What we've learned from a year of building with LLMs

#42

Post by mvdtnz »

fnordpiglet wrote: post #41
dbs wrote: post #17

Show me the use cases you have supported in production. Then I might read all the 30 pages praising the dozens (soon to be hundreds?) of “best practices” to build LLMs.

We use LLMs in dozens of different production applications for critical business flows. They allow for a lot of dynamism in our flows that aren’t amenable to direct quantitative reasoning or structured workflows. Double digit percents of our growth in the last year are entirely due to them. The biggest challenge is tool chain, limits on inference capacity, and developer understanding of the abilities, limits, and tec…

Yet another post claiming "dozens" of production use cases without listing a single one.

Re: What we've learned from a year of building with LLMs

#43

Post by adamsbriscoe »

robbiemitchell wrote: post #30
dbs wrote: post #17

Show me the use cases you have supported in production. Then I might read all the 30 pages praising the dozens (soon to be hundreds?) of “best practices” to build LLMs.

Processing high volumes of unstructured data (text)… we’re using a STAG architecture. - Generate targeted LLM micro summaries of every record (ticket, call, etc.) continually - Use layers of regex, semantic embeddings, and scoring enrichments to identify report rows (pivots on aggregates) worth attention, running on a schedule - Proactively explain each report row by identifying what’s unusual about it and LLM summar…

> Lack of JSON schema restriction is a significant barrier to entry on hooking LLMs up to a multi step process.

(Plug) I shipped a dedicated OpenAI-compatible API for this, jsonmode.com a couple weeks ago and just integrated Groq (they were nice enough to bump up the rate limits) so it's crazy fast. It's a WIP but so far very comparable to JSON output from frontier models, with some bonus features (web crawling etc).

Re: What we've learned from a year of building with LLMs

#44

Post by fnordpiglet »

mvdtnz wrote: post #42
fnordpiglet wrote: post #41

Earlier quoted context omitted.

We use LLMs in dozens of different production applications for critical business flows. They allow for a lot of dynamism in our flows that aren’t amenable to direct quantitative reasoning or structured workflows. Double digit percents of our growth in the last year are entirely due to them. The biggest challenge is tool chain, limits on inference capacity, and developer understanding of the abilities, limits, and tec…

Yet another post claiming "dozens" of production use cases without listing a single one.

I’ve listed plenty in my comment history. I don’t generally feel compelled to trot them all out all the time - I don’t need to “prove” anything and if you think I’m lying that’s your choice. Finally, many of our uses are trade secrets and a significant competitive advantage so I don’t feel the need to disclose them to the world if our competitors don’t believe in the tech. We can keep eating their lunch.

Re: What we've learned from a year of building with LLMs

#45

Post by yeahwhatever10 »

robbiemitchell wrote: post #32
BoorishBears wrote: post #31

Earlier quoted context omitted.

> Lack of JSON schema restriction is a significant barrier to entry on hooking LLMs up to a multi step process. How are you struggling with this, let alone as a significant barrier? JSON adherence with a well thought out schema hasn't been a worry between improved model performance and various grammar based constraint systems in a while. > Another is preventing LLMs from adding intro or conclusion text. Also trivial…

Asking even a top-notch LLM to output well formed JSON simply fails sometimes. And when you’re running LLMs at high volume in the background, you can’t use the best available until the last mile. You work around it with post-processing and retries. But it’s still a bit brittle given how much stuff happens downstream without supervision.

The phrase you want to search is "constrained decoding".

Re: What we've learned from a year of building with LLMs

#46

Post by OutOfHere »

solidasparagus wrote: post #27
OutOfHere wrote: post #24

Earlier quoted context omitted.

A well-trained 8B model will already be over-saturated with information from the start. It will therefore easily forget much old information when fine-tuning it with new materials. It just doesn't have the capacity to take in too much information. Don't get me wrong. I think an 70B or larger model would be worth fine-tuning, especially if it can be grown further with more layers.

> A well-trained 8B model will already be over-saturated with information from the start Any evidence of that that I can look at? This doesn't match what I've seen nor have I heard this from the world-class researchers I have worked with. Would be interested to learn more.

Upon further thought, if fine-tuning involves adding layers, then the initial saturation should not matter. Let's say if an 8B model adds 0.8*2 = 1.6B of new layers for fine-tuning, then with some assumptions, a ballpark is that this could be good for 16 million articles for fine-tuning.

Re: What we've learned from a year of building with LLMs

#47

Post by [deleted] »

fnordpiglet wrote: post #44
mvdtnz wrote: post #42

Earlier quoted context omitted.

Yet another post claiming "dozens" of production use cases without listing a single one.

I’ve listed plenty in my comment history. I don’t generally feel compelled to trot them all out all the time - I don’t need to “prove” anything and if you think I’m lying that’s your choice. Finally, many of our uses are trade secrets and a significant competitive advantage so I don’t feel the need to disclose them to the world if our competitors don’t believe in the tech. We can keep eating their lunch.

[deleted]

Re: What we've learned from a year of building with LLMs

#48

Post by obiefernandez »

dbs wrote: post #17

Show me the use cases you have supported in production. Then I might read all the 30 pages praising the dozens (soon to be hundreds?) of “best practices” to build LLMs.

The book I'm writing is almost finished and is based almost entirely on production use cases: https://leanpub.com/patterns-of-application-development-usin...

Re: What we've learned from a year of building with LLMs

#49

Post by threeseed »

fnordpiglet wrote: post #41
dbs wrote: post #17

Show me the use cases you have supported in production. Then I might read all the 30 pages praising the dozens (soon to be hundreds?) of “best practices” to build LLMs.

We use LLMs in dozens of different production applications for critical business flows. They allow for a lot of dynamism in our flows that aren’t amenable to direct quantitative reasoning or structured workflows. Double digit percents of our growth in the last year are entirely due to them. The biggest challenge is tool chain, limits on inference capacity, and developer understanding of the abilities, limits, and tec…

No one is saying that all of AI is hype. It clearly isn't.

But the facts are that today LLMs are not suitable for use cases that need accurate results. And there is no evidence or research that suggests this is changing anytime soon. Maybe for ever.

There are very strong parallels to crypto in that (a) people are starting with the technology and trying to find problems and (b) there is a cult like atmosphere where non-believers are seen as being anti-progress and anti-technology.

Re: What we've learned from a year of building with LLMs

#50

Post by gandalfgeek »

solidasparagus wrote: post #10
gandalfgeek wrote: post #7

Earlier quoted context omitted.

This was kind of conventional wisdom ("fine tune only when absolutely necessary for your domain", "fine-tuning hurts factuality"), but some recent research (some of which they cite) has actually quantitatively shown that RAG is much preferable to FT for adding domain-specific knowledge to an LLM: - "Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?" https://arxiv.org/abs//2405.05904 - "Fine-Tuning or R…

Thanks, I'll read those more fully. But "knowledge injection" is still pretty narrow to me. Here's an example of a very simple but extremely valuable usecase - taking a model that was trained on language+code and finetuning it on a text-to-DSL task, where the DSL is a custom one you created (and thus isn't in the training data). I would consider that close to infeasible if your only tool is a RAG hammer, but it's a v…

Agree that your use-case is different. The papers above are dealing mostly with adding a domain-specific textual corpus, still answering questions in prose.

"Teaching" the LLM an entirely new language (like a DSL) might actually need fine-tuning, but you can probably build a pretty decent first-cut of your system with n-shot prompts, then fine-tune to get the accuracy higher.

Post reply on HN