Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we've learned from a year of building with LLMs

eugeneyan.com

Post reply on HN
Print view View original

51–60 of 148 posts

Re: What we've learned from a year of building with LLMs

#51

Post by fnordpiglet »

threeseed wrote: post #49
fnordpiglet wrote: post #41

Earlier quoted context omitted.

We use LLMs in dozens of different production applications for critical business flows. They allow for a lot of dynamism in our flows that aren’t amenable to direct quantitative reasoning or structured workflows. Double digit percents of our growth in the last year are entirely due to them. The biggest challenge is tool chain, limits on inference capacity, and developer understanding of the abilities, limits, and tec…

No one is saying that all of AI is hype. It clearly isn't. But the facts are that today LLMs are not suitable for use cases that need accurate results. And there is no evidence or research that suggests this is changing anytime soon. Maybe for ever. There are very strong parallels to crypto in that (a) people are starting with the technology and trying to find problems and (b) there is a cult like atmosphere where no…

Yeah I think a key is LLMs in business are not generally useful alone. They require classical computing techniques to really be powerful. Accurate computation is a generally well established field and you don’t need an LLM to do optimization or math or even deductive logical reasoning. That’s a waste of their power which is typically abstract semantic abductive “reasoning” and natural language processing. Overlaying this with constraints, structure, and augmenting with optimizers, solvers, etc, you get a form of computing that was impossible more than 5 years prior and is only practical in the last 9 months.

On the crypto stuff yeah I get it - especially if you’re not in the weeds of its use. A lot of people formed opinions from GPT3.5, Gemini, copilot, and other crappy experiences and haven’t kept up with the state of the art. The rate of change in AI is breathtaking and I think hard to comprehend for most people. Also the recent mess of crypto and the fact grifters grift etc also hurts. But people who doubt -are- stuck in the past. That’s not necessarily their fault and it might not even apply to their career or lives in the present and the flaws are enormous as you point out. But it’s such a remarkably powerful new mode of compute that it in combination with all the other powerful modes of compute is changing everything and will continue too, especially if next generation models keep improving as they seem to be likely to.

Re: What we've learned from a year of building with LLMs

#52

Post by BoorishBears »

robbiemitchell wrote: post #32
BoorishBears wrote: post #31

Earlier quoted context omitted.

> Lack of JSON schema restriction is a significant barrier to entry on hooking LLMs up to a multi step process. How are you struggling with this, let alone as a significant barrier? JSON adherence with a well thought out schema hasn't been a worry between improved model performance and various grammar based constraint systems in a while. > Another is preventing LLMs from adding intro or conclusion text. Also trivial…

Asking even a top-notch LLM to output well formed JSON simply fails sometimes. And when you’re running LLMs at high volume in the background, you can’t use the best available until the last mile. You work around it with post-processing and retries. But it’s still a bit brittle given how much stuff happens downstream without supervision.

The best available actually have the fewest knobs for JSON schema enforcement (ie. OpenAI's JSON mode, which technically can still produce incorrect JSON)

If you're using anything less you should have a grammar that enforces exactly what tokens are allowed to be output. Fine Tuning can help too in case you're worried about the effects of constraining the generation, but in my experience it's not really a thing

Re: What we've learned from a year of building with LLMs

#53

Post by bgrainger »

sheepscreek wrote: post #39

I’m sure this has some decent insights but it’s from almost 1 year ago! A lot has changed in this space since then.

Are you sure? The article says "cite this as Yan et al. (May 2024)" and published-time in the metadata is 2024-05-12.

Weird: I just refreshed the page and it now redirects to a different domain (than the originally-submitted URL) and has a date of June 8, 2023. It still cites articles and blog posts from 2024, though.

Re: What we've learned from a year of building with LLMs

#54

Post by jph00 »

bgrainger wrote: post #53
sheepscreek wrote: post #39

I’m sure this has some decent insights but it’s from almost 1 year ago! A lot has changed in this space since then.

Are you sure? The article says "cite this as Yan et al. (May 2024)" and published-time in the metadata is 2024-05-12. Weird: I just refreshed the page and it now redirects to a different domain (than the originally-submitted URL) and has a date of June 8, 2023. It still cites articles and blog posts from 2024, though.

Looks like they made a mistake in the article metadata - they definitely just released this article.

Re: What we've learned from a year of building with LLMs

#55

Post by altdataseller »

solidasparagus wrote: post #20
CuriouslyC wrote: post #11

Earlier quoted context omitted.

Fine tuning has been on the way out for a while. It's hard to do right and costly. LoRAs are better for influencing output style as they don't dumb down the model, and they're easier to create. This is on top of RAG just being better for new facts like the other reply mentioned.

How much of that is just the flood of traditional engineers into the space and the fact that collecting data and then fine-tuning models is orders of magnitude more complex than just throwing in RAG? I suspect a huge amount of RAG's popularity is just that any engineer can do a version of it + ChatGPT API calls in a day. As for lora - in the context of my comment, that's just splitting hairs IMO. It falls in the cate…

I disagree. No amount of fine tuning will ever give the LLM the relevant context with which to answer my question. Maybe if your context is a static Wikipedia or something that will never change, you can fine tune it. But if your data and docs keep changing, how is fine tuning going to be better than RAG?

Re: What we've learned from a year of building with LLMs

#56

Post by solidasparagus »

altdataseller wrote: post #55
solidasparagus wrote: post #20

Earlier quoted context omitted.

How much of that is just the flood of traditional engineers into the space and the fact that collecting data and then fine-tuning models is orders of magnitude more complex than just throwing in RAG? I suspect a huge amount of RAG's popularity is just that any engineer can do a version of it + ChatGPT API calls in a day. As for lora - in the context of my comment, that's just splitting hairs IMO. It falls in the cate…

I disagree. No amount of fine tuning will ever give the LLM the relevant context with which to answer my question. Maybe if your context is a static Wikipedia or something that will never change, you can fine tune it. But if your data and docs keep changing, how is fine tuning going to be better than RAG?

Continuous retraining and deployment maybe? But I'm actually not anti-RAG (although I think it is overrated because the retrieval problem is still handled extremely naively), I just think that fine-tuning should also be in your toolkit.

Re: What we've learned from a year of building with LLMs

#57

Post by jph00 »

jph00 wrote: post #54
bgrainger wrote: post #53

Earlier quoted context omitted.

Are you sure? The article says "cite this as Yan et al. (May 2024)" and published-time in the metadata is 2024-05-12. Weird: I just refreshed the page and it now redirects to a different domain (than the originally-submitted URL) and has a date of June 8, 2023. It still cites articles and blog posts from 2024, though.

Looks like they made a mistake in the article metadata - they definitely just released this article.

OK I let them know, and they've fixed it now.

Re: What we've learned from a year of building with LLMs

#58

Post by janpieterz »

jncfhnb wrote: post #40
robbiemitchell wrote: post #32

Earlier quoted context omitted.

Asking even a top-notch LLM to output well formed JSON simply fails sometimes. And when you’re running LLMs at high volume in the background, you can’t use the best available until the last mile. You work around it with post-processing and retries. But it’s still a bit brittle given how much stuff happens downstream without supervision.

… why would you have the LLM spit out a json rather than define the json yourself and have the LLM supply values?

How would I do this reliably? Eg give me 10 different values, all in one prompt for performance reasons?

Might not need JSON but whatever format it outputs, it needs to be reliable.

Re: What we've learned from a year of building with LLMs

#59

Post by jph00 »

solidasparagus wrote: post #2

No offense, but I'd love to see what they've successfully built using LLMs before taking their advice too seriously. The idea that fine-tuning isn't even a consideration (perhaps even something they think is absolutely incorrect if the section titles of the unfinished section is anything to go by) is very strange to me and suggests a pretty narrow perspective IMO

> The idea that fine-tuning isn't even a consideration (perhaps even something they think is absolutely incorrect if the section titles of the unfinished section is anything to go by) is very strange to me and suggests a pretty narrow perspective IMO

The article has a section called "When to finetune", along with links to separate pages describing how to do so. They absolutely don't say that "fine-tuning isn't even a consideration". Instead, they describe the situations in which fine-tuning is likely to be helpful.

Re: What we've learned from a year of building with LLMs

#60

Post by mloncode »

This is Hamel, one of the authors of the article. We published the article with OReilly here:

Part 1: https://www.oreilly.com/radar/what-we-learned-from-a-year-of... Part 2: https://www.oreilly.com/radar/what-we-learned-from-a-year-of...

We were working on this webpage to collect the entire three part article in one place (the third part isn't published yet). We didn't expect anyone to notice the site! Either way, part 3 should be out in a week or so.

Post reply on HN