Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we've learned from a year of building with LLMs

eugeneyan.com

Post reply on HN
Print view View original

31–40 of 148 posts

Re: What we've learned from a year of building with LLMs

#31

Post by BoorishBears »

robbiemitchell wrote: post #30
dbs wrote: post #17

Show me the use cases you have supported in production. Then I might read all the 30 pages praising the dozens (soon to be hundreds?) of “best practices” to build LLMs.

Processing high volumes of unstructured data (text)… we’re using a STAG architecture. - Generate targeted LLM micro summaries of every record (ticket, call, etc.) continually - Use layers of regex, semantic embeddings, and scoring enrichments to identify report rows (pivots on aggregates) worth attention, running on a schedule - Proactively explain each report row by identifying what’s unusual about it and LLM summar…

> Lack of JSON schema restriction is a significant barrier to entry on hooking LLMs up to a multi step process.

How are you struggling with this, let alone as a significant barrier? JSON adherence with a well thought out schema hasn't been a worry between improved model performance and various grammar based constraint systems in a while.

> Another is preventing LLMs from adding intro or conclusion text.

Also trivial to work around by pre-filling and stop tokens, or just extremely basic text parsing.

Also would recommend writing out Stream-Triggered Augmented Generation since the term is so barely used it might as well be made up from the POV of someone trying to understand the comment

Re: What we've learned from a year of building with LLMs

#32

Post by robbiemitchell »

BoorishBears wrote: post #31
robbiemitchell wrote: post #30

Earlier quoted context omitted.

Processing high volumes of unstructured data (text)… we’re using a STAG architecture. - Generate targeted LLM micro summaries of every record (ticket, call, etc.) continually - Use layers of regex, semantic embeddings, and scoring enrichments to identify report rows (pivots on aggregates) worth attention, running on a schedule - Proactively explain each report row by identifying what’s unusual about it and LLM summar…

> Lack of JSON schema restriction is a significant barrier to entry on hooking LLMs up to a multi step process. How are you struggling with this, let alone as a significant barrier? JSON adherence with a well thought out schema hasn't been a worry between improved model performance and various grammar based constraint systems in a while. > Another is preventing LLMs from adding intro or conclusion text. Also trivial…

Asking even a top-notch LLM to output well formed JSON simply fails sometimes. And when you’re running LLMs at high volume in the background, you can’t use the best available until the last mile.

You work around it with post-processing and retries. But it’s still a bit brittle given how much stuff happens downstream without supervision.

Re: What we've learned from a year of building with LLMs

#33

Post by benreesman »

robbiemitchell wrote: post #30
dbs wrote: post #17

Show me the use cases you have supported in production. Then I might read all the 30 pages praising the dozens (soon to be hundreds?) of “best practices” to build LLMs.

Processing high volumes of unstructured data (text)… we’re using a STAG architecture. - Generate targeted LLM micro summaries of every record (ticket, call, etc.) continually - Use layers of regex, semantic embeddings, and scoring enrichments to identify report rows (pivots on aggregates) worth attention, running on a schedule - Proactively explain each report row by identifying what’s unusual about it and LLM summar…

I only became aware of it recently and therefore haven’t done more than play with in a fairly cursory way, but unstructured.io seems to have a lot of traction and certainly in my little toy tests their open-source stuff seems pretty clearly better than the status quo.

Might be worth checking out.

Re: What we've learned from a year of building with LLMs

#34

Post by threeseed »

RAGs do not prevent hallucinations nor does it guarantee that the quality of your output is contingent solely on the quality of your input. Using LLMs for legal use cases for example has shown it to be poor for anything other than initial research as it is accurate at best 65%:

https://dho.stanford.edu/wp-content/uploads/Legal_RAG_Halluc...

So would strongly disagree that LLMs have become “good enough” for real-world applications" based on what was promised.

Re: What we've learned from a year of building with LLMs

#35

Post by lmeyerov »

solidasparagus wrote: post #29
lmeyerov wrote: post #25

Earlier quoted context omitted.

We work in some pretty serious domains and try to stay away from fine tuning: - Most of our accuracy ROI is from agentic loops over top models, and dynamic RAG example injection goes far here that the relative lift of adding fine-tuning isn't worth the many costs - A lot of fine-tuning is for OSS models that do worse than agentic loops over the proprietary GPT4/Opus3 - For distribution, it's a lot easier to deploy fo…

See if the article said this, I would have agreed - fine-tuning is a tool and it should be used thoughtfully. Although I personally believe that in this funding climate it makes sense to make data collection and model training a core capability of any AI product. However that will only be available and wise for some founders.

Agreed, model training and data collection are great!

The subtle bit is just doesn't have to be for LLMs, as these are typically part of a system-of-models. E.g., we <3 RAG, and GNNs for improving your KG is fascinating. Likewise, dspy's explorations in optimizing prompts, vs LLMs, is very cool.

Re: What we've learned from a year of building with LLMs

#36

Post by fancy_pantser »

robbiemitchell wrote: post #32
BoorishBears wrote: post #31

Earlier quoted context omitted.

> Lack of JSON schema restriction is a significant barrier to entry on hooking LLMs up to a multi step process. How are you struggling with this, let alone as a significant barrier? JSON adherence with a well thought out schema hasn't been a worry between improved model performance and various grammar based constraint systems in a while. > Another is preventing LLMs from adding intro or conclusion text. Also trivial…

Asking even a top-notch LLM to output well formed JSON simply fails sometimes. And when you’re running LLMs at high volume in the background, you can’t use the best available until the last mile. You work around it with post-processing and retries. But it’s still a bit brittle given how much stuff happens downstream without supervision.

Constrained output with GBNF or JSON is much more efficient and less error-prone. I hope nobody outside of hobby projects is still using error/retry loops.

Re: What we've learned from a year of building with LLMs

#37

Post by solidasparagus »

lmeyerov wrote: post #35
solidasparagus wrote: post #29

Earlier quoted context omitted.

See if the article said this, I would have agreed - fine-tuning is a tool and it should be used thoughtfully. Although I personally believe that in this funding climate it makes sense to make data collection and model training a core capability of any AI product. However that will only be available and wise for some founders.

Agreed, model training and data collection are great! The subtle bit is just doesn't have to be for LLMs, as these are typically part of a system-of-models. E.g., we <3 RAG, and GNNs for improving your KG is fascinating. Likewise, dspy's explorations in optimizing prompts, vs LLMs, is very cool.

> we Oh man I am so torn between this being a fantastic idea and this being "building a better slide-rule in the age of the computer".

dspy is definitely a project I want to dig into more

Re: What we've learned from a year of building with LLMs

#38

Post by 127 »

Multicomp wrote: post #8

Anyone have a convenience solution for doing multi-step workflows? For example, I'm filling out the basics of an NPC character sheet on my game prep. I'm using a certain rule system, give the enemy certain tactics, certain stats, certain types of weapons, right now I have a 'god prompt' trying to walk the LLM through creating the basic character sheet, but the responses get squeezed down into what one or two prompt r…

Did you force it into a parser? You can define a simple language in llama.cpp for the LLM to obey.

Re: What we've learned from a year of building with LLMs

#40

Post by jncfhnb »

robbiemitchell wrote: post #32
BoorishBears wrote: post #31

Earlier quoted context omitted.

> Lack of JSON schema restriction is a significant barrier to entry on hooking LLMs up to a multi step process. How are you struggling with this, let alone as a significant barrier? JSON adherence with a well thought out schema hasn't been a worry between improved model performance and various grammar based constraint systems in a while. > Another is preventing LLMs from adding intro or conclusion text. Also trivial…

Asking even a top-notch LLM to output well formed JSON simply fails sometimes. And when you’re running LLMs at high volume in the background, you can’t use the best available until the last mile. You work around it with post-processing and retries. But it’s still a bit brittle given how much stuff happens downstream without supervision.

… why would you have the LLM spit out a json rather than define the json yourself and have the LLM supply values?
Post reply on HN