Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we've learned from a year of building with LLMs

eugeneyan.com

Post reply on HN
Print view View original

81–90 of 148 posts

Re: What we've learned from a year of building with LLMs

#81

Post by mattmanser »

hamelsmu wrote: post #64
TeMPOraL wrote: post #62

Earlier quoted context omitted.

> We use LLMs in dozens of different production applications for critical business flows. They allow for a lot of dynamism in our flows that aren’t amenable to direct quantitative reasoning or structured workflows. Double digit percents of our growth in the last year are entirely due to them. The biggest challenge is tool chain, limits on inference capacity, and developer understanding of the abilities, limits, and t…

Hi, Hamel here. I'm one of the co-authors. I'm an independent consultant and not all clients allow me to talk about their work. However, I have two that do, which I've discussed in the article. These are two production use cases that I have supported (which again, are explicitly mentioned in the article): 1. https://www.honeycomb.io/blog/introducing-query-assistant 2. https://www.youtube.com/watch?v=B_DMMlDuJB0 Other…

You've linked to a query generator for a custom programming language and a 1 hour video about LLM tools. The cynic in me feels like the former could probably be done by chatgpt off the shelf.

But those do not seem to be real world business cases.

Can you expand a bit more why you think they are? We don't have hours to spend reading, and you say you've been allowed to talk about them.

So can you summarise the business benefits for us, which is what people are asking for, instead of linking to huge articles?

Re: What we've learned from a year of building with LLMs

#82

Post by punkspider »

Multicomp wrote: post #8

Anyone have a convenience solution for doing multi-step workflows? For example, I'm filling out the basics of an NPC character sheet on my game prep. I'm using a certain rule system, give the enemy certain tactics, certain stats, certain types of weapons, right now I have a 'god prompt' trying to walk the LLM through creating the basic character sheet, but the responses get squeezed down into what one or two prompt r…

Perhaps this would be of use? https://github.com/langgenius/dify/ I use it for quick workflows and it's pretty intuitive.

Re: What we've learned from a year of building with LLMs

#83

Post by 80hd »

mattmanser wrote: post #81
hamelsmu wrote: post #64

Earlier quoted context omitted.

Hi, Hamel here. I'm one of the co-authors. I'm an independent consultant and not all clients allow me to talk about their work. However, I have two that do, which I've discussed in the article. These are two production use cases that I have supported (which again, are explicitly mentioned in the article): 1. https://www.honeycomb.io/blog/introducing-query-assistant 2. https://www.youtube.com/watch?v=B_DMMlDuJB0 Other…

You've linked to a query generator for a custom programming language and a 1 hour video about LLM tools. The cynic in me feels like the former could probably be done by chatgpt off the shelf. But those do not seem to be real world business cases. Can you expand a bit more why you think they are? We don't have hours to spend reading, and you say you've been allowed to talk about them. So can you summarise the business…

Sounds like something you could do with an LLM

Re: What we've learned from a year of building with LLMs

#84

Post by idf00 »

altdataseller wrote: post #55
solidasparagus wrote: post #20

Earlier quoted context omitted.

How much of that is just the flood of traditional engineers into the space and the fact that collecting data and then fine-tuning models is orders of magnitude more complex than just throwing in RAG? I suspect a huge amount of RAG's popularity is just that any engineer can do a version of it + ChatGPT API calls in a day. As for lora - in the context of my comment, that's just splitting hairs IMO. It falls in the cate…

I disagree. No amount of fine tuning will ever give the LLM the relevant context with which to answer my question. Maybe if your context is a static Wikipedia or something that will never change, you can fine tune it. But if your data and docs keep changing, how is fine tuning going to be better than RAG?

Luckily it's not one or the other. You can fine tune and use RAG.

Sometimes RAG is enough. Sometimes fine tuning on top of RAG is better. It depends on the use case. I can't think of any examples where you would want to fine tune and not use rag as well.

Sometimes you fine tune a small model so it performs close to a larger varient on that specific narrow task and you improve inference performance by using a smaller model.

Re: What we've learned from a year of building with LLMs

#85

Post by anon373839 »

mloncode wrote: post #63
dbs wrote: post #17

Show me the use cases you have supported in production. Then I might read all the 30 pages praising the dozens (soon to be hundreds?) of “best practices” to build LLMs.

Hi, Hamel here. I'm one of the co-authors. I'm an independent consultant and not all clients allow me to talk about their work. However, I have two that do, which I've discussed in the article. These are two production use cases that I have supported (which again, are explicitly mentioned in the article): 1. https://www.honeycomb.io/blog/introducing-query-assistant 2. https://www.youtube.com/watch?v=B_DMMlDuJB0 Other…

I know it’s a snarky comment you responded to, but I’m glad you did. Those are great resources, as is your excellent article. Thanks for posting!

Re: What we've learned from a year of building with LLMs

#86

Post by mercurialsolo »

As we go about moving LLM enabled products into production we definitely see a bunch of what is being spoken about resonate. We also see the below as areas which need to be expanded upon for developers building in the space to take products to production :

I would love to see this article also expand to touch upon things like : - data management - (tooling, frameworks, open vs closed data management, labelling & annotations) - inference as a pipeline - frameworks for breaking down model inference into smaller tasks & combining outputs (do DAG's have a role to play here?) - prompts - areas like caching, management, versioning, evaluations - model observability - tokens, costs, latency, drift? - evals for multimodality - how do we tackle evals here which in turn can go into loops e.g. quality of audio, speech or visual outputs

Re: What we've learned from a year of building with LLMs

#87

Post by lmeyerov »

tarasglek wrote: post #76
lmeyerov wrote: post #35

Earlier quoted context omitted.

Agreed, model training and data collection are great! The subtle bit is just doesn't have to be for LLMs, as these are typically part of a system-of-models. E.g., we <3 RAG, and GNNs for improving your KG is fascinating. Likewise, dspy's explorations in optimizing prompts, vs LLMs, is very cool.

Can you give a concrete example of GNNs helping?

Entity resolution - RAG often mixes vector & symbolic queries, and ER improves reverse indexing, which is a starting point for a lot of the symbolic ones

Identifying misinfo - Ranking & summarization based on internet data should be a lot more careful, and sometimes the controversy is the interesting part

For both, GNNs are generally SOTA

Re: What we've learned from a year of building with LLMs

#88

Post by idf00 »

mattmanser wrote: post #81
hamelsmu wrote: post #64

Earlier quoted context omitted.

Hi, Hamel here. I'm one of the co-authors. I'm an independent consultant and not all clients allow me to talk about their work. However, I have two that do, which I've discussed in the article. These are two production use cases that I have supported (which again, are explicitly mentioned in the article): 1. https://www.honeycomb.io/blog/introducing-query-assistant 2. https://www.youtube.com/watch?v=B_DMMlDuJB0 Other…

You've linked to a query generator for a custom programming language and a 1 hour video about LLM tools. The cynic in me feels like the former could probably be done by chatgpt off the shelf. But those do not seem to be real world business cases. Can you expand a bit more why you think they are? We don't have hours to spend reading, and you say you've been allowed to talk about them. So can you summarise the business…

They think they are real business use cases, because real businesses use them to solve their use cases. They know that chatgpt can't solve this off the shelf, because they tried that first and were forced to do more in order to solve their problem.

There's a summary for ya! More details in the stuff that they linked if you want to learn. Technical skills do require a significant time investment to learn, and LLM usage is no different.

Re: What we've learned from a year of building with LLMs

#89

Post by lmeyerov »

solidasparagus wrote: post #37
lmeyerov wrote: post #35

Earlier quoted context omitted.

Agreed, model training and data collection are great! The subtle bit is just doesn't have to be for LLMs, as these are typically part of a system-of-models. E.g., we <3 RAG, and GNNs for improving your KG is fascinating. Likewise, dspy's explorations in optimizing prompts, vs LLMs, is very cool.

> we Oh man I am so torn between this being a fantastic idea and this being "building a better slide-rule in the age of the computer". dspy is definitely a project I want to dig into more

Yeah I would recommend sticking to RAG on naively chunked data for weekend projects by 1 person. Likewise, a consumer tool like perplexity's search engine where you minimize spend per user task or go bankrupt, same thing, do the cheap thing and move on, good enough

Once RAG projects become important and good answers matter - we work with governments, manufacturers, banks, cyber teams, etc - working through data quality, data representation, & retrieval quality helps

Note that we didn't start here: We began with naive RAG, then relevancy filtering, then agentic & neurosymbolic querying, then dynamic example prompt injection, and now are getting into cleaning up the database/kg itself

For folks doing investigative/analytics projects in this space, happy to chat about what we are doing w Louie.AI. These are more implementation details we don't normally write about.

Re: What we've learned from a year of building with LLMs

#90

Post by jncfhnb »

esafak wrote: post #74
jncfhnb wrote: post #40

Earlier quoted context omitted.

… why would you have the LLM spit out a json rather than define the json yourself and have the LLM supply values?

If the LLM doesn't output data that conforms to a schema, you can't reliably parse it, so you're back to square one.

It’s significantly easier to output an integer than a JSON with a key value structure where the value is an integer and everything else is exactly as desired
Post reply on HN