Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we've learned from a year of building with LLMs

eugeneyan.com

Post reply on HN
Print view View original

131–140 of 148 posts

Re: What we've learned from a year of building with LLMs

#131

Post by root_axis »

exhaze wrote: post #122
jakubmazanec wrote: post #119

Earlier quoted context omitted.

But it seems to me that's what they're doing: "We have LLMs, what to do with them?" But anyway, I'm seriously just looking for an example of app that is build with stuff described in the article. Me personally, I only used LLM for one "serious" application: I used GPT-3.5Turbo for transforming unstructured text into JSON; it was basically just ad-hoc Node.js script that called API (prompt was few examples of input-ou…

https://github.com/hrishioa/lumentis Since you seem to have not noticed my comment above, here's another example of a project that implements many of these techniques. Me and many others have used this to transcribe hour long videos into a well organized "docs site" that makes the content easy to read. Example: https://matadoc.vercel.app/ This was completely auto-generated in a few minutes. The author of the library…

I've tried this type of thing quite a bit (generating documentation based on code I've written), and it's generally pretty bad. Even just generating a README for a single source file project produces bloviated fluff that I have to edit rigorously. I'd say it does about 40% of the job, which is obviously a technical marvel, but in a practical sense it's more novelty than utility.

Re: What we've learned from a year of building with LLMs

#132

Post by altdataseller »

solidasparagus wrote: post #130
altdataseller wrote: post #104

Earlier quoted context omitted.

Why is the retrieval part overrated? There isnt even a single way to retrieve. It could be a simple keyword sesrch, a vector sesrch, a combo, or just simply retrieving a single doc and stuffing it in the context

People will disagree, but my problem with retrieval is that every technique that is popular uses one-hop thinking - you retrieve information that is directly related to the prompt using old-school techniques (even though the embeddings are new, text similarity is old). LLMs are most powerful, IMO, at horizontal thinking. Building a prompt using one-hop narrow AI techniques and then feeding it into a powerful generall…

People cant do that because of cost. If every single query involved taking everything even remotely related to the query, and passing it to OpenAI, it would get expensive very very fast.

Its not a technical issue, its a practicality issue imo.

Re: What we've learned from a year of building with LLMs

#133

Post by threeseed »

mattyyeung wrote: post #91
threeseed wrote: post #34

RAGs do not prevent hallucinations nor does it guarantee that the quality of your output is contingent solely on the quality of your input. Using LLMs for legal use cases for example has shown it to be poor for anything other than initial research as it is accurate at best 65%: https://dho.stanford.edu/wp-content/uploads/Legal_RAG_Halluc... So would strongly disagree that LLMs have become “good enough” for real-world…

You may be interested "Deterministic Quoting"[1]. This doesn't completely "solve" hallucinations, but I would argue that we do get "good enough" in several applications Disclosure: author on [1] [1] https://mattyyeung.github.io/deterministic-quoting

Have seen this approach before.

It's the yes we hallucinate but don't worry because we provide the sources for users to check.

Even though everyone knows that users will never check unless the hallucination is egregious.

It's such a disingenuous way of handling this.

Re: What we've learned from a year of building with LLMs

#134

Post by fnordpiglet »

phillipcarter wrote: post #123
mattmanser wrote: post #81

Earlier quoted context omitted.

You've linked to a query generator for a custom programming language and a 1 hour video about LLM tools. The cynic in me feels like the former could probably be done by chatgpt off the shelf. But those do not seem to be real world business cases. Can you expand a bit more why you think they are? We don't have hours to spend reading, and you say you've been allowed to talk about them. So can you summarise the business…

> The cynic in me feels like the former could probably be done by chatgpt off the shelf. Hello! I'm the owner of the feature in question who experimented with chatgpt last year in the course of building the feature (and working with Hamel to improve it via fine-tuning later). Even today, it could not work with ChatGPT. To generate valid queries, you need to know which subset of a user's dataset schema is relevant to…

Thanks for the reply. Huge fan of honeycomb and the feature. Spent many years in observability and built a some of the large in use log platforms. Tracing is the way of the future and hope to see you guys eat that market. I did some executive tech strategy stuff at some megacorp on observability and it’s really hard to unwedge metrics and logs but I’ve done my best when it was my focus. Good luck and thanks for all you’re doing over there.

Re: What we've learned from a year of building with LLMs

#135

Post by exhaze »

root_axis wrote: post #131
exhaze wrote: post #122

Earlier quoted context omitted.

https://github.com/hrishioa/lumentis Since you seem to have not noticed my comment above, here's another example of a project that implements many of these techniques. Me and many others have used this to transcribe hour long videos into a well organized "docs site" that makes the content easy to read. Example: https://matadoc.vercel.app/ This was completely auto-generated in a few minutes. The author of the library…

I've tried this type of thing quite a bit (generating documentation based on code I've written), and it's generally pretty bad. Even just generating a README for a single source file project produces bloviated fluff that I have to edit rigorously. I'd say it does about 40% of the job, which is obviously a technical marvel, but in a practical sense it's more novelty than utility.

Please just go and try the lumentis library I mentioned - that is what was used to generate this. It works. For the library docs I showed, I literally just wrote a zsh script to concat all the code together into one file, each one wrapped with XML open/close tags, and fed that in. Just because you weren't able to do it doesn't mean it's a novelty.

Re: What we've learned from a year of building with LLMs

#136

Post by exhaze »

jakubmazanec wrote: post #126
exhaze wrote: post #122

Earlier quoted context omitted.

https://github.com/hrishioa/lumentis Since you seem to have not noticed my comment above, here's another example of a project that implements many of these techniques. Me and many others have used this to transcribe hour long videos into a well organized "docs site" that makes the content easy to read. Example: https://matadoc.vercel.app/ This was completely auto-generated in a few minutes. The author of the library…

But that seems to belong to the category "text transformation" (e.g. translating, converting unstructed notes into structured data, etc.), which I acknowledge LLMs are good at; instead of category "I'll magically debug your SQL wish!".

I believe we were discussing the former not the latter? I agree that for lots of problem solving tasks it can be hit or miss - in my experience, all the models are quite bad at writing decent frontend code when it comes to the rendered page looking the way you want it to.

What you're describing is more about reasoning abilities - that's not really what the article was about or the problems the techniques are for. The techniques in article are more for stuff like Q&A, classification, summarization, etc.

Re: What we've learned from a year of building with LLMs

#137

Post by OutOfHere »

robrenaud wrote: post #102
OutOfHere wrote: post #21

Earlier quoted context omitted.

I don't consider a small 8B model to be worth fine-tuning. Fine-tuning is worthwhile when you have a larger model with capacity to add data, perhaps one that can even grow its layers with the data. In contrast, fine-tuning a small saturated model will easily cause it to forget older information. All things considered, in relative terms, as much as I think fine-tuning would be nice, it will remain significantly more e…

The reason to fine tune is to get a model that performs well on a specific task. It could lose 90 percent of it's knowledge and beat the unturned model at the narrow task at hand. That's the point, no?

It is not really possible to lose 90% of one's brain and do well on certain narrow tasks. If the tasks truly were so narrow, you would be better off training a small model just for them from scratch.

Re: What we've learned from a year of building with LLMs

#138

Post by int_19h »

joatmon-snoo wrote: post #69
fancy_pantser wrote: post #36

Earlier quoted context omitted.

Constrained output with GBNF or JSON is much more efficient and less error-prone. I hope nobody outside of hobby projects is still using error/retry loops.

Constraining output means you don’t get to use ChatGPT or Claude though, and now you have to run your own stuff. Maybe for some folks that’s OK, but really annoying for others.

I do wonder why, though. Constraining output based on logits is a fairly simple and easy-to-implement idea, so why is this not part of e.g. the OpenAI API yet? They don't even have to expose it at the lowest level, just use it to force valid JSON in the output on their end.

Re: What we've learned from a year of building with LLMs

#139

Post by int_19h »

harrisoned wrote: post #26
dbs wrote: post #17

Show me the use cases you have supported in production. Then I might read all the 30 pages praising the dozens (soon to be hundreds?) of “best practices” to build LLMs.

It certainly has use cases, just not as many as the hype lead people to believe. For me: -Regex expressions: ChatGPT is the best multi-million regex parser to date. -Grammar and semantic check: It's a very good revision tool, helped me a lot of times, specially when writing in non-native languages. -Artwork inspiration: Not only for visual inspiration, in the case of image generators, but descriptive as well. The ver…

GPT-4 has amazing translation capabilities, too. Actually usable for long conversations.

Re: What we've learned from a year of building with LLMs

#140

Post by lmeyerov »

qeternity wrote: post #116
lmeyerov wrote: post #35

Earlier quoted context omitted.

Agreed, model training and data collection are great! The subtle bit is just doesn't have to be for LLMs, as these are typically part of a system-of-models. E.g., we <3 RAG, and GNNs for improving your KG is fascinating. Likewise, dspy's explorations in optimizing prompts, vs LLMs, is very cool.

Have you actually used DSPy? I still can't figure out what it's useful for beyond optimizing basic few shot prompts.

We tried dspy and a couple others like it. They're neat and I'm happy those teams are experimenting with these frameworks. At the same time, they try to do "too much" by taking over the control flow of your code and running autotuning everywhere over it. We needed to write our own agent framework as even tools like langchain are too insecure and inefficient for being an enterprise platform, and frameworks like dspy are even more far out there.

A year+ later, the most interesting kernel of insight to us from dspy is autotuning a single prompt: it's an optimizeable model just like any other. As soon as you have an eval framework in place for your prompts, having something like dspy tune your prompts on a per-LLM basis would be very cool. I'm not sure where they are on that, it seems against the grain for their focus. We're only now reaching the point where we would see ROI on that kind of thing, it took a long time to get here.

We do run an agentic framework, so doing cross-prompt autotuning would be neat too -- especially for how the orchestrator (ex: CoT) composes with individual agents. We call this the "composition problem" and it's frustrating. However, again, dspy and friends do "too much", by trying to also be the agent framework & runtime, while we just want the autotuner.

Post reply on HN