Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we've learned from a year of building with LLMs

eugeneyan.com

Post reply on HN
Print view View original

141–148 of 148 posts

Re: What we've learned from a year of building with LLMs

#141

Post by solidasparagus »

altdataseller wrote: post #132
solidasparagus wrote: post #130

Earlier quoted context omitted.

People will disagree, but my problem with retrieval is that every technique that is popular uses one-hop thinking - you retrieve information that is directly related to the prompt using old-school techniques (even though the embeddings are new, text similarity is old). LLMs are most powerful, IMO, at horizontal thinking. Building a prompt using one-hop narrow AI techniques and then feeding it into a powerful generall…

People cant do that because of cost. If every single query involved taking everything even remotely related to the query, and passing it to OpenAI, it would get expensive very very fast. Its not a technical issue, its a practicality issue imo.

That's very true. Although it is feasible if you are well-resourced and make the investment to own the toolchain end-to-end. Serving costs can be quite low (relatively speaking) if you control everything. And you have to pick the correct problem where the cost is worthwhile.

Re: What we've learned from a year of building with LLMs

#142

Post by xnx »

mloncode wrote: post #60

This is Hamel, one of the authors of the article. We published the article with OReilly here: Part 1: https://www.oreilly.com/radar/what-we-learned-from-a-year-of... Part 2: https://www.oreilly.com/radar/what-we-learned-from-a-year-of... We were working on this webpage to collect the entire three part article in one place (the third part isn't published yet). We didn't expect anyone to notice the site! Either way, pa…

The link to part II from part I points back to part I.

Re: What we've learned from a year of building with LLMs

#143

Post by phillipcarter »

fnordpiglet wrote: post #134
phillipcarter wrote: post #123

Earlier quoted context omitted.

> The cynic in me feels like the former could probably be done by chatgpt off the shelf. Hello! I'm the owner of the feature in question who experimented with chatgpt last year in the course of building the feature (and working with Hamel to improve it via fine-tuning later). Even today, it could not work with ChatGPT. To generate valid queries, you need to know which subset of a user's dataset schema is relevant to…

Thanks for the reply. Huge fan of honeycomb and the feature. Spent many years in observability and built a some of the large in use log platforms. Tracing is the way of the future and hope to see you guys eat that market. I did some executive tech strategy stuff at some megacorp on observability and it’s really hard to unwedge metrics and logs but I’ve done my best when it was my focus. Good luck and thanks for all y…

Glad you like HNY and the feature! Here’s hoping we can help move a lot more of the world over to tracing :)

Re: What we've learned from a year of building with LLMs

#144

Post by jncfhnb »

janpieterz wrote: post #127
jncfhnb wrote: post #93

Earlier quoted context omitted.

Don’t do it all in one prompt.

Right, but now I’m basically running a huge performance hit, need to parallelize my queries etc. I was parsing a document recently, 10-ish questions for 1 document, would make things expensive. Might be what’s needed but not ideal.

LLM performance is a function of the number of tokens, not queries

Re: What we've learned from a year of building with LLMs

#145

Post by qeternity »

lmeyerov wrote: post #140
qeternity wrote: post #116

Earlier quoted context omitted.

Have you actually used DSPy? I still can't figure out what it's useful for beyond optimizing basic few shot prompts.

We tried dspy and a couple others like it. They're neat and I'm happy those teams are experimenting with these frameworks. At the same time, they try to do "too much" by taking over the control flow of your code and running autotuning everywhere over it. We needed to write our own agent framework as even tools like langchain are too insecure and inefficient for being an enterprise platform, and frameworks like dspy a…

It’s funny: I found the optimizer (which you could quite easily rip out from DSPy) to be the most underwhelming part of the equation.

Re: What we've learned from a year of building with LLMs

#146

Post by lmeyerov »

qeternity wrote: post #145
lmeyerov wrote: post #140

Earlier quoted context omitted.

We tried dspy and a couple others like it. They're neat and I'm happy those teams are experimenting with these frameworks. At the same time, they try to do "too much" by taking over the control flow of your code and running autotuning everywhere over it. We needed to write our own agent framework as even tools like langchain are too insecure and inefficient for being an enterprise platform, and frameworks like dspy a…

It’s funny: I found the optimizer (which you could quite easily rip out from DSPy) to be the most underwhelming part of the equation.

The rest is neat but scary for most production scenarios, while a prompt autotuner can give significant lift + resilience in a predictable & maintainable way to most typical LLM apps

Again... I'm truly happy and supportive that academics are exploring a wild side of the design space. Just, as we are in the 'we ship code people rely on' side of the universe, it's hard to find scenarios where its potential benefits outweigh its costs.

Re: What we've learned from a year of building with LLMs

#147

Post by evrydayhustling »

lastdong wrote: post #75
robbiemitchell wrote: post #30

Earlier quoted context omitted.

Processing high volumes of unstructured data (text)… we’re using a STAG architecture. - Generate targeted LLM micro summaries of every record (ticket, call, etc.) continually - Use layers of regex, semantic embeddings, and scoring enrichments to identify report rows (pivots on aggregates) worth attention, running on a schedule - Proactively explain each report row by identifying what’s unusual about it and LLM summar…

“Use layers of regex, semantic embeddings, and scoring enrichments to identify report rows (pivots on aggregates) worth attention, running on a schedule” This is really interesting, is there any architecture documentation/articles that you can recommend?

I'm late to this party, but here's a post I wrote about it. This is more motivation but we are working on technical posts/papers for release. Happy to field emails in the meantime if this is timely for you.

https://www.linkedin.com/pulse/ai-2024-more-answers-fewer-qu...

Re: What we've learned from a year of building with LLMs

#148

Post by lastdong »

evrydayhustling wrote: post #147
lastdong wrote: post #75

Earlier quoted context omitted.

“Use layers of regex, semantic embeddings, and scoring enrichments to identify report rows (pivots on aggregates) worth attention, running on a schedule” This is really interesting, is there any architecture documentation/articles that you can recommend?

I'm late to this party, but here's a post I wrote about it. This is more motivation but we are working on technical posts/papers for release. Happy to field emails in the meantime if this is timely for you. https://www.linkedin.com/pulse/ai-2024-more-answers-fewer-qu...

Awesome! Thank you
Post reply on HN