Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we've learned from a year of building with LLMs

eugeneyan.com

Post reply on HN
Print view View original

71–80 of 148 posts

Re: What we've learned from a year of building with LLMs

#71

Post by solidasparagus »

bbischof wrote: post #67
solidasparagus wrote: post #2

No offense, but I'd love to see what they've successfully built using LLMs before taking their advice too seriously. The idea that fine-tuning isn't even a consideration (perhaps even something they think is absolutely incorrect if the section titles of the unfinished section is anything to go by) is very strange to me and suggests a pretty narrow perspective IMO

Hello, it’s Bryan, an author on this piece. I’d you’re interested in using one of the LLM-applications I have in prod, check out https://hex.tech/product/magic-ai/ It has a free limit every month to give it a try and see how you like it. If you have feedback after using it, we’re always very interested to hear from users. As far as fine-tuning in particular, our consensus is that there are easier options first. I per…

I took at look at Magic earlier today and it didn't work at all for me, sorry to say. After the example prompt, I tried to learn about a table and it generated bad SQL (correct query to pull a row, but with limit 0). I asked it to show me the DDL and it generated invalid SQL. Then I tried to ask it to do some population statistics on the customer table and ended up confused about why there appears to be two windows in the cell, with the previously generated SQL on the left and the newly generated SQL on the right. The new SQL wouldn't run when I hit run cell, the error showed the originally generated SQL. I gave up and bounced.

I went back while writing this comment and realized it might be showing me a diff (better use of color would have helped, I have been trained by github). But I was at a loss for what to do with that. I just now figured out the Keep button exists and it accepted the diff and now it sort of makes sense, but the SQL still doesn't return any results.

My honest feedback is that there is way too much stuff I don't understand on the screen and it makes me confused and a little stressed. Ease me into it please, I'm dumb. There seems to be cells that are linked together and cells that aren't(? separated by purplish background) and I don't understand it. I am a jupyter user and I feel like this should be intuitive to me, but it isn't. I am not a designer, but I suspect the structural markings like cell boundaries are too faint compared to the content of the cells and/or the exterior of a cell having the same color as the interior is making it hard for me. I feel lost in a sea of white.

But the core issue is that, excluding the prompt I copy-pasted word for word which worked like a charm, I am 0 out of 4 on actually leveraging AI to solve the problems I asked of Magic. I like the concept of natural language BI (I worked on in the early days when Alexa came out) so I probably gave it more chances than I would have for a different product.

For me, it doesn't fit my criteria for good problems to solve with AI in 2024 - the conversational interface and binary right/wrong nature of querying/presenting data accurately make the cost of failure too high, which is a death sentence for AI products IMO (compare to proactive, non-blocking products like copilot or shades-of-wrong problems like image generation or conversations with imaginary characters). But text-to-SQL and data presentation make sense as AI capabilities in 2024 so I can see why that could be a good product to pursue. If it worked, I would definitely use it.

Re: What we've learned from a year of building with LLMs

#72

Post by [deleted] »

jph00 wrote: post #59
solidasparagus wrote: post #2

No offense, but I'd love to see what they've successfully built using LLMs before taking their advice too seriously. The idea that fine-tuning isn't even a consideration (perhaps even something they think is absolutely incorrect if the section titles of the unfinished section is anything to go by) is very strange to me and suggests a pretty narrow perspective IMO

> The idea that fine-tuning isn't even a consideration (perhaps even something they think is absolutely incorrect if the section titles of the unfinished section is anything to go by) is very strange to me and suggests a pretty narrow perspective IMO The article has a section called "When to finetune", along with links to separate pages describing how to do so. They absolutely don't say that "fine-tuning isn't even a…

[deleted]

Re: What we've learned from a year of building with LLMs

#73

Post by solidasparagus »

jph00 wrote: post #59
solidasparagus wrote: post #2

No offense, but I'd love to see what they've successfully built using LLMs before taking their advice too seriously. The idea that fine-tuning isn't even a consideration (perhaps even something they think is absolutely incorrect if the section titles of the unfinished section is anything to go by) is very strange to me and suggests a pretty narrow perspective IMO

> The idea that fine-tuning isn't even a consideration (perhaps even something they think is absolutely incorrect if the section titles of the unfinished section is anything to go by) is very strange to me and suggests a pretty narrow perspective IMO The article has a section called "When to finetune", along with links to separate pages describing how to do so. They absolutely don't say that "fine-tuning isn't even a…

Huh. Well that's embarrassing. I guess I missed it when I lost interest in the caching section and jumped straight to Evaluation and Monitoring.

Re: What we've learned from a year of building with LLMs

#74

Post by esafak »

jncfhnb wrote: post #40
robbiemitchell wrote: post #32

Earlier quoted context omitted.

Asking even a top-notch LLM to output well formed JSON simply fails sometimes. And when you’re running LLMs at high volume in the background, you can’t use the best available until the last mile. You work around it with post-processing and retries. But it’s still a bit brittle given how much stuff happens downstream without supervision.

… why would you have the LLM spit out a json rather than define the json yourself and have the LLM supply values?

If the LLM doesn't output data that conforms to a schema, you can't reliably parse it, so you're back to square one.

Re: What we've learned from a year of building with LLMs

#75

Post by lastdong »

robbiemitchell wrote: post #30
dbs wrote: post #17

Show me the use cases you have supported in production. Then I might read all the 30 pages praising the dozens (soon to be hundreds?) of “best practices” to build LLMs.

Processing high volumes of unstructured data (text)… we’re using a STAG architecture. - Generate targeted LLM micro summaries of every record (ticket, call, etc.) continually - Use layers of regex, semantic embeddings, and scoring enrichments to identify report rows (pivots on aggregates) worth attention, running on a schedule - Proactively explain each report row by identifying what’s unusual about it and LLM summar…

“Use layers of regex, semantic embeddings, and scoring enrichments to identify report rows (pivots on aggregates) worth attention, running on a schedule”

This is really interesting, is there any architecture documentation/articles that you can recommend?

Re: What we've learned from a year of building with LLMs

#76

Post by tarasglek »

lmeyerov wrote: post #35
solidasparagus wrote: post #29

Earlier quoted context omitted.

See if the article said this, I would have agreed - fine-tuning is a tool and it should be used thoughtfully. Although I personally believe that in this funding climate it makes sense to make data collection and model training a core capability of any AI product. However that will only be available and wise for some founders.

Agreed, model training and data collection are great! The subtle bit is just doesn't have to be for LLMs, as these are typically part of a system-of-models. E.g., we <3 RAG, and GNNs for improving your KG is fascinating. Likewise, dspy's explorations in optimizing prompts, vs LLMs, is very cool.

Can you give a concrete example of GNNs helping?

Re: What we've learned from a year of building with LLMs

#77

Post by jeffreygoesto »

fnordpiglet wrote: post #51
threeseed wrote: post #49

Earlier quoted context omitted.

No one is saying that all of AI is hype. It clearly isn't. But the facts are that today LLMs are not suitable for use cases that need accurate results. And there is no evidence or research that suggests this is changing anytime soon. Maybe for ever. There are very strong parallels to crypto in that (a) people are starting with the technology and trying to find problems and (b) there is a cult like atmosphere where no…

Yeah I think a key is LLMs in business are not generally useful alone. They require classical computing techniques to really be powerful. Accurate computation is a generally well established field and you don’t need an LLM to do optimization or math or even deductive logical reasoning. That’s a waste of their power which is typically abstract semantic abductive “reasoning” and natural language processing. Overlaying…

That text applies to basically every new technology. Point is that you can't predict it's usefulness in 20 years from that.

To me it still looks like a hammer made completely from rubber. You can practice to get some good hits, but it is pretty hard to get something reliable. And a beginner will basically just bounce it around. But it is sold as rescue for beginners.

Re: What we've learned from a year of building with LLMs

#78

Post by tarasglek »

adamsbriscoe wrote: post #43
robbiemitchell wrote: post #30

Earlier quoted context omitted.

Processing high volumes of unstructured data (text)… we’re using a STAG architecture. - Generate targeted LLM micro summaries of every record (ticket, call, etc.) continually - Use layers of regex, semantic embeddings, and scoring enrichments to identify report rows (pivots on aggregates) worth attention, running on a schedule - Proactively explain each report row by identifying what’s unusual about it and LLM summar…

> Lack of JSON schema restriction is a significant barrier to entry on hooking LLMs up to a multi step process. (Plug) I shipped a dedicated OpenAI-compatible API for this, jsonmode.com a couple weeks ago and just integrated Groq (they were nice enough to bump up the rate limits) so it's crazy fast. It's a WIP but so far very comparable to JSON output from frontier models, with some bonus features (web crawling etc).

The metallica-esque lightning logo is cool
Post reply on HN