Live data from Hacker News

DSPy: Framework for programming with foundation models

github.com

21–30 of 56 posts

Re: DSPy: Framework for programming with foundation models

#21
post #5
post #3

Earlier quoted context omitted.

"Specifically, the DSPy compiler will internally trace your program and then craft high-quality prompts for large LMs" I'm having trouble understanding the value provided here. A prompt is a string. Why abstract that string away from me like this? My instinct is that this will make it harder, not easier, for me to understand what's going on and make necessary changes.

"A neural network layer is just a matrix. Why abstract that matrix and learn it?" Well, because it's not your job to figure out how to hardcode delicate string or floats that work well for a given architecture & backend. We want developers to iterate quickly on system designs: How should we break down the task? Where do we call LMs? What should they do? --- If you can guess the right prompts right away for each LLM,…

One time (ironically, after I learned about model free methods) I got sucked into writing a heuristic for an A* algorithm. It turned into a bottomless pit of manually tuning various combinations of rules. I learned the value of machine learning the hard way.

If prompts can be learned, then eventually it will be better to learn them than to manually tune them. However, these ideas need not be mutually exclusive. When we reject the tyranny of the “or” and we can have a prompt prior we manually tune and then update it with a learning process, right?

P.S. whoever wrote the title, I think it’s pretty silly to write “The Framework…” for anything because this presumes you have the only member of some category, which is never true!

Re: DSPy: Framework for programming with foundation models

#22
Looks interesting and seems to not make some of the mistakes that other frameworks make (langchain, llamaindex, etc.) I was pretty apprehensive when I looked at your short hand signature API. I'm really not a fan of these custom mini languages, but it looks like it's fairly constrained at the moment and has an expanded form with a sane Python API. My only concern is that the short hand signatures spiral out of control with "features" and we're back to debugging new languages with poor tooling.

If a new language is the correct answer for working with LLMs than I would prefer a real language spec, compiler/interpreter, debugger, and language server before considering adoption. Which is a lot of work ofc and why I'm apprehensive of how it grows.

Re: DSPy: Framework for programming with foundation models

#25
post #9
post #5

Earlier quoted context omitted.

"A neural network layer is just a matrix. Why abstract that matrix and learn it?" Well, because it's not your job to figure out how to hardcode delicate string or floats that work well for a given architecture & backend. We want developers to iterate quickly on system designs: How should we break down the task? Where do we call LMs? What should they do? --- If you can guess the right prompts right away for each LLM,…

I agree with you on all of those points - but my conclusion is different: those are the reasons it's so important to me that the prompts are not abstracted away from me! I'm working with Llama 2 a bunch at the moment and much of the challenge is learning how to prompt it differently from how I prompt GPT-4. I'm not yet convinced that an abstraction will solve that problem for me.

this is one of the reasons why Langchain sucks. People seem to underestimate and overlook the importance of prompts.

Re: DSPy: Framework for programming with foundation models

#29
post #28

Earlier quoted context omitted.

Please do not copy and paste content from the link, it is unnecessary

These are answers to specific questions below.

This is faux pas on HN, as you see the other comments you where you have done this in have been flagged and removed.

Link to the answer as a reply to the specific comment instead

Re: DSPy: Framework for programming with foundation models

#30
This is really cool! It took me a bit to understand what this thing is for so allow me to summarize:

For simple LLM tasks, don't bother using this tool. It won't do much for you.

If you have a more complicated task (eg. knowledge database lookups, chain of thought reasoning, multi-hop lookups...) then DSPy offers 2 things: a clean class-based representation of your workflow, and a way to *solve* for the best prompt structure to solve your problem.

To me, the last part is the most interesting because it promises to eliminate tedious prompt engineering. All you need is a set of examples to "train" your prompts on.

Post reply on HN