Live data from Hacker News

DSPy: Framework for programming with foundation models

github.com

11–20 of 56 posts

Re: DSPy: Framework for programming with foundation models

#11
post #5
post #3

Earlier quoted context omitted.

"Specifically, the DSPy compiler will internally trace your program and then craft high-quality prompts for large LMs" I'm having trouble understanding the value provided here. A prompt is a string. Why abstract that string away from me like this? My instinct is that this will make it harder, not easier, for me to understand what's going on and make necessary changes.

"A neural network layer is just a matrix. Why abstract that matrix and learn it?" Well, because it's not your job to figure out how to hardcode delicate string or floats that work well for a given architecture & backend. We want developers to iterate quickly on system designs: How should we break down the task? Where do we call LMs? What should they do? --- If you can guess the right prompts right away for each LLM,…

@simonw it sounds like we'd agree that:

1] when prototyping, it's useful to not have to tweak each prompt by hand as long as you can inspect them easily

2] when the system design is "final", it's important to be able to tweak any prompts or finetunes with full flexibility

But we may or may not agree on:

3] automatic optimization can basically make #2 above only very rarely needed

---

Anyway, the entire DSPy project has zero hard-coded prompts for tasks. It's all bootstrapped and validated for your logic. In case you're worried that we're doing some opinionated prompting on your behalf.

Re: DSPy: Framework for programming with foundation models

#12
post #6

[flagged]

I think I need to see some really convincing and detailed examples to understand what's going on here.

Does DSPy have good debugging hooks for things like "print out every prompt that is generated to a log"?

When I'm evaluating a new piece of technology the number one question I want to answer is "what are the projects that adopting this technology enables me to build that I couldn't have built without it?" - either by giving me a new capability or by reducing the amount of work I have to put in to the point that a project now fits in my available time where it didn't before.

Re: DSPy: Framework for programming with foundation models

#13

Just yesterday I was thinking to myself "I wonder how long it will be until we start generating prompts using code and then we're just writing code again"

I welcome programming English++ with open arms so long as I can scold it when it makes mistakes and it doesn’t require move semantics.

I’m joking of course but I do think LLM’s will become part of the programming language lexer of some kind. If not already being looked into.

Re: DSPy: Framework for programming with foundation models

#15
post #12
post #6

[flagged]

I think I need to see some really convincing and detailed examples to understand what's going on here. Does DSPy have good debugging hooks for things like "print out every prompt that is generated to a log"? When I'm evaluating a new piece of technology the number one question I want to answer is "what are the projects that adopting this technology enables me to build that I couldn't have built without it?" - either…

"print out every prompt that is generated to a log" --- yes of course

This Colab is full of prompts and examples of improving the quality of gpt-3.5-turbo: https://t.co/Oa1RDp3XbZ

Paper incoming, but basically we've seen > 50% quality gains by just compiling in various settings.

This Twitter/X thread discusses doing a simple program for Llama2, with massive quality gains too: https://twitter.com/lateinteraction/status/16947484013744909...

Re: DSPy: Framework for programming with foundation models

#16
post #11
post #5

Earlier quoted context omitted.

"A neural network layer is just a matrix. Why abstract that matrix and learn it?" Well, because it's not your job to figure out how to hardcode delicate string or floats that work well for a given architecture & backend. We want developers to iterate quickly on system designs: How should we break down the task? Where do we call LMs? What should they do? --- If you can guess the right prompts right away for each LLM,…

@simonw it sounds like we'd agree that: 1] when prototyping, it's useful to not have to tweak each prompt by hand as long as you can inspect them easily 2] when the system design is "final", it's important to be able to tweak any prompts or finetunes with full flexibility But we may or may not agree on: 3] automatic optimization can basically make #2 above only very rarely needed --- Anyway, the entire DSPy project h…

It sounds fascinating! Is there anything one could read to figure out more about how this is being done (From reading the docs by the "Teleprompter"s right)?

Re: DSPy: Framework for programming with foundation models

#17

Just yesterday I was thinking to myself "I wonder how long it will be until we start generating prompts using code and then we're just writing code again"

There probably is a cool mix of both that is better than either one separately. I'm thinking something like

  func = llm('function that sorts two input argument lists')
where llm calls openai or a local llm (cached for later use). This way you don't lose the benefits of a coding interface (e.g. make all code with openai and the maintainability mess that can come with that). And you get readability through the prompt etc etc. (I mean this project is sort of in that direction already.) It's basically like writing code with a framework that is 'filled out' automatically by a coworker.

Worth being creative with ideas like this at least.

Re: DSPy: Framework for programming with foundation models

#20
post #18

How does the compilation logic work? It’s described as optimizing the prompts just like you optimize the weights of a neural net, but what does that look like in practice?

See the discussion of teleprompters here:

https://colab.research.google.com/github/stanfordnlp/dspy/bl...

Post reply on HN