Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DSPy – Programming–not prompting–LMs

dspy.ai

Post reply on HN
Print view View original

21–30 of 46 posts

Re: DSPy – Programming–not prompting–LMs

#21

Post by sdesol »

dr_kiszonka wrote: post #5
behnamoh wrote: post #4

Every time I've seen a dspy article, I end up thinking: ok, but what does it do exactly? I've been using guidance, outlines, GBF grammars, etc. What advantage does dspy have over those alternatives? I've learnt that the best package to use LLMs is just Python. These "LLM packages" just make it harder to do customizations as they all make opinionated assumptions and decisions.

Question from a casual AI user, if you have a minute. It seems to me that I could get much more productive by making my own personal AI "system". For example, write a simple pipeline where Claude would scrutinize OpenAI's answers and vice versa. Are there any beginner-friendly Python packages that you would recommend to facilitate fast experimentation with such ideas?

> For example, write a simple pipeline where Claude would scrutinize OpenAI's answers and vice versa.

I'm working on a naive approach to identify errors in LLM responses which I talk about at https://news.ycombinator.com/item?id=42313401#42313990, which can be used to scrutinize responses. It's written in Javascript though, but you will be able to create a new chat by calling a http endpoint.

I'm hoping to have the system in place in a couple of weeks.

Re: DSPy – Programming–not prompting–LMs

#22

Post by choppaface »

scosman wrote: post #9

Can someone explain what DSPy does that fine tuning doesn’t? Structured IO, optimized to better results. Sure. But why just just go straight to weights, instead of trying to optimize the few-shot space?

The main idea behind DSPy is that you can’t modify the weights, but you can perhaps modify the prompts. DSPy’s original primary customer was multi-llm-agent systems where you have a chain / graph of LLM calls (perhaps mostly or all to OpenAI GPT) and you have some metric (perhaps vague) that you want to increase. While the idea may seem a bit weird, there have been various success stories, such as a UoT team winning medical-notes-oriented competition using DSPy https://arxiv.org/html/2404.14544v1

Re: DSPy – Programming–not prompting–LMs

#23

Post by jackmpcollins »

dr_kiszonka wrote: post #5
behnamoh wrote: post #4

Every time I've seen a dspy article, I end up thinking: ok, but what does it do exactly? I've been using guidance, outlines, GBF grammars, etc. What advantage does dspy have over those alternatives? I've learnt that the best package to use LLMs is just Python. These "LLM packages" just make it harder to do customizations as they all make opinionated assumptions and decisions.

Question from a casual AI user, if you have a minute. It seems to me that I could get much more productive by making my own personal AI "system". For example, write a simple pipeline where Claude would scrutinize OpenAI's answers and vice versa. Are there any beginner-friendly Python packages that you would recommend to facilitate fast experimentation with such ideas?

I'm building magentic for use cases like this!

https://github.com/jackmpcollins/magentic

It's based on pydantic and aims to make writing LLM queries as easy/compact as possible by using type annotations, including for structured outputs and streaming. If you use it please reach out!

Re: DSPy – Programming–not prompting–LMs

#25

Post by thomasahle »

Here is a simple example of dspy:

    classify = dspy.Predict(f"text -> label:Literal{CLASSES}")

    optimized = dspy.BootstrapFewShot(metric=(lambda x, y: x.label == y.label))
                    .compile(classify,  trainset=load_dataset('Banking77'))

    label = optimized(text="What does a pending cash withdrawal mean?").label
What this does is optimize a prompt given a dataset (here Banking77).

The optimizer, BootstrapFewShot, simply selects a bunch of random subsets from the training set, and measures which gives the best performance on the rest of the dataset when used as few-shot examples.

There are also more fancy optimizers, including ones that first optimize the prompt, and then use the improved model as a teacher to optimize the weights. This has the advantage that you don't need to pay for a super long prompt on every inference call.

dspy has more cool features, such as the ability to train a large composite LLM program "end to end", similar to backprop.

The main advantage, imo, is just not having "stale" prompts everywhere in your code base. You might have written some neat few-shot examples for the middle layers of your pipeline, but then you change something at the start, and you have to manually rewrite all the examples for every other module. With dspy you just keep your training datasets around, and the rest is automated.

(Note, the example above is taken from the new website: https://dspy.ai/#__tabbed_3_3 and simplified a bit)

Re: DSPy – Programming–not prompting–LMs

#26

Post by d4rkp4ttern »

dr_kiszonka wrote: post #5
behnamoh wrote: post #4

Every time I've seen a dspy article, I end up thinking: ok, but what does it do exactly? I've been using guidance, outlines, GBF grammars, etc. What advantage does dspy have over those alternatives? I've learnt that the best package to use LLMs is just Python. These "LLM packages" just make it harder to do customizations as they all make opinionated assumptions and decisions.

Question from a casual AI user, if you have a minute. It seems to me that I could get much more productive by making my own personal AI "system". For example, write a simple pipeline where Claude would scrutinize OpenAI's answers and vice versa. Are there any beginner-friendly Python packages that you would recommend to facilitate fast experimentation with such ideas?

You can have a look at Langroid -- it's an agent-oriented LLM programming framework from CMU/UW-Madison researchers. We started building it in Apr 2023 out of frustration with the bloat of then-existing libs.

In langroid you set up a ChatAgent class which encapsulates an LLM-interface plus any state you'd like. There's a Task class that wraps an Agent and allows inter-agent communication and tool-handling. We have devs who've found our framework easy to understand and extend for their purposes, and some companies are using it in production (some have endorsed us publicly). A quick tour gives a flavor of Langroid: https://langroid.github.io/langroid/tutorials/langroid-tour/

Feel free to drop into our discord for help.

Re: DSPy – Programming–not prompting–LMs

#28

Post by thatsadude »

byefruit wrote: post #2

I've seen a couple of talks on DSPy and tried to use it for one of my projects but the structure always feels somewhat strained. It seems to be suited for tasks that are primarily show, don't tell but what do you do when you have significant prior instruction you want to tell? e.g Tests I want applied to anything retrieved from the database. What I'd like is to optimise the prompt around those (or maybe even the test…

You can optimize prompt with MIPROv2 without examples (set the max number of examples to 0)

Re: DSPy – Programming–not prompting–LMs

#29

Post by dcreater »

aaronvg wrote: post #15

I found it interesting how DSPy created the Signatures concept: https://dspy.ai/learn/programming/signatures/ We took this kind of concept all the way to making a DSL called BAML, where prompts look like literal functions, with input and output types. Playground link here https://www.promptfiddle.com/ https://github.com/BoundaryML/baml (tried pasting code but the formatting is completely off here, sorry). We think we…

Good dx? BAML looks even worse than the current API call based paradigm.

Even your toy examples look bad - wouldn't want to see what an actual program would look like.

Hopefully this, dspy and the like that have poor design, inelegant won't become common standards

Post reply on HN