Show HN: Magentic – Use LLMs as simple Python functions • Hacker News Board

Re: Show HN: Magentic – Use LLMs as simple Python functions

#21

Post by cosmonoot » Tue, Sep 26, 2023, 6:39 PM UTC

jstarfish wrote: post #11

This looks really useful. Langchain is not my idea of a fun time. Love the examples too. Low-effort humor is the best: > create_superhero("Garden Man") > # Superhero(name='Garden Man', age=30, power='Control over plants', enemies=['Pollution Man', 'Concrete Woman'])

Would check out https://www.askmarvin.ai/ if you're into this.

I haven't downloaded 1.5 yet, but they released this last week: https://www.askmarvin.ai/prompting/prompt_function/

Re: Show HN: Magentic – Use LLMs as simple Python functions

#22

Post by pphysch » Tue, Sep 26, 2023, 6:40 PM UTC

This is neat. It makes it easy to prototype, and then you can just remove the decorator and write a specific implementation if you need to.

Re: Show HN: Magentic – Use LLMs as simple Python functions

#23

Post by visarga » Tue, Sep 26, 2023, 6:41 PM UTC

Write some tests for those functions. It will be worth it. No, I am not kidding, especially for AI we need tests, but we should report accuracy instead of a hard fail/pass.

Re: Show HN: Magentic – Use LLMs as simple Python functions

#24

Post by avindroth » Tue, Sep 26, 2023, 6:45 PM UTC

We need a new language/DSL. Python is a lost cause for strings as first-class.

Re: Show HN: Magentic – Use LLMs as simple Python functions

#25

Post by [deleted] » Tue, Sep 26, 2023, 6:50 PM UTC

visarga wrote: post #23

Write some tests for those functions. It will be worth it. No, I am not kidding, especially for AI we need tests, but we should report accuracy instead of a hard fail/pass.

[deleted]

Re: Show HN: Magentic – Use LLMs as simple Python functions

#26

Post by [deleted] » Tue, Sep 26, 2023, 6:50 PM UTC

avindroth wrote: post #24

We need a new language/DSL. Python is a lost cause for strings as first-class.

[deleted]

Re: Show HN: Magentic – Use LLMs as simple Python functions

#27

Post by jumploops » Tue, Sep 26, 2023, 6:58 PM UTC

I built a similar package for Typescript[0], with the goal of having type-safe LLM responses automagically.

It's pretty fun, but I've found that having the LLM write code is often-times what I actually want most of the time.

[0] https://github.com/jumploops/magic

Re: Show HN: Magentic – Use LLMs as simple Python functions

#28

Post by jackmpcollins » Tue, Sep 26, 2023, 7:00 PM UTC

jedberg wrote: post #12

Curious as to why you chose to do it as a decorator instead of just a function call?

I found this was the most compact way to represent what I wanted to define, and makes it easy to keep the type hints for parameters. If you look inside `@prompt` it's creating a `PromptFunction` instance which I think would be a similar API to what you would end up with without using decorators https://github.com/jackmpcollins/magentic/blob/afdb22513385b...

Re: Show HN: Magentic – Use LLMs as simple Python functions

#29

Post by Difwif » Tue, Sep 26, 2023, 7:07 PM UTC

Looks great! I don't normally like these LLM libraries but this one sparks joy. I'll try it out on my next experiment.

Could you highlight how you're parsing to structured objects and how it can fail? Ever since I discovered guidance's method of pattern guides I've been wanting this more and more (only works for local hugging face models though). Wish OpenAI offered a similar API.

Re: Show HN: Magentic – Use LLMs as simple Python functions

#30

Post by jackmpcollins » Tue, Sep 26, 2023, 7:23 PM UTC

conor_f wrote: post #4

Looks super cool! A few questions: 1) Can you get the actual code output or will this end up calling OpenAI each function call? 2) What latency does it add? What about token usage? 3) Is the functionality deterministic?

1) The OpenAI API will be queried each time a "prompt-function" is called in python code. If you provide the `functions` argument in order to use function-calling then magentic will not execute the function the LLM has chosen, instead it returns a `FunctionCall` instance which you can validate before calling.

2) I haven't measured additional latency but it should be negligible in comparison to the speed of generation of the LLM. And since it makes it easy to use streaming and async functions you might be able to achieve much faster generation speeds overall - see the Async section in the README. Token usage should also be a negligible change from calling the OpenAI API directly - the only "prompting" magentic does currently is in naming the functions sent to OpenAI, all other input tokens are written by the user. A user switching from explicitly defining the output schema in the prompt to using function-calling via magentic might actually save a few tokens.

3) Functionality is not deterministic, even with `temperature=0`, but since we're working with python functions one option is to just add the `@cache` decorator. This would save you tokens and time when calling the same prompt-function with the same inputs.

---

1) https://github.com/jackmpcollins/magentic#usage 2) https://github.com/jackmpcollins/magentic#asyncio 3) https://docs.python.org/3/library/functools.html#functools.c...