Do you support custom LLMs?
At the moment only those that support the OpenAI Chat API, with function calling for the structured outputs. For example you can use LocalAI[0][1] to run models locally.
Show HN: Magentic – Use LLMs as simple Python functions
51–60 of 70 posts
Re: Show HN: Magentic – Use LLMs as simple Python functions
#52Earlier quoted context omitted.
Oh, and some companies offer APIs that match the OpenAI API and there are some open-source projects that do this for llama running locally. Since those would be compatible with the openai python package they will work with magentic too - though some of these do not support function calling. See for example Anyscale Endpoints https://app.endpoints.anyscale.com/landing and https://github.com/AmineDiro/cria
There's also LocalAI[0] which allows the use of local LLMs with an OpenAI compatible API. [0] https://github.com/go-skynet/LocalAI
Thanks for sharing! LocalAI supports function calling[0] so this should work for most or all features of magentic - I'm interested to see if concurrent requests work. I will test this out.
Re: Show HN: Magentic – Use LLMs as simple Python functions
#53Awesome job with the simplicity, gonna play with it. Have you tried using yaml as the format with the models instead of JSOn? Feel like you'll use far fewer tokens to describe the same thing. Perhaps it's a bit more forgiving as well. EDIT: Just tried using the decorator to output a fairly complex pydantic model and it failed with "magentic.chat_model.openai_chat_model.StructuredOutputError: Failed to parse model out…
Yes, I'm working on allowing few-shot examples to be provided as part of defining the prompt-function, which should help in cases like this. Unfortunately from my testing just now it appears that OpenAI ignores examples added to the model config.
In the meantime, have a look at the ValidationError traceback which might highlight a specific field that is causing the issue. Some options to resolve the issue might be: the type for this field could be made more lenient (e.g. str); the `Annotated` type hint could be used to give the field a description to help correct the error [0]; the field could be removed. You could also try using gpt-4 by setting the env var MAGENTIC_OPENAI_MODEL [1].
If none of these help resolve it or it appears to be an issue with magentic itself please file a github issue with an example. Comments on how to improve error messages and debugging are also welcome! Thanks for trying it out.
[0] https://docs.pydantic.dev/latest/concepts/fields/#using-anno...
Re: Show HN: Magentic – Use LLMs as simple Python functions
#54Does this do System vs. Assistant vs. User prompting?
Right now we just pass a single user prompt to the chat model. Setting the system prompt could also be done in the `@prompt` decorator. I've added a github issue to track https://github.com/jackmpcollins/magentic/issues/31
Re: Show HN: Magentic – Use LLMs as simple Python functions
#55Re: Show HN: Magentic – Use LLMs as simple Python functions
#56At first I was like: "okay, it's just a decorator to add a prompt when you have str as an input and str as an output.
Then I kept on reading, and I have to admit that the object creation with LLMs is really amazing!
Re: Show HN: Magentic – Use LLMs as simple Python functions
#57Curious as to why you chose to do it as a decorator instead of just a function call?
I found this was the most compact way to represent what I wanted to define, and makes it easy to keep the type hints for parameters. If you look inside `@prompt` it's creating a `PromptFunction` instance which I think would be a similar API to what you would end up with without using decorators https://github.com/jackmpcollins/magentic/blob/afdb22513385b...
I never got on board of decorators in python, but you sold me on it.
Re: Show HN: Magentic – Use LLMs as simple Python functions
#58The API looks very clean. Today I learned about "..." in Python
It is just a noop, but here it looks very appropriate/readable because it reads as saying "AI will fill this in".
It's a misuse of the Python Ellipsis, though PEP has no opinion on it. The Ellipsis is "Special value used mostly in conjunction with extended slicing syntax for user-defined container data types."
In other words, it happens to work and look neat, but pass is the correct way to do it.
Re: Show HN: Magentic – Use LLMs as simple Python functions
#59I am amazed that `...` is a valid syntax in Python, not a pseudo grammar.
This library is impressive, I appreciate it and I will apply it to my project.
Re: Show HN: Magentic – Use LLMs as simple Python functions
#60I am amazed that `...` is a valid syntax in Python, not a pseudo grammar. This library is impressive, I appreciate it and I will apply it to my project.
What's the difference between '...' and the more common 'pass'?