Show HN: Fructose – LLM calls as strongly typed functions

Hi HN! Erik here from Banana (formerly the serverless GPU platform), excited to show you what we’ve been working on next:

Fructose

Fructose is a python package to call LLMs as strongly typed functions. It uses function type signatures to guide the generation and guarantee a correctly typed output, in whatever basic/complex python datatype requested.

By guaranteeing output structure, we believe this will enable more complex applications to be built, interweaving code with LLMs with code. For now, we’ve shipped Fructose as a client-only library simply calling gpt-4 (by default) with json mode, pretty simple and not unlike other packages such as marvin and instructor, but we’re also working on our own lightweight formatting model that we’ll host and/or distribute to the client, to help reduce token burn and increase accuracy.

We figure, no time like the present to show y’all what we’re working on! Questions, compliments, and roasts welcomed.

Show HN: Fructose – LLM calls as strongly typed functions
github.com

Very Cool! Would it work for Pydantic out of the box? Or that's something coming along?

Can this be a F# Type Provider?

This approach may be too high-level "magic" to the point of being difficult to work with and iterate upon.

Looking at the prompt templates (https://github.com/bananaml/fructose/tree/main/src/fructose/... ), they use LangChain-esque "just try to make the output to be valid JSON" when APIs such as GPT-4 Turbo which this model uses by default now support function calling/structured data natively and do a very good job of it (https://news.ycombinator.com/item?id=38782678), and libraries such as outlines (https://github.com/outlines-dev/outlines) which is more complex but can better ensure a dictionary output for local LLMs.

It's not 100% yet. Route to that: 1. Clientside, retry strategy on failed parse. Not yet implemented, we throw an exception on parse fail right now, but soon to be implemented. Not ideal because of token burn and latency, but the best quick solution. 2. For the custom model we're building, we use strict grammar definitions to bias outputs toward the needed structure (or if there is only one structurally correct token, outright skip the generation of that token and insert it directly).

I've been impressed at how well gpt-4 does with the default prompt template we use. Even better if you enable the chain_of_thought flavor.