Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

Hi HN, Jack here! I'm one of the creators of MonkeyPatch, an easy tool that helps you build LLM-powered functions and apps that get cheaper and faster the more you use them.

For example, if you need to classify PDFs, extract product feedback from tweets, or auto-generate synthetic data, you can spin up an LLM-powered Python function in After about 200-300 calls, these functions will begin to get cheaper and faster. We've seen 8-10x reduction in cost and latency in some use-cases! This happens via progressive knowledge distillation - MonkeyPatch incrementally fine-tunes smaller, cheaper models in the background, tests them against the constraints defined by the developer, and retains the smallest model that meets accuracy requirements, which typically has significantly lower costs and latency.

As an LLM researcher, I kept getting asked by startups and friends to build specific LLM features that they could embed into their applications. I realized that most developers have to either 1) use existing low-level LLM clients (GPT4/Claude), which can be unreliable, untyped, and pricey, or 2) pore through LangChain documentation for days to build something.

We built MonkeyPatch to make it easy for developers to inject LLM-powered functions into their code and create tests to ensure they behave as intended. Our goal is to help developers easily build apps and functions without worrying about reliability, cost, and latency, while following best software engineering practices.

We're only available in Python currently but actively working on a Typescript version. The repo has all the instructions you need to get up and running in a few minutes.

The world of LLMs is changing by the day and so we're not 100% sure how MonkeyPatch will evolve. For now, I'm just excited to share what we've been working on with the HN community. Would love to know what you guys think!

Open-source repo: https://github.com/monkeypatch/monkeypatch.py

Sample use-cases: https://github.com/monkeypatch/monkeypatch.py/tree/master/ex...

Benchmarks: https://github.com/monkeypatch/monkeypatch.py#scaling-and-fi...

Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python
github.com

Great questions! The tests act as few-shot examples for the LLMs, which has been shown to guide the style and accuracy of model outputs and improve performance quite well. For instance we’ve seen accuracy go from <70% to 93%+ vs without including the tests. The hallucinations are still an inherent risk with LLMs, especially with long-form context, but adding more diverse and well aligned examples as tests does reduce the hallucination risk and align the outputs with user intent. In terms of performance management and monitoring, QA for LLMs is a difficult process to get right and we’re looking into ways how to a) make it easy for users to test out different function descriptions and tests on their own datasets to gauge performance and b) introduce ways how to seamlessly carry out continuous monitoring of function outputs with low effort. Still WIP but will keep you posted!

this is super cool! what's the use case you're most excited about?

This is really interesting! What would be a good example of when I would want to use monkeypatch vs langchain or OpenAI functions?

The type constraints are indeed enforced but not by the tests but by the type-hints you give to the patched functions. The constraints and enforced structure are followed, there is also a repair feedback loop in place if the original LLM output is invalid for the types you've declared. Tests are more to align the model how to act for different inputs. Hope this makes it more clear!