Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

github.com

Post reply on HN
Print view View original

21–30 of 74 posts

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#21

Post by JackHopkins »

pietz wrote: post #19

Could you explain the differences to Marvin AI? I see a large overlap.

Hey! There are 2 main similarities to Marvin, namely: (a) functions that act as APIs to the LLM backend, and (b) type coercion to ensure that the responses fit into the data model of your application.

However, there are a couple of big additions to Marvin as well, namely: Test Driven Alignment - by using ‘assert’ statements that declare the behavior of a patched function, we create a contract that makes invocations much more predictable, which makes it possible to use these functions in production settings.Automatic distillation - a combination of the function contract defined in function type signature and the alignment tests means we can automatically swap out bigger models for smaller ones. This saves up to 80% of the latency, and 90% of the cost of running these functions (check the benchmarks) Check out the readme, as there is more detail on these points there!

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#22

Post by JackHopkins »

jumploops wrote: post #20
whoiskatrin wrote: post #14

Would love to try a typescript implementation. Any plans to do that?

I built a similar library for Typescript: https://github.com/jumploops/magic Please note: it requires the use of ttypescript or ts-patch, as Typescript transformers aren’t supported by default!

Cool! Thanks for sharing. What do you mean that Ts transformers aren’t supported by default? Is this like a runtime modification of types?

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#25

Post by JackHopkins »

OJFord wrote: post #23

Why 'Monkeypatch', when it's for Python, where that has an established and as far as I can tell (?) completely irrelevant meaning?

(As I understand it) monkey-patch means to modify code in runtime. I thought the naming was relevant because by adding ‘@monkey.patch’ to an unimplemented function, this library gives it an implementation at runtime.

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#26

Post by bb88 »

JackHopkins wrote: post #25
OJFord wrote: post #23

Why 'Monkeypatch', when it's for Python, where that has an established and as far as I can tell (?) completely irrelevant meaning?

(As I understand it) monkey-patch means to modify code in runtime. I thought the naming was relevant because by adding ‘@monkey.patch’ to an unimplemented function, this library gives it an implementation at runtime.

I feel like you're hijacking a term at least 15 to maybe 20 years old though. It's kind of a brilliant marketing idea, but you're just confounding the vocabulary.

It's not to say this isn't a fucking great idea (because it is!). But you know, just don't piss in the community well.

Further, PostgreSQL means one and only one thing. But Monkey Patch means 2 now, apparently!

For reference: Here's the google ngram viewer link for "monkey patch" from 2000 to 2019.

https://books.google.com/ngrams/graph?content=monkey+patch&y...

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#29

Post by JackHopkins »

Scipio_Afri wrote: post #27

Can I use open source LLMs with this? Would be great if everything was available self hosted with open source models.

Native support for OS LLMs is on the roadmap - the main challenge is to figure out how to manage the knowledge distillation for local models. It’s a top priority (along with typescript support), so check back in a few weeks?

Right now any ‘plug-in’ model needs to conform to the OpenAI API.

(P.S Carthago delenda est)

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#30

Post by JackHopkins »

CyberDildonics wrote: post #28

MonkeyPatch is a specific programming term that people have been using for decades. What would posses someone to name a programming tool "MonkeyPatch" when the tool doesn't even have something to do with patching?

Monkeypatch (as I understand it) means to modify code at runtime. This library modifies functions at runtime to use an LLM as an execution target - I thought it was an apt (if admittedly cheeky) name! Appreciate the critique regardless.
Post reply on HN