Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

github.com

Post reply on HN
Print view View original

11–20 of 74 posts

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#11

Post by JackHopkins »

ian_dot_so wrote: post #7

This is really interesting! What would be a good example of when I would want to use monkeypatch vs langchain or OpenAI functions?

Thanks! A big part of MonkeyPatch, which Langchain or OpenAI are lacking, is the model distillation aspect, which can reduce costs up to 10x and latency up to 6x in some of the tests we've been running. This means the more you use MonkeyPatch the cheaper the function calls get, which is beneficial for high usage applications with lots of calls

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#12

Post by javidlakha »

JackHopkins wrote: post #11
ian_dot_so wrote: post #7

This is really interesting! What would be a good example of when I would want to use monkeypatch vs langchain or OpenAI functions?

Thanks! A big part of MonkeyPatch, which Langchain or OpenAI are lacking, is the model distillation aspect, which can reduce costs up to 10x and latency up to 6x in some of the tests we've been running. This means the more you use MonkeyPatch the cheaper the function calls get, which is beneficial for high usage applications with lots of calls

How does that work?

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#13

Post by JackHopkins »

javidlakha wrote: post #12
JackHopkins wrote: post #11

Earlier quoted context omitted.

Thanks! A big part of MonkeyPatch, which Langchain or OpenAI are lacking, is the model distillation aspect, which can reduce costs up to 10x and latency up to 6x in some of the tests we've been running. This means the more you use MonkeyPatch the cheaper the function calls get, which is beneficial for high usage applications with lots of calls

How does that work?

Currently we distill the general GPT-4 down to function specific GPT3.5 turbo model using pseudo-labelling. The input-output pairs from the aligned few-shot GPT-4 are saved and this dataset is used to finetune a function-specific GPT3.5 model. Then that finetuned GPT3.5 is switched as the primary model used to carry out the function, which results in multiple times lower costs as the need for few-shot examples is removed and lower latency as well. If the finetuned model output does not follow the enforced constraints, we employ GPT-4 to "repair" the output and include that datapoint in the dataset used for future finetuning resulting in continuous improvements.

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#15

Post by m_vyas123 »

Hey Jack! Thanks for sharing this. The incremental fine-tuning of smaller and cheaper models for cost reduction is definitely a really interesting differentiator. I had a few questions regarding the reliability of the LLM-powered functions MonkeyPatch facilitates and the testing process. How does MonkeyPatch ensure the reliability of LLM-powered functions it helps developers create, and do the tests employed provide sufficient confidence in maintaining consistent output? If tests fall short of 100% guarantee, how does MonkeyPatch address concerns similar to historical challenges faced with testing traditional LLMs? Thanks.

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#16

Post by JackHopkins »

whoiskatrin wrote: post #14

Would love to try a typescript implementation. Any plans to do that?

Great to know! We're working on extending MonkeyPatch to typescript, the work-in-progress repository can be found here https://github.com/monkeypatch/monkey-patch.ts

We will keep you posted on when it'll be ready for trying out!

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#17

Post by javidlakha »

JackHopkins wrote: post #13
javidlakha wrote: post #12

Earlier quoted context omitted.

How does that work?

Currently we distill the general GPT-4 down to function specific GPT3.5 turbo model using pseudo-labelling. The input-output pairs from the aligned few-shot GPT-4 are saved and this dataset is used to finetune a function-specific GPT3.5 model. Then that finetuned GPT3.5 is switched as the primary model used to carry out the function, which results in multiple times lower costs as the need for few-shot examples is rem…

How much control do I have over this process? I might not want this to be abstracted.

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#18

Post by JackHopkins »

javidlakha wrote: post #17
JackHopkins wrote: post #13

Earlier quoted context omitted.

Currently we distill the general GPT-4 down to function specific GPT3.5 turbo model using pseudo-labelling. The input-output pairs from the aligned few-shot GPT-4 are saved and this dataset is used to finetune a function-specific GPT3.5 model. Then that finetuned GPT3.5 is switched as the primary model used to carry out the function, which results in multiple times lower costs as the need for few-shot examples is rem…

How much control do I have over this process? I might not want this to be abstracted.

Currently the distillation happens automatically in the background for all functions but we're aiming to implement ways for the user to be able to turn it off if they wish to keep using the teacher models. Good to know that this'd be a wanted feature!

Re: Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python

#20

Post by jumploops »

whoiskatrin wrote: post #14

Would love to try a typescript implementation. Any plans to do that?

I built a similar library for Typescript: https://github.com/jumploops/magic

Please note: it requires the use of ttypescript or ts-patch, as Typescript transformers aren’t supported by default!

Post reply on HN