Show HN: Shoehorn – Quantize any model down to run on your machine • Hacker News Board

Show HN: Shoehorn – Quantize any model down to run on your machine

#1

Post by rhgraysonii » Tue, Aug 18, 2026, 2:29 PM UTC

Working on Mac, Linux, and Windows now. I include a simple GUI to find new models and get things built and set up. It is working quite well across a few models for me. The GitHub README and DESIGN.md files go into detail of the how/why and it's working remarkably well so far. https://github.com/notactuallytreyanastasio/shoehorn

Show HN: Shoehorn – Quantize any model down to run on your machine
notactuallytreyanastasio.github.io

Re: Show HN: Shoehorn – Quantize any model down to run on your machine

#2

Post by mbuchel-hn » Tue, Aug 18, 2026, 3:19 PM UTC

does this work similar to airllm? i am wondering how it would handle something like quantizing kimi k3 on a budget of 8 gbs, or is that something you are not attempting to solve yet?

Re: Show HN: Shoehorn – Quantize any model down to run on your machine

#3

Post by hmokiguess » Tue, Aug 18, 2026, 3:49 PM UTC

Reminds me of https://github.com/AlexsJones/llmfit

Re: Show HN: Shoehorn – Quantize any model down to run on your machine

#4

Post by jaylane » Tue, Aug 18, 2026, 6:32 PM UTC

tried it out but based on the model sizing result i got i got an insufficient memory error when the server started running

Re: Show HN: Shoehorn – Quantize any model down to run on your machine

#5

Post by rhgraysonii » Tue, Aug 18, 2026, 10:11 PM UTC

hmokiguess wrote: post #3

Reminds me of https://github.com/AlexsJones/llmfit

LLMFit tells you what can run on something. I built something quite similar to their search into Shoehorn now.

Re: Show HN: Shoehorn – Quantize any model down to run on your machine

#6

Post by rhgraysonii » Tue, Aug 18, 2026, 10:11 PM UTC

mbuchel-hn wrote: post #2

does this work similar to airllm? i am wondering how it would handle something like quantizing kimi k3 on a budget of 8 gbs, or is that something you are not attempting to solve yet?

Yes that is exactly what this does.

Re: Show HN: Shoehorn – Quantize any model down to run on your machine

#7

Post by rhgraysonii » Tue, Aug 18, 2026, 10:12 PM UTC

jaylane wrote: post #4

tried it out but based on the model sizing result i got i got an insufficient memory error when the server started running

If you could post an issue if you still have the error around that would be awesome.

Re: Show HN: Shoehorn – Quantize any model down to run on your machine

#8

Post by kennywinker » Wed, Aug 19, 2026, 1:02 AM UTC

rhgraysonii wrote: post #6

mbuchel-hn wrote: post #2

does this work similar to airllm? i am wondering how it would handle something like quantizing kimi k3 on a budget of 8 gbs, or is that something you are not attempting to solve yet?

Yes that is exactly what this does.

Could you explain what happens when you try to shoehorn a 2.4T parameter model into a 24gb m4 mac?

Re: Show HN: Shoehorn – Quantize any model down to run on your machine

#9

Post by akshay_akula » Wed, Aug 19, 2026, 2:05 PM UTC

This is interesting. I wonder how it could work with something like https://github.com/JustVugg/colibri.

Re: Show HN: Shoehorn – Quantize any model down to run on your machine

#10

Post by akshay_akula » Wed, Aug 19, 2026, 2:09 PM UTC

kennywinker wrote: post #8

rhgraysonii wrote: post #6

Earlier quoted context omitted.

Yes that is exactly what this does.

Could you explain what happens when you try to shoehorn a 2.4T parameter model into a 24gb m4 mac?

Wondering the same thing but for 48gb M5 Max.