Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Willow Inference Server: Optimized ASR/TTS/LLM for Willow/WebRTC/REST

github.com

Post reply on HN
Print view View original

11–15 of 15 posts

Re: Show HN: Willow Inference Server: Optimized ASR/TTS/LLM for Willow/WebRTC/REST

#11

Post by kunwon1 »

kkielhofner wrote: post #10
kunwon1 wrote: post #9

Thanks for this, I've been looking forward to it. I used Amazon Echo devices during their first 6 months of public availability before I got sufficiently creeped out to pull the plug permanently. Since then, I've wished for something similar that wasn't a 'black box' doing unknown things with my data. When you posted about Willow here on HN, I immediately purchased an ESP-BOX (glad I didn't wait, they sold out quickl…

Thanks! One note - if you're going to self-host WIS (YOU SHOULD) I suggest making good use of the more powerful hardware otherwise and putting HA on it. HA on Raspberry Pi (while popular) is pretty slow when compared to the kinds of response times Willow provides. It's frustrating to see perfect speech recognition come back from Willow + WIS in 200ms (or whatever) and then take HA another 150ms to do something with i…

Thanks for the tips. I do plan to self-host WIS. I only have a single GPU currently, on my home desktop, and I'm going to test with this. Next step is going to be looking into a standalone fanless GPU system and determining whether I want to spend money on that or look at used hardware. The hardware adventure is part of the fun.

Re: Show HN: Willow Inference Server: Optimized ASR/TTS/LLM for Willow/WebRTC/REST

#12

Post by kkielhofner »

kunwon1 wrote: post #11
kkielhofner wrote: post #10

Earlier quoted context omitted.

Thanks! One note - if you're going to self-host WIS (YOU SHOULD) I suggest making good use of the more powerful hardware otherwise and putting HA on it. HA on Raspberry Pi (while popular) is pretty slow when compared to the kinds of response times Willow provides. It's frustrating to see perfect speech recognition come back from Willow + WIS in 200ms (or whatever) and then take HA another 150ms to do something with i…

Thanks for the tips. I do plan to self-host WIS. I only have a single GPU currently, on my home desktop, and I'm going to test with this. Next step is going to be looking into a standalone fanless GPU system and determining whether I want to spend money on that or look at used hardware. The hardware adventure is part of the fun.

WIS has been tested with WSL (if you're running Windows) and supports anything from a GTX 1060 3GB up to H100 so you can certainly throw it on your gaming desktop to start.

Re: Show HN: Willow Inference Server: Optimized ASR/TTS/LLM for Willow/WebRTC/REST

#13

Post by unverbraucht »

It looks amazing, I'll definitely try it out! Quick question: would a GTX 960 also work for inference? I happen to have one lying around and could whip up a system with it. Thanks for the great work, I think especially around the Smart home OSS has a lot to add.

Re: Show HN: Willow Inference Server: Optimized ASR/TTS/LLM for Willow/WebRTC/REST

#14

Post by kkielhofner »

unverbraucht wrote: post #13

It looks amazing, I'll definitely try it out! Quick question: would a GTX 960 also work for inference? I happen to have one lying around and could whip up a system with it. Thanks for the great work, I think especially around the Smart home OSS has a lot to add.

Thank you!

Unfortunately the oldest GPUs we support are Pascal and your GTX 960 is Maxwell. We have this cutoff for GPU hardware support for two reasons:

1) Nvidia doesn't support Maxwell with recent versions of CUDA (fair enough, Maxwell is 9 years old).

2) Anything other than a GTX Titan X doesn't have the required VRAM.

That said, we do support and have recommended hardware configurations for cards such as the Tesla P4, GTX 1070, etc which can be had for roughly $100 on the used market.

WIS does support CPU only configurations but GPUs offer such significant fundamental architecture improvements a $100 six year old GPU will best the fastest CPUs in the world for this application at significantly less cost and power usage. A CPU only configuration is fundamentally incapable of providing our target end-user experience - self-hosted, private Alexa without compromise.

Copied from a comment below:

I have hardware arriving this week that I purchased from eBay for $320 with shipping and tax - Dell Precision with i7-7700, 16GB RAM, 512GB SSD[0] and GTX 1070[1]. Seems to be the current "best bang for the buck" if you don't have any of this stuff just sitting around.

The idea is because Willow devices are so cost-effective you can buy this hardware to host WIS, HA, other homelab stuff, etc and still come out ahead (even with power) compared to Raspberry Pi:

Qty 5 Willows - $270 with power supplies (support wake word, far-field audio, LCD display, speaker, mics, etc)

WIS hardware - $320 (or less - it's up to you!)

Total cost: $590

Six Raspberry Pis (no wake word, poor audio quality, unusably slow, cumbersome, very DIY):

$720 (retail kit with board, SD, LCD display, mic array, speaker, enclosure). MSRP - can't actually be purchased for that.

I expect this hardware to do well (returning well below 500ms for Willow speech locally with excellent quality) and I'll be documenting the power consumption optimization work I'm doing over the weekend.

[0] - https://www.ebay.com/itm/234908676168

[1] - https://www.ebay.com/itm/115536328587

Re: Show HN: Willow Inference Server: Optimized ASR/TTS/LLM for Willow/WebRTC/REST

#15

Post by unverbraucht »

kkielhofner wrote: post #14
unverbraucht wrote: post #13

It looks amazing, I'll definitely try it out! Quick question: would a GTX 960 also work for inference? I happen to have one lying around and could whip up a system with it. Thanks for the great work, I think especially around the Smart home OSS has a lot to add.

Thank you! Unfortunately the oldest GPUs we support are Pascal and your GTX 960 is Maxwell. We have this cutoff for GPU hardware support for two reasons: 1) Nvidia doesn't support Maxwell with recent versions of CUDA (fair enough, Maxwell is 9 years old). 2) Anything other than a GTX Titan X doesn't have the required VRAM. That said, we do support and have recommended hardware configurations for cards such as the Tes…

Thanks for the reply - I didn't know about Cuda versions. Makes sense.
Post reply on HN