Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Willow – Open-source privacy-focused voice assistant hardware

github.com

Post reply on HN
Print view View original

41–50 of 141 posts

Re: Show HN: Willow – Open-source privacy-focused voice assistant hardware

#41

Post by kkielhofner »

dingledork69 wrote: post #8

Cool! What software is used for the wake word detection, speech to text and text to speech?

For wake word and voice activity detection, audio processing, etc we use the ESP SR (speech recognition) framework from Espressif[0].

For speech to text there are two options and more to come:

1) Completely on device command recognition using the ESP SR Multinet 6 model. Willow will (currently) pull your light and switch entities from Home Assistant and generate the grammar and command definition required by Multinet. We want to develop a Willow Home Assistant component that will provide tighter Willow integration with HA and allow users to do this point and click with dynamic updates for new/changed entities, different kinds of entities, etc all in the HA dashboard/config.

The only "issue" with Multinet is that it only supports 400 defined commands. You're not going to get something like "What's the weather like in $CITY?" out of it.

For that we have:

2-?) Our own highly optimized inference server using Whisper, LLamA/Vicuna, and Speecht5 from transformers (more to come soon). We're open sourcing it next week. Willow streams audio after wake in realtime, gets the STT output, and sends it wherever you want. With the Willow Home Assistant component (doesn't exist yet) it will sit in between our inference server implementation doing STT/TTS or any other STT/TTS implementation supported by Home Assistant and handle all of this for you.

[0] - https://github.com/espressif/esp-sr

Re: Show HN: Willow – Open-source privacy-focused voice assistant hardware

#42

Post by kkielhofner »

freedomben wrote: post #39
kkielhofner wrote: post #24

Earlier quoted context omitted.

Thanks! HN was my first stop (of course) - I'll be heading over there shortly to post. Oh yeah, we're well aware of how much of a "pain" getting Willow going can be. I don't like it (at all). That said, you configure and build once for your environment and then get a .bin that can be flashed to the ESP BOX with anything that does ESP flashing (like various web interfaces, etc) or you can re-run the flash command acro…

IMHO better to release early like this to a group of hackers than to wait until you have a nice out of the box setup going. This way you're going to get a lot of great feedback and hopefully some help. Awesome project!

Bingo, thanks!

Re: Show HN: Willow – Open-source privacy-focused voice assistant hardware

#43

Post by kkielhofner »

vlugorilla wrote: post #37
kkielhofner wrote: post #15

Earlier quoted context omitted.

Our inference server (open source - releasing next week) has support for loading LLaMA and derivative models complete with 4-bit quantization, etc. I like Vicuna 13B myself :). Not to mention extremely fast and memory optimized Whisper via ctranslate2 and a bunch of our own tweaks. Our inference server also supports long-lived sessions via WebRTC for transcription, etc applications ;). You can chain speech to text ->…

Awesome work! May I ask what are you using for text-to-speech?

Thanks, of course!

For wake word and voice activity detection, audio processing, etc we use the ESP SR (speech recognition) framework from Espressif[0]. For speech to text there are two options and more to come:

1) Completely on device command recognition using the ESP SR Multinet 6 model. Willow will (currently) pull your light and switch entities from Home Assistant and generate the grammar and command definition required by Multinet. We want to develop a Willow Home Assistant component that will provide tighter Willow integration with HA and allow users to do this point and click with dynamic updates for new/changed entities, different kinds of entities, etc all in the HA dashboard/config.

The only "issue" with Multinet is that it only supports 400 defined commands. You're not going to get something like "What's the weather like in $CITY?" out of it.

For that we have:

2-?) Our own highly optimized inference server using Whisper, LLamA/Vicuna, and Speecht5 from transformers (more to come soon). We're open sourcing it next week. Willow streams audio after wake in realtime, gets the STT output, and sends it wherever you want. With the Willow Home Assistant component (doesn't exist yet) it will sit in between our inference server implementation doing STT/TTS or any other STT/TTS implementation supported by Home Assistant and handle all of this for you - including chaining together other HA components, APIs, etc.

[0] - https://github.com/espressif/esp-sr

Re: Show HN: Willow – Open-source privacy-focused voice assistant hardware

#45

Post by canadiantim »

kkielhofner wrote: post #35
canadiantim wrote: post #32

Wow this looks beyond epic. I've been looking for something like this. Going to try to hack this into something my mom can use (who has trouble with confusion and memory). Could potentially be very great. Thank you

Thanks! We are really, truly, and seriously committed to building a device that with support from Home Assistant and other integrations doesn't leave any reason whatsoever to buy an Echo or similar creepy commercial device. No compromises on cost, performance, accuracy, speed, usability, functionality, etc. We're really looking forward to getting additional testing and feedback from the community on speech recognitio…

Wow yeah I think you're really onto something here. No one actually wants the creepiness from Echo or Alexa etc. That's what prevented me from trying any Home Assistant thing before, but I know it could be very useful if actually sensitive to privacy-concerns.

Best of luck with the development! I'll definitely be following closely. Do you sell the pre-built hardware yourself?

Re: Show HN: Willow – Open-source privacy-focused voice assistant hardware

#46

Post by kkielhofner »

mikece wrote: post #10

I love the privacy-focused aspect but playing devil's advocate: how could a device like this be hijacked and used for anti-privacy purposes? Does this require physical access or has it been subjected to the likes of the Black Hat conference to see if it can be owned from the street outside someone's home?

It's Wifi client only and supports WPA3, protected management frames, etc, etc. It doesn't listen on any network sockets. Even bluetooth is currently disabled.

Other than low level issues in the Espressif wifi stack (which is very robust, mature, and has been beat on heavily) I don't see any potential security issues.

That said the old expression "it's easy for someone to design a lock they can't pick" certainly applies.

We'd welcome someone owning it and bringing any issues to our attention!

Re: Show HN: Willow – Open-source privacy-focused voice assistant hardware

#48

Post by discardedrefuse »

So I was just looking at the installation process for this device's dev environment (ESP-IDF from espresiff) and it seems kind of...insane.

The manual install method in the directions is not manual at all. It's a script that calls several python scripts. One has 2660 LOC and installs a root certificate (hard coded in the script itself) because of course, even though you just cloned the whole repo, it still has to download stuff from the internet. According to the code, "This works around the issue with outdated certificate stores in some installations".

Does anyone familiar with espressiff have an actual manual method of installing a dev environment for this device that doesn't involved pwning myself?

Re: Show HN: Willow – Open-source privacy-focused voice assistant hardware

#49

Post by r1cht3r »

stavros wrote: post #40
kkielhofner wrote: post #36

Earlier quoted context omitted.

You never know if people are going to love your pet project as much as you do. We had a hunch the community would appreciate Willow but like I said, you just never know. My suspicion is Espressif (until now, hah) hasn't sold a lot of ESP Boxes. We were concerned that if Willow takes off they will sell out. That already appears to be happening. Espressif has tremendous manufacturing capacity and we hope they will scal…

I really hope so, I've been waiting for good audio assistant hardware forever. I hope this is finally the time where I ditch Alexa once and for all, thanks for releasing Willow!

fwiw I found them in stock on adafruit.com

Re: Show HN: Willow – Open-source privacy-focused voice assistant hardware

#50

Post by detaro »

discardedrefuse wrote: post #48

So I was just looking at the installation process for this device's dev environment (ESP-IDF from espresiff) and it seems kind of...insane. The manual install method in the directions is not manual at all. It's a script that calls several python scripts. One has 2660 LOC and installs a root certificate (hard coded in the script itself) because of course, even though you just cloned the whole repo, it still has to dow…

yes, do it in a container or VM. Welcome to the wonderful world of hardware manufacturer SDKs.
Post reply on HN