Whisper – open source speech recognition by OpenAI

Post by dom96 » Wed, Sep 21, 2022, 8:55 PM UTC

MacsHeadroom wrote: post #186

dom96 wrote: post #163

This really makes me want to build a Amazon Echo/Google Nest/etc replacement that's open hardware, open source and most importantly recognises voice completely offline. I find that I don't use these smart devices for much more than setting timers anyway so this seems like an easy project. I just wonder what system requirements Whisper has and whether there are open source voice recognition models that are specificall…

I really want all this too. The smallest model is ~80mb and the largest is 3gb. Not sure about system requirements yet; but models that small suggest this may be doable locally on a single board computer. Edit: According to this comment[0] the base model runs in real time on an M1 CPU. The tiny model apparently decodes an audio file twice as fast. These are promising results. [0] https://news.ycombinator.com/item?id=…

Post by aidenn0 » Wed, Sep 21, 2022, 9:03 PM UTC

aidenn0 wrote: post #119

For those on NixOS, here's a quick and dirty flake.nix that will let you make a venv in which to "pip install"' Just put it in a flake.nix, and "nix develop" followed by "virtualenv ./venv; . ./venv/bin/activate; pip install git+ https://github.com/openai/whisper.git " { description = "Python 3.9 development environment"; outputs = { self, nixpkgs }: let system = "x86_64-linux"; pkgs = import nixpkgs { inherit system…

    {
      description = "Python 3.9 development environment";
      outputs = { self, nixpkgs }:
      let
        system = "x86_64-linux";
        pkgs = import nixpkgs {
          inherit system;
          config.allowUnfree = true;
          config.cudaSupport = true;
        };
      in {
        devShells.${system}.default = pkgs.mkShell {
          buildInputs = with pkgs; [
            cudatoolkit linuxPackages.nvidia_x11
            cudaPackages.cudnn
            libGLU libGL
            xorg.libXi xorg.libXmu freeglut
            xorg.libXext xorg.libX11 xorg.libXv xorg.libXrandr zlib 
            ncurses5 stdenv.cc binutils
            ffmpeg
            python39
            python39Packages.pip
            python39Packages.numpy
            python39Packages.pytorch-bin
            python39Packages.virtualenv
          ];

          shellHook = ''
              export LD_LIBRARY_PATH="${pkgs.linuxPackages.nvidia_x11}/lib"
          '';          
        };
      };
    }

Post by adeptima » Wed, Sep 21, 2022, 9:03 PM UTC

mwlp wrote: post #233

Super impressive. I tested it on a Japanese streamer whose enunciation isn't exactly perfect and it did a decent job: https://www.youtube.com/watch?v=ROiOU1scaNA [00:00.000 --> 00:06.500] Since the last one started, the number of times I've eaten has decreased. [00:06.500 --> 00:11.000] If I get too carried away with the last one, I'll get hungry and do it. [00:11.000 --> 00:14.500] I don't have time to eat. [00:15.5…

Post by lunixbochs » Wed, Sep 21, 2022, 9:05 PM UTC

coder543 wrote: post #194

lunixbochs wrote: post #190

Earlier quoted context omitted.

17x realtime on a 3090 I did some basic tests on CPU, the "small" Whisper model is in the ballpark of 0.5x realtime, which is probably not great for interactive use. My models in Talon run closer to 100x realtime on CPU.

“CPU” isn’t necessarily the benchmark, though. Most smartphones going back years have ML inference accelerators built in, and both Intel and AMD are starting to build in instructions to accelerate inference. Apple’s M1 and M2 have the same inference accelerator hardware as their phones and tablets. The question is whether this model is a good fit for those inference accelerators, and how well it works there, or how w…

  tiny.en: ~18 sec/sec
  base.en: ~14 sec/sec
  small.en: ~6 sec sec/sec
  medium.en: ~2.2 sec/sec
  large: ~1.0 sec/sec (fairly wide variance when ramping up as this is slow to process individual clips)

Post by amrrs » Wed, Sep 21, 2022, 9:14 PM UTC

abidlabs wrote: post #239

Here [1] is a video tutorial on building a web UI that accepts microphone input and runs it through Whisper for speech transcription [1] https://www.youtube.com/watch?v=ywIyc8l1K1Q&ab_channel=1litt...