You could try ours. They're very small and english only right now (training Spanish now). But they're a CPU optimized architecture and even the largest size will work on a single thread with realtime streaming.
You can run them client-side using wasm-simd. I setup a space for the small sized model. There's a big step up in accuracy going to the medium size though:
Hopefully smaller models soon that can be deployed locally for personal use, definitely don't want the speech data sent outside. But having the speech ability is going to improve usefulness of the already great gen AI models drastically!
How did they integrate Voxtral Large exactly? Does it output a special token to indicate it is asking for help, or is it running a command?
Anyway, if it is trained to decide when to think/ask for help that's good.
Too bad it's not open source though. But for large companies in Europe or elsewhere that want to control their data, this might be one of the best options now.
Although the frontier is true full duplex. Anyone know if there is anything that is open source for that other than MoshiRag or PersonaPlex?
Also did the CSM update some months back fix it? I never got a chance to test it.
Interesting. I've found the Voxtral series pretty nice actually. Good transcription and diarization which is the primary personal use-case where I want to turn podcast content into text since I read faster than I listen. The older ones were pretty cheap too and had the nice property that they don't hallucinate text when there's no speech.
The examples they give of how they “improved” the voices aren’t great. Version 1 sounded like actual people, while this version sounds like generic AI slop voice.
I like that a lot of ASR models now offer native context biasing capabilities. If I have enough volume, I would probably just self-host my model and do my own biasing but good enough to roll out custom solutions.
Closed model means I'm immediately not interested. There are lots of closed STT/TTS models that are insanely good, better than any open model, so unless this is open then not sure I'd ever use this.
I think the special sauce of this isn't the STT/TSS (although it does appear to be quite good) - it's the integration of a real-time reasoning model to actually do something with the raw text of the conversation, and whatever they're doing to hide tool call latency if the model needs to search for something.
Closed model unfortunately, continues the theme of TTS / STT being really not great for local deployments.
Too bad it's not open, unlike the previous version.
However, I found Qwen ASR and QwenTTS to be quite good, and not too large. Also https://github.com/kyutai-labs/pocket-tts was also great (but only a few languages)
Funny because Gradium just released a local TTS that runs on many devices
https://x.com/GradiumAI/status/2112563128487514512?s=20
What options would you run locally at this stage? I'm finding a ton of success with local models in this space ..
You could try ours. They're very small and english only right now (training Spanish now). But they're a CPU optimized architecture and even the largest size will work on a single thread with realtime streaming.
https://huggingface.co/collections/futo-org/asr4all
You can run them client-side using wasm-simd. I setup a space for the small sized model. There's a big step up in accuracy going to the medium size though:
https://huggingface.co/spaces/futo-org/asr4all-client-side
Parakeet. For my use it seems about equal to Whisper in WER, but is >10x faster.
I'm looking for a local model that can do diarization, have any suggestions for this?
I have https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize open in a tab to try out
A leader board on HF: https://huggingface.co/spaces/hf-audio/open_asr_leaderboard
I have been using different Parakeet models for STT over the past months and been pretty impressed with the local performance.
Breeze is pretty good imho.
> Available via Mistral’s EU-hosted API or self-deployment
What does this mean then?
You have to call sales to get access for on-prem deployment and the models
If it were open weights, it would be on Huggingface and they would have put a link in the post
Dang, this could have been fun for home assistent.
Hopefully smaller models soon that can be deployed locally for personal use, definitely don't want the speech data sent outside. But having the speech ability is going to improve usefulness of the already great gen AI models drastically!
keep an eye on https://huggingface.co/spaces/hf-audio/open_asr_leaderboard
How did they integrate Voxtral Large exactly? Does it output a special token to indicate it is asking for help, or is it running a command?
Anyway, if it is trained to decide when to think/ask for help that's good.
Too bad it's not open source though. But for large companies in Europe or elsewhere that want to control their data, this might be one of the best options now.
Although the frontier is true full duplex. Anyone know if there is anything that is open source for that other than MoshiRag or PersonaPlex?
Also did the CSM update some months back fix it? I never got a chance to test it.
Interesting. I've found the Voxtral series pretty nice actually. Good transcription and diarization which is the primary personal use-case where I want to turn podcast content into text since I read faster than I listen. The older ones were pretty cheap too and had the nice property that they don't hallucinate text when there's no speech.
The examples they give of how they “improved” the voices aren’t great. Version 1 sounded like actual people, while this version sounds like generic AI slop voice.
I like that a lot of ASR models now offer native context biasing capabilities. If I have enough volume, I would probably just self-host my model and do my own biasing but good enough to roll out custom solutions.
Closed model means I'm immediately not interested. There are lots of closed STT/TTS models that are insanely good, better than any open model, so unless this is open then not sure I'd ever use this.
I think the special sauce of this isn't the STT/TSS (although it does appear to be quite good) - it's the integration of a real-time reasoning model to actually do something with the raw text of the conversation, and whatever they're doing to hide tool call latency if the model needs to search for something.
My main issue with STT models are - WER - price - speed - language support
Wait, did they release any new model or just changed the voxtral page? Voxal Mini 3B rocks hard in Handy!
Closed :(
One more hard question. How real time is this conversation? They're showing something with zero lag between responses.