Nice. I did transcriptions on a casual project before that went through something like this. Transcribing videos or audio files with Whisper? Very common. But having to swap it out with Qwen3 or a different family of ASR models? Oops, not as straightforward. For Qwen for example you gotta deal with the forced aligner or it won't be good as subtitles, and then gotta deal with some requirements and considerations if you want to make use of MLX on a Mac or something.
Will definitely check this out since it sounds like it eases through the pain of dealing with these.
Handy is an amazing cross-platform app for dictation from the author. There are other awesome open-source dictation tools as well like native macOS ones. You do not need SaaS subscription in this day and age for transcription.
I maintain this list of all the best open-source ones in this awesome-style GitHub repo. People looking for open-source dictation tools, hope you find something that works for you here:
Congrats on shipping this. I love handy on my Mac, my phone for STT in situations where it’s not possible/poor performance of the native
Model for STT(e.g apple’s thing is not upto scruff, like mistranslating words corresponding to a domain).
Noob question: How do you think about funding from a foundation(i have no clue if you need it or not, I do hope you have a way to get paid one way or another because handy is amazing) for maintenance of this? if you did or were going to get paid by asking for maintaining such a project what might be the kind of organizations you would look for to get supported and how would you do it?
Thanks! What an excellent question, I’m not sure I have a good answer. I kind of became an open source maintainer by accident as Handy became popular
Certainly I am very lucky that quite a few people donate to Handy, and also some people and organizations who sponsor the work I do
To be honest I just love contributing to open source and wish to continue to do so. So anyone who supports this is good to me. Organizations which believe in OSS and push it forward are typically most aligned with me
Of course you can always email me (contact@handy.computer) and we can discuss in more detail
OS-native dictation on iOS requires uploading your address book to Apple on every request, even if you don’t use iCloud. I unfortunately have to leave it disabled for this reason.
For anyone looking to build on top of this. I have tried a few different STT systems, and they accurately capture what I am saying. Unfortunately, they don't support the reasonable workflow
I want to open an office document, for example, and start talking. And I want the software to continuously type what I am saying at the cursor with minimal latency. The continuous part is crucial. Many software will paste whatever I said after I have stopped recording, but that is not useful.
Totally understandable, but I’ve found that software that transcribes everything after I finish recording actually works better for me. I’ve tried both kinds, and systems that continuously type what I’m saying distract me from completing my thought. I end up reading what’s being typed and noticing transcription mistakes instead of focusing on what I’m trying to say.
I often prefer to dictate everything in my head about a particular thing for 5–10 minutes and then go through it afterward. I find that much more useful because it doesn’t break my thought process the way continuous transcription does.
I suspect the hard bit is that it sometimes needs to back up and redo, and that's an interface they haven't got figured out. I'm fairly sure I remember Dragon Naturally Speaking doing it in Word years ago though, so the interfaces should be there.
But this is still possible to do if you track the whole run of text. You could replace all of it each time so it LOOKS like it’s streaming but earlier words also change. I’m hoping the streaming models do this eventually.
I believe the built-in iOS dictation already does this.
More accuracy. Like others have said, homonyms (their, they're, there) is easier to determine once you have more context. So then you may need to go back a couple words and update them.
Same with punctuation, you could determine that a comma belonged in a certain place once you have enough words.
In iOS this means you can edit the text as it’s being transcribed. For example, I want to dictate a todo list and after each item I can hit enter to go to the next line.
It may be interesting to have it immediately insert the words, even if they are wrong, and when a sentence is finished, replace what has been written with the final corrected sentence.
Google released this awkwardly named app called edge eloquent recently that does exactly that.
In fact, it cleans up the entire paragraph that you just said, and even if you have meandering thoughts, it cleans those up too.
Actually, this above statement was fully dictated with iOS and it added all the punctuation automatically, so I think that iOS is also doing some of this natively. In fact, I’m on the iOS 27 beta and it seems to be doing an even better job of correcting itself and correcting earlier words and adding punctuation too.
Model should be able to understand where logical sentence ends, to stop buffering, and optionally rewrite some of the test that has already been output.
What good timing to spot this. I've been reading more and more people talk about bringing TTS into their prompting toolkit and wanted to give that a try. The idea of rambling brain dump into a doc -> edit pass -> send to the robot loop sounds appealing.
Amazing, i've been looking for something like this and ended up doing transcription + diarization on a local server for now. Are you looking for contributions? Have you tried this one for diarization - https://huggingface.co/pyannote/speaker-diarization-communit... - it performed much better than Sortformer for me.
Contributions are always welcome! There’s a WIP diarization PR rn, and after it’s merged would love to have support if it fits well into the interface. And if not would love to figure out a good interface for it
Nice. Here's the Python one: https://github.com/handy-computer/transcribe.cpp/tree/main/b... - looks like it's not yet available as a binary wheel on PyPI with the dependency included (the library on PyPI right now uses ctypes to call a separately installed library) but that's planned for a future release.
Yep, but I am in the process of also porting NVIDIAs Sortformer for multi speaker diarization as well :)
I’m not sure how many specific models will be supported as the library is more focused on transcription specifically. But the models which support diarization natively must be supported I think. And parakeet multitalker was the primary driving force for this change
How close do you aim for when it comes to drop-in vs whisper.cpp? Are timestamps per word and character something aimed for? How about multi-lingual transcription or hallucination suppression?
The github page doesn't seem to go into depth on these orthogonal topics. May have missed it.
> I think as we look forward to the future, more inference will start happening locally for one reason or the other. This brings the distribution story front and center. In order to have more applications running inference locally, we need to make running inference easier.
This makes these projects so much more trustworthy and easier to approach:
> Were any of the words here written using AI? Nope. They came from my mouth or my fingers.
>This makes these projects so much more trustworthy and easier to approach:
>> Were any of the words here written using AI? Nope. They came from my mouth or my fingers.
I have to push back on this a bit, as I believe (quite strongly) that we're shaped by the tools we use; text-to-speech LLMs are still LLMs, and generally their mistakes are shaped by the expectations inherent in their training. This, in turn, shapes the words that appear on the screen. For those who regularly use them, you then learn which word sequences are likely to be accurately transcribed, and this definitively becomes part of your thinking process. Over time, the LLM becomes tangled into your thinking; the use of AI, even in this way, very much can and often does shape the resulting words.
Isn't this like saying "my words are not really my own when I speak to my family, because I know my father is a non-native English speaker and hard of hearing so I try to use words which are well enunciated and are few in syllable count"?
You can take it one step further! As Tyutchev wrote, "A thought once uttered is a lie." [1] Speech is a projection of a thought, and a lossy one. So no matter who is the listener, the speaking/writing does affect the thinking. Though comment on LLM transcribing is spot on.
This is an incredible contribution to the community and it's just... one guy? I kept reading expecting a Series A funding announcement at the bottom.
It's a nice reminder: You can use AI to slop cannon at maximum speed, or you can use it to scale your ambitions and build something more rigorous and lasting than ever before.
I'd build Transcribe.cpp into the apps I maintain, but I feel like this functionality should (generally) be integrated into the OS or "everywhere" via an app like Handy.
Hey, yep author and maintainer here! Certainly sponsors help and the wonderful community who donates to Handy as well! Mozilla AI was very helpful in getting this work off the ground. It was a pipe dream for me to build for Handy and they helped to sponsor me so I could make time to take this project seriously and get a v0.1.0 release out the door
I agree this should be everywhere and I hope to distribute libtranscribe some day properly so it is more a system library! It will take time to stabilize but I think we can get there
Not in the library itself, it’s pure inference. Some models have this trained out of them anyhow. Otherwise this is a post processing task which is not really inference
What's the best local TTS model right now? I'm running parakeet on a mac which transcribes all my uh's and aahs. I'm running whisper on linux/cuda and I by far prefer that one over parakeet.
Congrats on delivering good value to the people. I have used transcribe.cpp a few weeks ago to do near realtime offline stt on a 10 year old phone, writing simple adhoc app for my use case, it's crazy what is happening right now.
Nice - I'm definitely going to take a look at this. I've built my own cross-platform (Mac/Win/Linux) live captioning app on top of Nemotron, and it works well but dealing with ONNX is kind of annoying. With this having Rust support (I built it on Rust/Tauri) it should be a pretty solid candidate; I'll have to see if I can find a Silero VAD implementation that doesn't depend on ONNX, or maybe I'll see if the clankers can migrate it for me.
Nice. I did transcriptions on a casual project before that went through something like this. Transcribing videos or audio files with Whisper? Very common. But having to swap it out with Qwen3 or a different family of ASR models? Oops, not as straightforward. For Qwen for example you gotta deal with the forced aligner or it won't be good as subtitles, and then gotta deal with some requirements and considerations if you want to make use of MLX on a Mac or something.
Will definitely check this out since it sounds like it eases through the pain of dealing with these.
Handy is an amazing cross-platform app for dictation from the author. There are other awesome open-source dictation tools as well like native macOS ones. You do not need SaaS subscription in this day and age for transcription.
I maintain this list of all the best open-source ones in this awesome-style GitHub repo. People looking for open-source dictation tools, hope you find something that works for you here:
https://github.com/primaprashant/awesome-voice-typing
Congrats on shipping this. I love handy on my Mac, my phone for STT in situations where it’s not possible/poor performance of the native Model for STT(e.g apple’s thing is not upto scruff, like mistranslating words corresponding to a domain).
Noob question: How do you think about funding from a foundation(i have no clue if you need it or not, I do hope you have a way to get paid one way or another because handy is amazing) for maintenance of this? if you did or were going to get paid by asking for maintaining such a project what might be the kind of organizations you would look for to get supported and how would you do it?
Thanks! What an excellent question, I’m not sure I have a good answer. I kind of became an open source maintainer by accident as Handy became popular
Certainly I am very lucky that quite a few people donate to Handy, and also some people and organizations who sponsor the work I do
To be honest I just love contributing to open source and wish to continue to do so. So anyone who supports this is good to me. Organizations which believe in OSS and push it forward are typically most aligned with me
Of course you can always email me (contact@handy.computer) and we can discuss in more detail
OS-native dictation on iOS requires uploading your address book to Apple on every request, even if you don’t use iCloud. I unfortunately have to leave it disabled for this reason.
handy has been invaluable in my workflow, and having a fast, local, c++-based transcription library with first-party ts bindings is incredibleee
tysm for shipping this, keep up the great work OP
For anyone looking to build on top of this. I have tried a few different STT systems, and they accurately capture what I am saying. Unfortunately, they don't support the reasonable workflow
I want to open an office document, for example, and start talking. And I want the software to continuously type what I am saying at the cursor with minimal latency. The continuous part is crucial. Many software will paste whatever I said after I have stopped recording, but that is not useful.
Totally understandable, but I’ve found that software that transcribes everything after I finish recording actually works better for me. I’ve tried both kinds, and systems that continuously type what I’m saying distract me from completing my thought. I end up reading what’s being typed and noticing transcription mistakes instead of focusing on what I’m trying to say.
I often prefer to dictate everything in my head about a particular thing for 5–10 minutes and then go through it afterward. I find that much more useful because it doesn’t break my thought process the way continuous transcription does.
You know English doesn't work like that. The word you're saying only becomes clear with the surrounding context. Eg, 'there' vs 'their'.
Handy already supports streaming transcription models, and you can see the words in the small Handy pop-up while you are talking.
So in general this definitely works. Handy is just missing the feature to insert these streamed words into the app where the cursor is.
I suspect the hard bit is that it sometimes needs to back up and redo, and that's an interface they haven't got figured out. I'm fairly sure I remember Dragon Naturally Speaking doing it in Word years ago though, so the interfaces should be there.
But this is still possible to do if you track the whole run of text. You could replace all of it each time so it LOOKS like it’s streaming but earlier words also change. I’m hoping the streaming models do this eventually.
I believe the built-in iOS dictation already does this.
What would be the benefit of this, besides from looking cool?
More accuracy. Like others have said, homonyms (their, they're, there) is easier to determine once you have more context. So then you may need to go back a couple words and update them.
Same with punctuation, you could determine that a comma belonged in a certain place once you have enough words.
In iOS this means you can edit the text as it’s being transcribed. For example, I want to dictate a todo list and after each item I can hit enter to go to the next line.
It may be interesting to have it immediately insert the words, even if they are wrong, and when a sentence is finished, replace what has been written with the final corrected sentence.
Google released this awkwardly named app called edge eloquent recently that does exactly that.
In fact, it cleans up the entire paragraph that you just said, and even if you have meandering thoughts, it cleans those up too.
Actually, this above statement was fully dictated with iOS and it added all the punctuation automatically, so I think that iOS is also doing some of this natively. In fact, I’m on the iOS 27 beta and it seems to be doing an even better job of correcting itself and correcting earlier words and adding punctuation too.
Model should be able to understand where logical sentence ends, to stop buffering, and optionally rewrite some of the test that has already been output.
IIRC that is exactly how Dragon Naturally speaking did it decades ago.
You can fairly easily modify [Handy](https://handy.computer) to do this if you want
I’m planning on having it as a first class feature of the app too just too many other issues to work on first
This is what I attempted with https://github.com/electronstudio/low_latency_dictation
However the accuracy of the real time models is poor, so I did a second pass with a higher accuracy model before committing the text.
You used to be able to do this with dragon naturally speaking (don’t remember if that was it’s exact name) 10 ish years ago
Agreed. It's something I've found annoying about a few systems.
It looks like the rust bindings have streaming examples so hopefully there is a nice solution here.
What good timing to spot this. I've been reading more and more people talk about bringing TTS into their prompting toolkit and wanted to give that a try. The idea of rambling brain dump into a doc -> edit pass -> send to the robot loop sounds appealing.
Amazing, i've been looking for something like this and ended up doing transcription + diarization on a local server for now. Are you looking for contributions? Have you tried this one for diarization - https://huggingface.co/pyannote/speaker-diarization-communit... - it performed much better than Sortformer for me.
Contributions are always welcome! There’s a WIP diarization PR rn, and after it’s merged would love to have support if it fits well into the interface. And if not would love to figure out a good interface for it
> Maintainer supported bindings in 4 Languages
Nice. Here's the Python one: https://github.com/handy-computer/transcribe.cpp/tree/main/b... - looks like it's not yet available as a binary wheel on PyPI with the dependency included (the library on PyPI right now uses ctypes to call a separately installed library) but that's planned for a future release.
Yes, I’ve put a PR up on pypi for extra storage for CUDA but it has not been accepted yet afaik
If there’s any issues or improvements on the bindings I would love help to make the DX the best it can be
What's the easiest way to add speaker separation to this?
Hey! It’s actually in progress right now, probably will come this week :)
Awesome! I found the in-progress diarization PR here: https://github.com/handy-computer/transcribe.cpp/pull/85
Looks like it's using IBM's Granite-Speech-4.1-2B-Plus https://huggingface.co/ibm-granite/granite-speech-4.1-2b-plu... and/or MOSS-Transcribe-Diarize https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize
Yep, but I am in the process of also porting NVIDIAs Sortformer for multi speaker diarization as well :)
I’m not sure how many specific models will be supported as the library is more focused on transcription specifically. But the models which support diarization natively must be supported I think. And parakeet multitalker was the primary driving force for this change
How close do you aim for when it comes to drop-in vs whisper.cpp? Are timestamps per word and character something aimed for? How about multi-lingual transcription or hallucination suppression?
The github page doesn't seem to go into depth on these orthogonal topics. May have missed it.
Spot on:
> I think as we look forward to the future, more inference will start happening locally for one reason or the other. This brings the distribution story front and center. In order to have more applications running inference locally, we need to make running inference easier.
This makes these projects so much more trustworthy and easier to approach:
> Were any of the words here written using AI? Nope. They came from my mouth or my fingers.
>This makes these projects so much more trustworthy and easier to approach:
>> Were any of the words here written using AI? Nope. They came from my mouth or my fingers.
I have to push back on this a bit, as I believe (quite strongly) that we're shaped by the tools we use; text-to-speech LLMs are still LLMs, and generally their mistakes are shaped by the expectations inherent in their training. This, in turn, shapes the words that appear on the screen. For those who regularly use them, you then learn which word sequences are likely to be accurately transcribed, and this definitively becomes part of your thinking process. Over time, the LLM becomes tangled into your thinking; the use of AI, even in this way, very much can and often does shape the resulting words.
Isn't this like saying "my words are not really my own when I speak to my family, because I know my father is a non-native English speaker and hard of hearing so I try to use words which are well enunciated and are few in syllable count"?
You can take it one step further! As Tyutchev wrote, "A thought once uttered is a lie." [1] Speech is a projection of a thought, and a lossy one. So no matter who is the listener, the speaking/writing does affect the thinking. Though comment on LLM transcribing is spot on.
1. https://www.poetryloverspage.com/poets/tyutchev/silentium/li...
This is an incredible contribution to the community and it's just... one guy? I kept reading expecting a Series A funding announcement at the bottom.
It's a nice reminder: You can use AI to slop cannon at maximum speed, or you can use it to scale your ambitions and build something more rigorous and lasting than ever before.
I'd build Transcribe.cpp into the apps I maintain, but I feel like this functionality should (generally) be integrated into the OS or "everywhere" via an app like Handy.
Hey, yep author and maintainer here! Certainly sponsors help and the wonderful community who donates to Handy as well! Mozilla AI was very helpful in getting this work off the ground. It was a pipe dream for me to build for Handy and they helped to sponsor me so I could make time to take this project seriously and get a v0.1.0 release out the door
I agree this should be everywhere and I hope to distribute libtranscribe some day properly so it is more a system library! It will take time to stabilize but I think we can get there
Does this support filtering of “umm”,”err”, “ugh”, or that is nit yet possible with open source models?
Not in the library itself, it’s pure inference. Some models have this trained out of them anyhow. Otherwise this is a post processing task which is not really inference
What's the best local TTS model right now? I'm running parakeet on a mac which transcribes all my uh's and aahs. I'm running whisper on linux/cuda and I by far prefer that one over parakeet.
Parakeet unified for me no longer does this and it’s also a streaming transcription model!
But the answer largely depends on you, the languages you speak, and personal preference. Whisper is still excellent and supported in transcribe.cpp
Cohere Transcribe is also excellent, but many of the new models are as well
> parakeet on a mac which transcribes all my uh's and aahs
You should be able to fix this by playing with the mic speech floor. It happens when to much ambient stuff slurps in.
It's actually gaslighting you, you don't say that many ums and ahs ;)
I saw that metal is almost x10 faster than vulkan? Why so much gap?
It very much depends on the hardware! An M4 max is being compared against a Ryzen 4750U with an integrated GPU!
The M4 max has probably 10x the compute and memory bandwidth hahaha
So it's mostly intended to be a better replacement for whisper? Mostly? With better support for more models and maybe acceleration backends?
More or less yes, for whisper.cpp, just trying to make local transcription more accessible to anyone building an app, etc
Excellent work CJ
Thanks :)
Excellent work, paired with the 500kb TTS model headlining today I can see the full stack coming together.
saw the demo its impressive but the audio was robotic
Is there a way to add speaker identification easily?
Hey! It’s actually in progress right now, probably will come this week :)
That would be Diarize.cpp, not Transcribe.cpp.
I don't suppose this works in the browser?
Out of the box no probably not, but if people are interested there’s probably ways forward
Excellent work
Congrats on delivering good value to the people. I have used transcribe.cpp a few weeks ago to do near realtime offline stt on a 10 year old phone, writing simple adhoc app for my use case, it's crazy what is happening right now.
Ha amazing, love to hear it
Nice - I'm definitely going to take a look at this. I've built my own cross-platform (Mac/Win/Linux) live captioning app on top of Nemotron, and it works well but dealing with ONNX is kind of annoying. With this having Rust support (I built it on Rust/Tauri) it should be a pretty solid candidate; I'll have to see if I can find a Silero VAD implementation that doesn't depend on ONNX, or maybe I'll see if the clankers can migrate it for me.
Looks interesting, I'll give it a try. Though I'm really happy with faster-whisper on a GPU.
would love to see a demo handy is fantastic although its still behind the frontier models
Pretty sure I saw Handy using it; if you have the latest version, you’re probably already demoing it.
Yep the latest version has support! Virtually all of the SOTA open models are supported by Handy including the streaming ones like
Nemotron Streaming
Parakeet Unified
Voxtral Mini Realtime
If something you want is not supported, open an issue on transcribe.cpp!
author of the blogpost is the maintainer of Handy, so almost guaranteed!
I installed it but I don't think I see the streaming transcriptions. I do think the transcription is a bit faster. I am using the latest version.
You have to change the model to one that supports streaming. The latest parakeet does. I've been using it the last week or so. It's good stuff :)