I bought an iBook G3 specifically for running Debian back in the early 2000s. It worked absolutely perfectly and one of the huge features I miss was that the small option key to the right of the space bar was converted to a middle mouse button :)
My problem with these types of apps will always be that most software it interacts with will unfortunately be designed without accessibility in mind.
I've trialed a handful of these mouse-less programs before and I always run into the same hurdles.
E.g. where one or more of my apps have sliders which I can't interact with using clicks instead of drags (so I can't interact with it at all), or needing to click things like my MacOS dock, or even resize a window.
Hello, I'm the developer behind Neverclick. Neverclick doesn't use accessibility apis. It uses local computer vision so it works in every app.
I've had a poor experience with accessibility apis, they're clunky, slow, and unpredictable, and as you said many apps aren't built with accessibility in mind. With computer vision you don't have to worry about that.
This might sound strange, but the cv system in Neverclick actually runs significantly faster than UI Automation (which is the accessibility api on windows) in nearly all of my personal tests. I can't believe that accessibility apis are so poorly optimized that raw pixel analysis is faster lol. Also, I have users that use Neverclick on 10 year old hardware and they tell me that the cv runs instantly whereas UI Automation is super laggy for them.
Also, accessibility apis typically don't give you useful bounding boxes for text in text editors, this is another advantage of the cv approach.
I really wish the title of the post was "Desktop application for performing mouse actions with your keyboard using computer vision" so that it's more obvious that it uses cv, although it's my fault since the current title is how I have it in the readme and they probably just copied that.
Slightly related. On wayland based linux desktops this would never work because an app is not allowed to hijack the mouse pointer to dispatch events. I believe only the compositor is allowed to do that.
I know, because I wanted to automate a GUI task using PyAutoGUI and it wouldnt work on wayland no matter what.
This is really nice, great job! I built something similar for myself on MacOS recently. Started off as a window manager then began adding keyboard navigation features as I was unhappy with other tools (vimium, shortcat etc).
Hint mode as a feature being able to click all active links on the screen is great and very useful.
Two features I've added recently that have made never using a mouse much more feasible:
1. Search mode - Search for any text on the page, jump to it and then use vim keybindings to select text from there and copy => I use this 20+ times a day
2. Scroll mode - Any scroll area on the page is highlighted with a hint, but when selected it clicks the scroll area and allows you to use keybindings like j/k/gg/G to scroll up and down. I loved this from vimium and wanted to everywhere
This looks cool! How is the planned Linux support (the website says "Coming soon for Linux") suppose to work? X11 only? Wayland? Gnome only (or any other DE)?
thanks for the heads up, I have a keychron gathering dust at home, so I'll test this later on. favved the thread anyway, since I'm interested, but will wait for the macOS version.
I bought an iBook G3 specifically for running Debian back in the early 2000s. It worked absolutely perfectly and one of the huge features I miss was that the small option key to the right of the space bar was converted to a middle mouse button :)
I tested also https://omarchy.org/ ...
My problem with these types of apps will always be that most software it interacts with will unfortunately be designed without accessibility in mind.
I've trialed a handful of these mouse-less programs before and I always run into the same hurdles.
E.g. where one or more of my apps have sliders which I can't interact with using clicks instead of drags (so I can't interact with it at all), or needing to click things like my MacOS dock, or even resize a window.
Hello, I'm the developer behind Neverclick. Neverclick doesn't use accessibility apis. It uses local computer vision so it works in every app.
I've had a poor experience with accessibility apis, they're clunky, slow, and unpredictable, and as you said many apps aren't built with accessibility in mind. With computer vision you don't have to worry about that.
This might sound strange, but the cv system in Neverclick actually runs significantly faster than UI Automation (which is the accessibility api on windows) in nearly all of my personal tests. I can't believe that accessibility apis are so poorly optimized that raw pixel analysis is faster lol. Also, I have users that use Neverclick on 10 year old hardware and they tell me that the cv runs instantly whereas UI Automation is super laggy for them.
Also, accessibility apis typically don't give you useful bounding boxes for text in text editors, this is another advantage of the cv approach.
I really wish the title of the post was "Desktop application for performing mouse actions with your keyboard using computer vision" so that it's more obvious that it uses cv, although it's my fault since the current title is how I have it in the readme and they probably just copied that.
What is needed is "Mouseless Friday" for UI designers/developers.
Slightly related. On wayland based linux desktops this would never work because an app is not allowed to hijack the mouse pointer to dispatch events. I believe only the compositor is allowed to do that.
I know, because I wanted to automate a GUI task using PyAutoGUI and it wouldnt work on wayland no matter what.
This is why I'm still on X11, the lack of deep level accessibility/automation is still a big hurdle when it comes to Wayland.
There is mouseless that seems to work on wayland
https://mouseless.click/
That is an option, but it's not free.
I wonder if blender works on wayland, because it has a feature that makes the mouse cursor "loop" across the window if you are dragging something.
This is really nice, great job! I built something similar for myself on MacOS recently. Started off as a window manager then began adding keyboard navigation features as I was unhappy with other tools (vimium, shortcat etc).
Hint mode as a feature being able to click all active links on the screen is great and very useful.
Two features I've added recently that have made never using a mouse much more feasible:
1. Search mode - Search for any text on the page, jump to it and then use vim keybindings to select text from there and copy => I use this 20+ times a day
2. Scroll mode - Any scroll area on the page is highlighted with a hint, but when selected it clicks the scroll area and allows you to use keybindings like j/k/gg/G to scroll up and down. I loved this from vimium and wanted to everywhere
Feel free to steal those :)
https://github.com/andyhmltn/nflow
Nice, this is like Vimium link hints but for everything. So long carpal tunnel!
This looks cool! How is the planned Linux support (the website says "Coming soon for Linux") suppose to work? X11 only? Wayland? Gnome only (or any other DE)?
Nice work, but how is this different from Shortcat?
Windows only.
Github has no source code.
Pass for me.
QMK-powered keyboards can do this out of the box
thanks for the heads up, I have a keychron gathering dust at home, so I'll test this later on. favved the thread anyway, since I'm interested, but will wait for the macOS version.
There is a MacOS app called ShortCat that gives similar functionality.
I mean, it can make mouse movements and clicks but you can't be suggesting it can do on-screen hints, which makes it a way better experience.
I'll admit I didn't look too closely at TFA :) But yeah, that part looks neat