It'd be great if it supports a wasm/web backend as well.
I bet a lot of trivial text capabilities (grammar checking, autocomplete, etc) will benefit from this rather than sending everything to a hosted model.
It's possible right now with onnx / transformers.js / tensorflow.js - but isn't quite there yet in terms of efficiency. Given the target for microcontrollers, it'd be great to bring that efficiency to browsers as well.
I am so confused by metas ecosystem. Perhaps others have the same issues. I have mountains of torchscript code. It worked fine for me - had no issues making the python compatible. Torchscript is now deprecated, and the ostensible replacement is torch.export and either: AOTInductor or executorch. torch.export is so limited - no control flow at runtime at all, less support of python than torchscript. It is far more work to hoist all the control flow out of the model than it ever was to make the model torchscript compatible. Feel like meta has moved on, but I'm still stuck in the past here.
I've heard from a friend who works in the embedded space that Tensorflow Lite is still the only realistic (supported by vendors) game in town for running ML models on microcontrollers such as ESP32, nRF, etc. The hardware support listed for this project seems like it's targeting much "fatter" MCUs (Android, etc).
yeah that checks out, although looks like they do have an example for running models on a raspberry pi pico 2: https://docs.pytorch.org/executorch/main/pico2_tutorial.html. The list of embedded platforms this can run on is probably greater than the list of backends, it just wouldn't have acceleration.
Yeah, it's targeting "micro"-controllers, not microcontrollers. I was hoping for a PyTorch solution to TF Lite.
This is still great, though. Previously, I thought a mobile model (eg speech/object recognition) would require me to learn both PyTorch and something like MLC in C++. Then, port them.
If this is as it appears, I could develop a small model that could run on mobile on my laptop, train it on cloud GPU's, test it locally, and use this tool to produce a mobile version (or save some steps?). That would keep us from having to learn C++ or MLC just to do mobile.
I mean, one still can learn other tools for their advantages. However, ML students and startups might benefit greatly from this by being able to rapidly develop or port mobile apps. Then, people learning other tools for their advantages build stuff that way. The overall ecosystem gets stronger with more competition.
It'd be great if it supports a wasm/web backend as well.
I bet a lot of trivial text capabilities (grammar checking, autocomplete, etc) will benefit from this rather than sending everything to a hosted model. It's possible right now with onnx / transformers.js / tensorflow.js - but isn't quite there yet in terms of efficiency. Given the target for microcontrollers, it'd be great to bring that efficiency to browsers as well.
If you need WASM, I think Candle is your current best bet: https://github.com/huggingface/candle
I am so confused by metas ecosystem. Perhaps others have the same issues. I have mountains of torchscript code. It worked fine for me - had no issues making the python compatible. Torchscript is now deprecated, and the ostensible replacement is torch.export and either: AOTInductor or executorch. torch.export is so limited - no control flow at runtime at all, less support of python than torchscript. It is far more work to hoist all the control flow out of the model than it ever was to make the model torchscript compatible. Feel like meta has moved on, but I'm still stuck in the past here.
it's quite the bummer. some models you simply can't export with dynamo. for the time being the jit exporter is the only good option.
in particular selective function scripting is essential!
I've heard from a friend who works in the embedded space that Tensorflow Lite is still the only realistic (supported by vendors) game in town for running ML models on microcontrollers such as ESP32, nRF, etc. The hardware support listed for this project seems like it's targeting much "fatter" MCUs (Android, etc).
yeah that checks out, although looks like they do have an example for running models on a raspberry pi pico 2: https://docs.pytorch.org/executorch/main/pico2_tutorial.html. The list of embedded platforms this can run on is probably greater than the list of backends, it just wouldn't have acceleration.
Yeah, it's targeting "micro"-controllers, not microcontrollers. I was hoping for a PyTorch solution to TF Lite.
This is still great, though. Previously, I thought a mobile model (eg speech/object recognition) would require me to learn both PyTorch and something like MLC in C++. Then, port them.
If this is as it appears, I could develop a small model that could run on mobile on my laptop, train it on cloud GPU's, test it locally, and use this tool to produce a mobile version (or save some steps?). That would keep us from having to learn C++ or MLC just to do mobile.
I mean, one still can learn other tools for their advantages. However, ML students and startups might benefit greatly from this by being able to rapidly develop or port mobile apps. Then, people learning other tools for their advantages build stuff that way. The overall ecosystem gets stronger with more competition.