I was a bit nervous about this trend when it started picking up because I care about Pyodide (Python running via WebAssembly) and libraries that use PyO3 might not work in Pyodide.
Thankfully that's now been mostly solved - you can compile and publish WASM builds of Rust or C extensions on PyPI now and a Pyodide can then use them.
I'm kind of surprised to hear that was a concern because my perception is that PyO3 is more analogous to wrapping native C code in Python libraries (which is pretty well established since long before either Rust or WASM), and Pyodide seems more like something for use in browsers rather on the server-side. When I looked into this earlier this year, trying to run Python via WASM on the server side with Pyodide had a bunch of hoops I needed to jump through, and it ended up being a lot simpler to use a build of Python from one of the core Python contributors that directly targeted WASM.
This is a nice party trick! I use it to provide easy installation of software that is written in rust, but is primarily used by Python users. (e.g.: Biology tools) They can do `pip install <name>`, since some people prefer this over downloading executables/installers. Bonus: Maturin/PyO3 can build "manylinux" binaries automatically, which helps with the Linux ABI diaspora.
But does it work everywhere Python works? My main issue with this Rust move has always been compatibility. Python can be embedded and ran in a heck of a lot of places. What's the story when libraries that I may want to depend on are actually implemented in Rust and my target doesn't/can't handle the toolchain and there's no build target?
It won't work in those cases. Do you have explicit examples? I suspect this won't come up much as Rust has, IME, much broader target support. (Including platforms without a GPOS)
I can't come up with any cases where cpython would compile that rustc can't target. There are some more niche interpreters that can run on embedded stuff with no rustc support, but they are also subsets of python so it isn't like you can just pull a wheel file and expect it to work
They’re not running Rust they’re running machine code compiled with the Rust compiler. Like many other Python libraries such as numpy are running machine code compiled with C++, C or Fortran. That was always the case for Python and is its main selling point, it’s a slow but easy to write glue language that can make faster native code scriptable.
Also the article reeks of AI slop, it’s just trying to sell you a Rust course.
One main difference is that pyo3 is safer and easier to use to build said modules and have them work correctly without issues. Thus modules which hadn’t been worth it before to build in this manner suddenly become so
There’s a whole legacy of wrapper interface generators like SWIG that are way more powerful than PyO3, QT is e.g. fully mapped to Python using a very powerful wrapper generator (SIP). PyO3 is just a really simple interface generator in comparison, look at what SWIG can do, or SIP, what PyO3 does is trivial in comparison.
It’s time to start figuring out how to block domains on hacker news. This slop is getting out of control. Who can read this? The first few sentences already induced a migraine.
I was a bit nervous about this trend when it started picking up because I care about Pyodide (Python running via WebAssembly) and libraries that use PyO3 might not work in Pyodide.
Thankfully that's now been mostly solved - you can compile and publish WASM builds of Rust or C extensions on PyPI now and a Pyodide can then use them.
Here's the WASM build of the Rust-including Pydantic-core package for example: https://pypi.org/project/pydantic_core/#pydantic_core-2.49.0...
I'm kind of surprised to hear that was a concern because my perception is that PyO3 is more analogous to wrapping native C code in Python libraries (which is pretty well established since long before either Rust or WASM), and Pyodide seems more like something for use in browsers rather on the server-side. When I looked into this earlier this year, trying to run Python via WASM on the server side with Pyodide had a bunch of hoops I needed to jump through, and it ended up being a lot simpler to use a build of Python from one of the core Python contributors that directly targeted WASM.
Yes, Pyodide is almost entirely used in web browsers. I want to be able to run existing code that uses libraries like Pyodide-core in a browser.
How much of the existing PyPI ecosystem do you think could realistically work this way without package authors doing anything specifically for WASM?
This is a nice party trick! I use it to provide easy installation of software that is written in rust, but is primarily used by Python users. (e.g.: Biology tools) They can do `pip install <name>`, since some people prefer this over downloading executables/installers. Bonus: Maturin/PyO3 can build "manylinux" binaries automatically, which helps with the Linux ABI diaspora.
`pip install zig` works today.
> reach for, a Rust extension does the work
ai; dr, sorry
But does it work everywhere Python works? My main issue with this Rust move has always been compatibility. Python can be embedded and ran in a heck of a lot of places. What's the story when libraries that I may want to depend on are actually implemented in Rust and my target doesn't/can't handle the toolchain and there's no build target?
It won't work in those cases. Do you have explicit examples? I suspect this won't come up much as Rust has, IME, much broader target support. (Including platforms without a GPOS)
I can't come up with any cases where cpython would compile that rustc can't target. There are some more niche interpreters that can run on embedded stuff with no rustc support, but they are also subsets of python so it isn't like you can just pull a wheel file and expect it to work
For what it’s worth the GCC backend for rustc is making progress and rustc more generally supports a lot of embedded stuff already
How’s that GCC backend coming along? I could look it up, but I bet others would like to hear about it too.
They’re not running Rust they’re running machine code compiled with the Rust compiler. Like many other Python libraries such as numpy are running machine code compiled with C++, C or Fortran. That was always the case for Python and is its main selling point, it’s a slow but easy to write glue language that can make faster native code scriptable.
Also the article reeks of AI slop, it’s just trying to sell you a Rust course.
One main difference is that pyo3 is safer and easier to use to build said modules and have them work correctly without issues. Thus modules which hadn’t been worth it before to build in this manner suddenly become so
There’s a whole legacy of wrapper interface generators like SWIG that are way more powerful than PyO3, QT is e.g. fully mapped to Python using a very powerful wrapper generator (SIP). PyO3 is just a really simple interface generator in comparison, look at what SWIG can do, or SIP, what PyO3 does is trivial in comparison.
It’s time to start figuring out how to block domains on hacker news. This slop is getting out of control. Who can read this? The first few sentences already induced a migraine.
Why not just run C inside of Python?
For the same reasons someone might prefer Rust to C in general; safety, ergonomics, etc. If you want to write C in Python, nothing is stopping you.
You can do that too. I generally prefer rust as it's a nice lang to work with.
yeah, Rust is eating C/C++...
Because you want to use other existing libraries that are written in Rust.