Fun fact: I’m working on turning ESPectre into a Wi‑Fi Theremin (the musical instrument you play by moving your hands near an antenna).
The idea of “playing” by simply moving around a room sounds a bit ridiculous… but also kind of fun.
The key is the Moving Variance of the spatial turbulence: this value is continuous and stable, making it perfect for mapping directly to pitch/frequency, just like the original Theremin. Other features can be mapped to volume and timbre.
It’s pure signal processing, running entirely on the ESP32. Has anyone here experimented with audio synthesis or sonification using real-time signal processing?
Ive worked on some sonification projects that used signals from xbox kinect lidar, piezos, and other sensors. Co-author on paper i wrote developed a "strummable" theremin that divided physical space with invisible "strings" of various tunings. We preferred running synthesis on PC when possible and just outputting midi and OSC, as DSP on ESP32 has limits for what can be achieved in under 5-10ms. If the goal is hardware audio output, you may need to look into dedicated DSP chips and an audio shield for better DAC- but for prototyping can easily bang a square wave through any of esp32 pins
Thanks for the insights Quinnjh! Would love to hear more about your invisible strings tuning system!
The ESP32-S3 extracts a moving variance signal from spatial turbulence (updates at 20-50 Hz), and I want to map this directly to audio frequency using a passive buzzer + PWM (square wave, 200-2000 Hz range).
Two quick questions:
1. Do you see any pitfalls with updating PWM frequency at 20-50 Hz for responsive theremin-like behavior?
2. Any recommendations on mapping strategies - linear, logarithmic (musical scale), or quantized to specific notes?
As for the tuning system, we didnt get great demo recordings of it, but the invisible strings were linearly mapped as a range onto degrees of a given scale. In our use-case (letting people with disabilities jam without too much dissonance) that key+scale and the master tempo were broadcast to each instrument.
Would have been interesting to play more with custom tunings, but the users we were designing for would have had a harder time using it consonantly. FWIW fully-abled folks like myself sound pretty bad on the theremin, and seeing people play them in orchestras etc displays an impressive level of "virtuosity" to place the hands properly. Quantizing the range of possible positions helps but the tradeoff is sacrificing expressivity.
As for 1) yes, there will definitely be some pitfalls with the relatively slow updates - which may show up as "zipper noise" artifacts in output.
For 2), logarithmic mapping between position and pitch is traditionally theremin-like, but as the theremin avoids zippering by being analog, youll have to get creative with some smoothing/lerping and potentially further quantization. Thats the fun and creative bit though!
Would love to hear about your project again and what approaches you take, and happy to answer other q's so feel free to drop me a line.
Great project and great idea, thank you for sharing !
I don't know if it's useful but one technique I have used in sonification during the experimentation phase is to skip the real time aspect, capture all the available "channels" and generate all the possible permutations of what is mapped where.
Then you can listen to the outputs, see what sounds good, and then test it in real time to check if the musicality is actually a result of the physical interaction and not an artifact or a product of noise.
My first step is to 'listen' to the raw channels and features to quickly find which mapping produces the most musically coherent (i.e., clean and physically predictable) output.
If it sounds like white noise, the mapping is bad or the signal is artifact.
If it sounds like a sine wave moving predictably, the physics are sound.
I wonder if somebody could make an open hardware version of the Leap Motion with this technique (though I'm not sure how accurate/repeatable the sensing is - Leap Motion could detect with an accuracy of < 0.7mm)
That's a great thought. The challenge is that Wi-Fi sensing (CSI) is measuring multipath changes across a few meters, not direct motion in a small volume like Leap Motion. I think its accuracy is measured in centimeters, not sub-millimeters.
You hit the nail on the head! That's precisely the motivation.
Having two kids myself, I've thought of turning it into a game: blindfolded hide-and-seek where the pitch of the Wi-Fi Theremin tells the seeker how close they are to the 'signal disruption' of the other person. It's essentially a real-time sonar game!
Been working on this very idea casually for couple years with ESP-IDF and I could never get the statistical signal processing just right (by my definition). Things I've tried: adapting filtering (LMS, Kalman), kernel methods (NEWMA, MMD), detectors (CUSUM, GLR), dimensionality reduction (random projection, online PCA), whitening, etc.
I use a single ESP32 in STA/AP mode which sniffs ACK packets with a specific destination mac, which come from any server on my WiFi network (uses a special sniffing mode IIRC). This way I can receive regular CSI packets originating from a fixed location and doesn't need another device running.
I'll have to look at this code, maybe I just overlooked the obvious or my requirements were too high!
ESPectre takes a different architectural approach that might address some of the challenges you encountered:
1. Instead of STA/AP mode on a single ESP32, ESPectre uses the natural traffic between your existing router and an ESP32-S3 in station mode. To ensure a stable, continuous CSI packet rate, I implemented a traffic generator that sends ICMP pings to the gateway at a configurable rate (default: 20 pps). This provides bidirectional traffic (request + reply) that reliably triggers CSI generation, giving you predictable packet timing without relying on ambient network traffic or special sniffing modes.
2. Rather than applying filters directly to raw CSI, ESPectre uses Moving Variance Segmentation (MVS) on unfiltered spatial turbulence (std dev of subcarrier amplitudes).
3. The filters are applied to features, not to the segmentation signal itself. This preserves motion sensitivity while cleaning up the feature data
I found that having a stable transmitter (the router) combined with controlled traffic generation provides more consistent multipath patterns and predictable CSI timing, which makes the segmentation more reliable.
Actually I misspoke. I previously used STA/AP mode (and two ESP32s) but I switched to something close to what you describe. I filter the pings to only get the ones targeting a specific MAC (in promiscuous mode). This way I get only specific CSI packets and they're perfectly periodic at whatever rate I want.
Sounds like your MVS approach is a sliding window variance of the cross channel variance, with some adaptive thresholding. My pre-processing has generally been an EWMA de-meaning filter followed by some type of dimensionality reduction and feature extraction (kernel or hand-crafted, like raw moments), which I think fits into your overall architecture.
I'll have to look more closely at your work, thanks for sharing!
Interesting note, I actually disabled promiscuous mode after some testing because it made the CSI signal noisier and consumed more resources.
I found that normal station mode with pings to gateway gave me cleaner, more predictable CSI data. But your MAC filtering approach might mitigate those issues!
You're spot on about the MVS approach. It's essentially a sliding window variance of the spatial turbulence (std dev across subcarriers), with adaptive thresholding based on the moving variance of that signal.
If you're interested in the MVS details, I wrote a free Medium article that walks through the segmentation algorithm step-by-step with visualizations. Links are in the README.
Your approach is actually quite similar to what I'm doing, just in a different order:
- My flow: Raw CSI → Segmentation (MVS) → Filters (Butterworth/Wavelet/Hampel/SG) → Feature extraction
- Your flow: Raw CSI → EWMA de-meaning → Dimensionality reduction → Feature extraction
The main difference is that I segment first to separate IDLE from MOTION states (keeping segmentation on raw, unfiltered CSI to preserve motion sensitivity), then only extract features during MOTION (to save CPU cycles).
Thanks for the thoughtful feedback! Always great to exchange notes with someone who's been in the trenches with CSI signal processing
The final statistic I threshold is the Moving Variance of Spatial Turbulence.
The decision is a binary comparison: When moving_variance > threshold then MOTION state (movement detected) else IDLE state.
The features are extracted only during MOTION segments (to save CPU cycles) and published via MQTT.
They serve as rich foundation data for potential external ML models (e.g., to capture nuances like gestures, running, or falling), but they are absolutely not used for the core segmentation decision itself.
You need one sensor for each area you want to monitor independently.
With devices more capable than the ESP32‑S3, the coverage would likely be greater.
It really depends on the environment: wall materials, interference, signal strength, and even temperature and humidity all play a role. Honestly, 100 m² is probably too large for a single sensor to cover reliably.
That said, ESP32 boards are very inexpensive, you can find them online for around €1 or even less.
Is the ESP-S3 your recommendation? Or should one look into the ESP-C6 / another model? The prices I'm seeing here in Sweden are more around €10 rather than €1, but I guess that's still much less than the average presence detector.
The price difference can vary wildly by region, but €10 is still super cheap compared to a dedicated presence sensor!
Regarding the chip recommendation, it’s a classic trade-off:
ESP32-S3: This is what I mainly recommend today. It gives you more raw processing power (dual-core), which is great for managing the OS, MQTT, and the CSI analysis without hiccups.
ESP32-C6: I just ordered a C6 myself to run extensive tests! The C6 might have a superior quality Wi-Fi module and better internal CSI extraction capabilities due to hardware improvements.
Tommysense creates a sensing mesh between devices, while ESPectre uses your existing Wi-Fi router as the transmitter. As a result, ESPectre needs only one device per area but requires a compatible router with solid 2.4 GHz coverage. The overall goal is similar, but ESPectre is open-source!
It's neat that Tommysense works on top of esphome... I'm currently using Bermuda BLE trilateration, but it doesn't quite work, especially in a multy-story living space (e.g. a townhouse). So I already have a bunch of esphome Bluetooth proxies all over the building.
But no source and "lifetime license if you join our discord" is kinda not my jam.
Founder of TOMMY here. I'm glad you like the ESPHome support. It was one of the most requested features before implementation.
Regarding the lifetime license for Discord members, that's primarily to ensure that beta testers aren't being "used" for testing and then asked to pay. A lot of my users had stories about that with previous companies, and I wanted to give a promise that wasn't going to be the case here. And building a community where people help each other with device placement, hardware suggestions, etc. is a nice addition.
Anyway, I think this project is really cool, francescopace. Many have asked for TOMMY to be open-sourced, so that's definitely something you're going to have success with. I wish you all the best!
This jogged some memories! Was working on something that used radio waves to detect objects and humans ~5 years ago, I see that we've come a long way since then.
One of our goals(abandoned) was to also extend to wifi routers, so I am excited to see continued interest in this space!
I see you have done a fair amount of work to document calibration for various scenarios. Have you tried to calibrate to ignore cats? Can you prioritize different algorithms to focus on size rather than speed of movement?
Also, I use an ebay purchased ruckus router designed for commercial settings. Will the stronger signal and beam forming from the router provide better or worse performance, or is that mainly down to the esp32?
Currently, ESPectre performs only binary motion detection (IDLE/MOTION) based on simple statistical thresholding.
It cannot ignore cats or prioritize size over speed directly on the device, but ESPectre's architecture is designed to enable this kind of advanced classification externally.
It collects a rich set of pre-processed features (spatial turbulence, entropy, etc.) and transmits them via MQTT.
Any external server (like a Home Assistant add-on or a dedicated Python script) can use these features as the input for a trained ML model to perform classification (e.g., Cat vs. Human vs. Fall detection vs. Gesture detection).
Regardin Ruckus Router / Beamforming: for CSI sensing, stability is generally more important than raw power. I recommend starting by disabling beamforming or reducing the power output if you experience poor motion sensitivity, as the stability of the ESP32 receiver is often the bottleneck.
Using two ESP32-S3 modules you can get ~6000 packets per second with CSI data. I'm using this as a cheap replacement for specialty high-G gyroscope modules, but it could see use for this type of motion detection as well.
Sure, the ESP32 will connect to whichever mesh node provides the best 2.4 GHz signal
- It monitors CSI from that specific node (the one it's associated with)
- If the ESP32 roams to a different mesh node, it will start monitoring CSI from the new node
The system doesn't care about the router's internal mesh topology, it just needs a stable connection to receive CSI data from the associated access point.
In terms of layout of rooms and useful monitoring, you have to be able to configure which node it attaches to, right? Because it's going to monitor the physical space between itself and that node.
So you might have an ESP32 placed across the room from one mesh node to monitor that particular room. But if that ESP32 roams to, say, the mesh node on the floor above it, it's going to monitoring a much less useful space - just the vertical space between itself and the mesh node on the floor above.
Am I envisioning this correctly? I'm thinking its a problem for systems like eero, where you can't lock a device to a particular mesh node.
On the critical topic of Mesh Routers and Roaming, a possible solution is to force the ESP32 to hook onto the MAC address of a single Access Point, as discussed here:
The project’s open-source nature acts as an ethical safeguard, and I am explicitly not pursuing any identity recognition features, just movement detection.
But you are absolutely right that, in theory, misuse of this technology could reveal certain behavioral patterns that might lead to identification.
However, it can also be extremely useful for safety purposes, for example, detecting people during a house fire or an earthquake.
This particular project isn't what terrifies me. It's the technology itself which I'm certain you won't be the last to develop and quite likely not even the first. There are plenty of state actors that likely already have their hands on a technology like this or are working on it
Fun fact: I’m working on turning ESPectre into a Wi‑Fi Theremin (the musical instrument you play by moving your hands near an antenna).
The idea of “playing” by simply moving around a room sounds a bit ridiculous… but also kind of fun.
The key is the Moving Variance of the spatial turbulence: this value is continuous and stable, making it perfect for mapping directly to pitch/frequency, just like the original Theremin. Other features can be mapped to volume and timbre.
It’s pure signal processing, running entirely on the ESP32. Has anyone here experimented with audio synthesis or sonification using real-time signal processing?
Ive worked on some sonification projects that used signals from xbox kinect lidar, piezos, and other sensors. Co-author on paper i wrote developed a "strummable" theremin that divided physical space with invisible "strings" of various tunings. We preferred running synthesis on PC when possible and just outputting midi and OSC, as DSP on ESP32 has limits for what can be achieved in under 5-10ms. If the goal is hardware audio output, you may need to look into dedicated DSP chips and an audio shield for better DAC- but for prototyping can easily bang a square wave through any of esp32 pins
Thanks for the insights Quinnjh! Would love to hear more about your invisible strings tuning system!
The ESP32-S3 extracts a moving variance signal from spatial turbulence (updates at 20-50 Hz), and I want to map this directly to audio frequency using a passive buzzer + PWM (square wave, 200-2000 Hz range).
Two quick questions:
1. Do you see any pitfalls with updating PWM frequency at 20-50 Hz for responsive theremin-like behavior?
2. Any recommendations on mapping strategies - linear, logarithmic (musical scale), or quantized to specific notes?
you may be interested in some tech details on that project's prototypes here: https://www.quinnjh.net/projects/adaptive-instruments-projec...
As for the tuning system, we didnt get great demo recordings of it, but the invisible strings were linearly mapped as a range onto degrees of a given scale. In our use-case (letting people with disabilities jam without too much dissonance) that key+scale and the master tempo were broadcast to each instrument.
Would have been interesting to play more with custom tunings, but the users we were designing for would have had a harder time using it consonantly. FWIW fully-abled folks like myself sound pretty bad on the theremin, and seeing people play them in orchestras etc displays an impressive level of "virtuosity" to place the hands properly. Quantizing the range of possible positions helps but the tradeoff is sacrificing expressivity.
As for 1) yes, there will definitely be some pitfalls with the relatively slow updates - which may show up as "zipper noise" artifacts in output.
For 2), logarithmic mapping between position and pitch is traditionally theremin-like, but as the theremin avoids zippering by being analog, youll have to get creative with some smoothing/lerping and potentially further quantization. Thats the fun and creative bit though!
Would love to hear about your project again and what approaches you take, and happy to answer other q's so feel free to drop me a line.
Great project and great idea, thank you for sharing !
I don't know if it's useful but one technique I have used in sonification during the experimentation phase is to skip the real time aspect, capture all the available "channels" and generate all the possible permutations of what is mapped where.
Then you can listen to the outputs, see what sounds good, and then test it in real time to check if the musicality is actually a result of the physical interaction and not an artifact or a product of noise.
Thank you 4goturnamesagain.
My first step is to 'listen' to the raw channels and features to quickly find which mapping produces the most musically coherent (i.e., clean and physically predictable) output.
If it sounds like white noise, the mapping is bad or the signal is artifact.
If it sounds like a sine wave moving predictably, the physics are sound.
By "GPLv3" do you mean "GPL-3.0-or-later"? https://www.gnu.org/licenses/gpl-faq.html#VersionThreeOrLate...
I wonder if somebody could make an open hardware version of the Leap Motion with this technique (though I'm not sure how accurate/repeatable the sensing is - Leap Motion could detect with an accuracy of < 0.7mm)
That's a great thought. The challenge is that Wi-Fi sensing (CSI) is measuring multipath changes across a few meters, not direct motion in a small volume like Leap Motion. I think its accuracy is measured in centimeters, not sub-millimeters.
I'm sure the kids will love this! Wi-Fi Theremin sounds great.
You hit the nail on the head! That's precisely the motivation.
Having two kids myself, I've thought of turning it into a game: blindfolded hide-and-seek where the pitch of the Wi-Fi Theremin tells the seeker how close they are to the 'signal disruption' of the other person. It's essentially a real-time sonar game!
Exactly! That is kind of fun that I had in mind.
Been working on this very idea casually for couple years with ESP-IDF and I could never get the statistical signal processing just right (by my definition). Things I've tried: adapting filtering (LMS, Kalman), kernel methods (NEWMA, MMD), detectors (CUSUM, GLR), dimensionality reduction (random projection, online PCA), whitening, etc.
I use a single ESP32 in STA/AP mode which sniffs ACK packets with a specific destination mac, which come from any server on my WiFi network (uses a special sniffing mode IIRC). This way I can receive regular CSI packets originating from a fixed location and doesn't need another device running.
I'll have to look at this code, maybe I just overlooked the obvious or my requirements were too high!
ESPectre takes a different architectural approach that might address some of the challenges you encountered:
1. Instead of STA/AP mode on a single ESP32, ESPectre uses the natural traffic between your existing router and an ESP32-S3 in station mode. To ensure a stable, continuous CSI packet rate, I implemented a traffic generator that sends ICMP pings to the gateway at a configurable rate (default: 20 pps). This provides bidirectional traffic (request + reply) that reliably triggers CSI generation, giving you predictable packet timing without relying on ambient network traffic or special sniffing modes.
2. Rather than applying filters directly to raw CSI, ESPectre uses Moving Variance Segmentation (MVS) on unfiltered spatial turbulence (std dev of subcarrier amplitudes).
3. The filters are applied to features, not to the segmentation signal itself. This preserves motion sensitivity while cleaning up the feature data
I found that having a stable transmitter (the router) combined with controlled traffic generation provides more consistent multipath patterns and predictable CSI timing, which makes the segmentation more reliable.
Actually I misspoke. I previously used STA/AP mode (and two ESP32s) but I switched to something close to what you describe. I filter the pings to only get the ones targeting a specific MAC (in promiscuous mode). This way I get only specific CSI packets and they're perfectly periodic at whatever rate I want.
Sounds like your MVS approach is a sliding window variance of the cross channel variance, with some adaptive thresholding. My pre-processing has generally been an EWMA de-meaning filter followed by some type of dimensionality reduction and feature extraction (kernel or hand-crafted, like raw moments), which I think fits into your overall architecture.
I'll have to look more closely at your work, thanks for sharing!
Interesting note, I actually disabled promiscuous mode after some testing because it made the CSI signal noisier and consumed more resources. I found that normal station mode with pings to gateway gave me cleaner, more predictable CSI data. But your MAC filtering approach might mitigate those issues!
You're spot on about the MVS approach. It's essentially a sliding window variance of the spatial turbulence (std dev across subcarriers), with adaptive thresholding based on the moving variance of that signal.
If you're interested in the MVS details, I wrote a free Medium article that walks through the segmentation algorithm step-by-step with visualizations. Links are in the README.
Your approach is actually quite similar to what I'm doing, just in a different order:
- My flow: Raw CSI → Segmentation (MVS) → Filters (Butterworth/Wavelet/Hampel/SG) → Feature extraction
- Your flow: Raw CSI → EWMA de-meaning → Dimensionality reduction → Feature extraction
The main difference is that I segment first to separate IDLE from MOTION states (keeping segmentation on raw, unfiltered CSI to preserve motion sensitivity), then only extract features during MOTION (to save CPU cycles).
Thanks for the thoughtful feedback! Always great to exchange notes with someone who's been in the trenches with CSI signal processing
I noticed your feature vector is large and you don't use ML. What's the final statistic that you threshold?
The final statistic I threshold is the Moving Variance of Spatial Turbulence.
The decision is a binary comparison: When moving_variance > threshold then MOTION state (movement detected) else IDLE state.
The features are extracted only during MOTION segments (to save CPU cycles) and published via MQTT.
They serve as rich foundation data for potential external ML models (e.g., to capture nuances like gestures, running, or falling), but they are absolutely not used for the core segmentation decision itself.
> The system does NOT use Machine Learning, it relies purely on Math.
You may be surprised to find out how machine learning works!
That's a fair point, and as a math graduate, I absolutely agree that ML is fundamentally applied math.
When I say 'No ML,' I mean there is no training phase, no labeled data needed, and no neural network model used to infer the rules.
The distinction here is that all the logic is based purely on signal processing algorithms.
Thanks for raising the point!
Amazing stuff!
Am I right in understanding that only a single ESP32 device is needed (plus a router)?
Probably one per room because all the examples at https://github.com/francescopace/espectre/blob/main/CALIBRAT... are about a single room.
Is the author reads this, how does the system cope with multiple rooms in the same house, maybe a two or three storeys house?
Yes, you’re both correct:
You need one sensor for each area you want to monitor independently. With devices more capable than the ESP32‑S3, the coverage would likely be greater.
The ESP32‑C6, in particular, offers significantly better performance. Check out this comparison video from Espressif: https://www.youtube.com/watch?v=JjdpzM6zVJ8
How about if I want to monitor the apartment as a whole? Would 100m2 be too much for one sensor to handle?
It really depends on the environment: wall materials, interference, signal strength, and even temperature and humidity all play a role. Honestly, 100 m² is probably too large for a single sensor to cover reliably.
That said, ESP32 boards are very inexpensive, you can find them online for around €1 or even less.
Is the ESP-S3 your recommendation? Or should one look into the ESP-C6 / another model? The prices I'm seeing here in Sweden are more around €10 rather than €1, but I guess that's still much less than the average presence detector.
The price difference can vary wildly by region, but €10 is still super cheap compared to a dedicated presence sensor!
Regarding the chip recommendation, it’s a classic trade-off:
ESP32-S3: This is what I mainly recommend today. It gives you more raw processing power (dual-core), which is great for managing the OS, MQTT, and the CSI analysis without hiccups.
ESP32-C6: I just ordered a C6 myself to run extensive tests! The C6 might have a superior quality Wi-Fi module and better internal CSI extraction capabilities due to hardware improvements.
Would you recommend multiple routers for this case, since the distance from sensor to router is important?
Interesting! Are you familiar with tommysense.com? I think it doing something similar? Did not yet have time to try it.
Tommysense creates a sensing mesh between devices, while ESPectre uses your existing Wi-Fi router as the transmitter. As a result, ESPectre needs only one device per area but requires a compatible router with solid 2.4 GHz coverage. The overall goal is similar, but ESPectre is open-source!
It's neat that Tommysense works on top of esphome... I'm currently using Bermuda BLE trilateration, but it doesn't quite work, especially in a multy-story living space (e.g. a townhouse). So I already have a bunch of esphome Bluetooth proxies all over the building.
But no source and "lifetime license if you join our discord" is kinda not my jam.
Founder of TOMMY here. I'm glad you like the ESPHome support. It was one of the most requested features before implementation.
Regarding the lifetime license for Discord members, that's primarily to ensure that beta testers aren't being "used" for testing and then asked to pay. A lot of my users had stories about that with previous companies, and I wanted to give a promise that wasn't going to be the case here. And building a community where people help each other with device placement, hardware suggestions, etc. is a nice addition.
Anyway, I think this project is really cool, francescopace. Many have asked for TOMMY to be open-sourced, so that's definitely something you're going to have success with. I wish you all the best!
- Mike
Mike, thank you so much! Coming from the founder of TOMMY, that means a lot.
I completely respect the way you've managed the beta community and licensing; it’s a smart way to reward early supporters and foster user engagement.
I wish you and the TOMMY project continued success as well!
This jogged some memories! Was working on something that used radio waves to detect objects and humans ~5 years ago, I see that we've come a long way since then.
One of our goals(abandoned) was to also extend to wifi routers, so I am excited to see continued interest in this space!
https://www.sensorsportal.com/HTML/ST_JOURNAL/PDF_Files/P_32...
Thanks for sharing the paper; I'll certainly be taking a look at that research!
I see you have done a fair amount of work to document calibration for various scenarios. Have you tried to calibrate to ignore cats? Can you prioritize different algorithms to focus on size rather than speed of movement?
Also, I use an ebay purchased ruckus router designed for commercial settings. Will the stronger signal and beam forming from the router provide better or worse performance, or is that mainly down to the esp32?
Currently, ESPectre performs only binary motion detection (IDLE/MOTION) based on simple statistical thresholding.
It cannot ignore cats or prioritize size over speed directly on the device, but ESPectre's architecture is designed to enable this kind of advanced classification externally.
It collects a rich set of pre-processed features (spatial turbulence, entropy, etc.) and transmits them via MQTT.
Any external server (like a Home Assistant add-on or a dedicated Python script) can use these features as the input for a trained ML model to perform classification (e.g., Cat vs. Human vs. Fall detection vs. Gesture detection).
Regardin Ruckus Router / Beamforming: for CSI sensing, stability is generally more important than raw power. I recommend starting by disabling beamforming or reducing the power output if you experience poor motion sensitivity, as the stability of the ESP32 receiver is often the bottleneck.
Thank you, it looks fascinating. Putting it in my hobby project queue at position 1, right after my current one.
Using two ESP32-S3 modules you can get ~6000 packets per second with CSI data. I'm using this as a cheap replacement for specialty high-G gyroscope modules, but it could see use for this type of motion detection as well.
Are you using two separate S3s as a dedicated Transmitter/Receiver pair, or are both transmitting data simultaneously?
Transmitter/Reciever pair.
Would this work with a mesh router?
Sure, the ESP32 will connect to whichever mesh node provides the best 2.4 GHz signal
- It monitors CSI from that specific node (the one it's associated with)
- If the ESP32 roams to a different mesh node, it will start monitoring CSI from the new node
The system doesn't care about the router's internal mesh topology, it just needs a stable connection to receive CSI data from the associated access point.
In terms of layout of rooms and useful monitoring, you have to be able to configure which node it attaches to, right? Because it's going to monitor the physical space between itself and that node.
So you might have an ESP32 placed across the room from one mesh node to monitor that particular room. But if that ESP32 roams to, say, the mesh node on the floor above it, it's going to monitoring a much less useful space - just the vertical space between itself and the mesh node on the floor above.
Am I envisioning this correctly? I'm thinking its a problem for systems like eero, where you can't lock a device to a particular mesh node.
On the critical topic of Mesh Routers and Roaming, a possible solution is to force the ESP32 to hook onto the MAC address of a single Access Point, as discussed here:
https://github.com/francescopace/espectre/discussions/6
The surveillance implications for this technology are fascinating and frightening.
The project’s open-source nature acts as an ethical safeguard, and I am explicitly not pursuing any identity recognition features, just movement detection.
But you are absolutely right that, in theory, misuse of this technology could reveal certain behavioral patterns that might lead to identification.
However, it can also be extremely useful for safety purposes, for example, detecting people during a house fire or an earthquake.
This particular project isn't what terrifies me. It's the technology itself which I'm certain you won't be the last to develop and quite likely not even the first. There are plenty of state actors that likely already have their hands on a technology like this or are working on it
my brain expected the part after `-` to be new speculative execution bugs
Hahaha, I love your brain!