This feels like something that’s a neat claim and will work against simple setups, but less accurate for more complicated scenarios (eg Tor). Then you’re really just relying on how accurate your knowledge of the proxies are.
Also, the readme has slightly incorrect logic I think:
> According to Special Relativity, information cannot travel faster than the speed of light. Therefore, if the round trip time (RTT) is 4ms, it's physically impossible for them to be farther than 2 light milliseconds away, which is approximately 600 kilometers.
It calls out the 33% for fiber but ignores that there’s not a straightline path between two points on the network and there could be wireless, cable, and DSL links somewhere on that hop.
Also, the controlled variable here is latency, not distance. Thus you can always increase latency through buffering and therefor you could be made to appear further than you are. And that buffering need not even be intentional - your perceived distance estimate will vary based upon queuing delays in intermediary depending on time of day (itself a fingerprint if you incorporate time-aware measurements, but a source of error if you don’t).
Fingerprinting is hard and I dislike the framing that it’s absolutely impossible to mask or that there’s not false positive and false negative error rates with the fingerprint.
About the straightline path I did think of that but apparently I forgot to address it when writing the README :p
The point I was trying to make is that if the RTT is low enough you can know the connection is being made from close, it's an upper bound, and making some assumptions you can get it lower, so it's not a way of knowing the exact distance but rather the max distance the connection can be made from. If someone is in Spain but they can't be more than 400km from Australia, something went terribly wrong somewhere hehe
In hindsight I think the issue with my explanation is that I was trying to explain the differences when fingerprinting two different protocols, but ended up going for a TCP-only approach since Fastly wouldn't expose to me the data I needed for the TLS and HTTP RTT. But in theory fingerprinting with protocol RTT difference where one protocol is proxied and the other is impossible to bypass, but this is only the theory.
I think I will edit the README in the future since I don't like how it turned out too much. Thanks for the feedback!
The difference in min TCP RTT and min RTT to respond to a websocket payload is a dead giveaway that there's a middlebox terminating TCP somewhere along the path. You can bypass this by sourcing your request within 30ms of wherever TCP is being terminated, anything under that threshold could be caused by regular noise and isn't a reliable fingerprint. Due to how many gateway's there are between you and a residential proxy exit node this makes fingerprinting them extremely easy.
I expect it won't be long until someone deploys the first proxy service that handles the initial CONNECT payload in the kernel before offloading packet forwarding to an eBPF script that will proxy packets between hosts at layer 3, making this fingerprinting technique obsolete. The cat and mouse game continues.
Every TCP proxy (that doesn't thwart this) is detectable :)
Countermeasure: pick some min-RTT >= the actual client RTT (you can do this as a TCP proxy by measuring client ping). Measure server RTT and artificially delay responses to be >= min-RTT. This will require an added delay during the handshake and ACKs, but no added delay for the response payloads.
Counter-countermeasure: the above may lead to TCP message types that don't make sense given a traditional TCP client state machine (e.g., delayed ACK would bundle ACK and PUSH but the system shows separate/simultaneous ACK and PUSH packets. Counter-counter-countermeasure is left to the reader.
I think you could also compare with TLS handshake timings, delay for client hello among other things. And you could also compare it with HTTP RTT, not to mention that you can do TCP fingerprinting and compare it with the TLS and HTTP fingerprint of the browser, you can also measure the IP TTL and ping, among many other things... What I mean is that there are a ton of things that can be done on both sides, but any company with enough people working at this and enough servers will surely make something miles away from my proof of concept, and they also have a lot of traffic to know what's baseline data and what isn't.
Just in case someone tries to use it to make some kind of judgement about the traffic - there's a whole world behind legit or enforced proxies. Especially corporate environments will often tunnel all the traffic for compliance and audit reasons.
When deployed on a popular server, one bit of "IP intelligence" this detector itself can gather is keep database of lowest-seen RTT per given source IP, maybe with some filtering - to cut out "faster-than-light" datapoints, gracefully update when actual network topology changes, etc.
That would establish a baseline, and from there, additional end-to-end RTT should become much more visible.
I imagine any big CDN implementing something like this could keep a database of all of this, combined with the old kind of IP intelligence and collecting not only RTT on other protocols like TLS, HTTP, IP (aka ping, and traceroutes too), TCP fingerprint, TLS fingerprint, HTTP fingerprint...
And with algorithms that combine and compare all these data points, I think very accurate models of the proxy could be made. And for things like credit card fraud this could be quite useful.
Do you mean originating traffic back to the IP where HTTP request came from? That's way too risky for a reputable server/CDN. There is a reason why driftnet, shodan, paloalto, and the likes, only ever originate scans from separately-registered, separately-announced subnets. Those end up in lists like [1] in no time at all.
I like this. I could see this being extra useful for people not using CDN's if they could easily plug it into nginx, haproxy and such. Currently for proxies I look for the proxy headers and also use a list of known proxy IP's but that is obviously nowhere near as complete as what you built. It might also be interesting to test assorted configurations of SSH forwards and MitM TLS caching proxies such as Squid SSL Bump.
I guess for this to work best you'd build your own CDN and have as many servers as possible. I have always dreamed of an Open Source CDN managed by a nonprofit and dedicated to offering CDN services for free or for a reasonable cost.
If you did the timings by comparing to other protocols, like TLS or HTTP you could do this with a single server, but that's a bit more complex than doing it on the same protocol since you have to account for more stuff, but it could be done, at the end of the day, my idea with Aroma was mostly to prove that it's possible, thanks for the feedback btw!
Neat demo. The unsettling part is how little signal you actually need: big CDNs and fraud teams already run much richer timing models than a simple min_rtt / rtt ratio. You can’t spoof away the speed of light, only add latency or jitter, and that itself becomes a fingerprint once you have enough traffic and a few global PoPs to compare from. So this doesn’t magically break L3 VPNs, but anyone relying on “just stick a TCP proxy in front and I’m anonymous/in-region” has been living with a pretty outdated threat model.
Why would one want this? Are there particular situation(s) that it's desirable to detect a TCP proxy? Does presence of a TCP proxy indicate some adverserial behaviour? E.g. surveillance, censorship, a particular attack?
<html><body><h1>You don't seem to be using a TCP Proxy!</h1><p>(If you are using a VPN or any other kind of proxy that is not a TCP Proxy, this will not detect it)</p></body></html>
Also, something I haven't included on the README is that apart from testing with Tor, WARP and some other proxies. I did some testing with the free one-week trial of Brightdata's residential proxies, and it does detect them too!!!
It's done by checking the difference between the initial TCP RTT and the subsequent TCP RTTs, both of which can be retrieved from the Linux Kernel easily without the need for PCAPing. There is more info about how it is done on the README
It's set at a low threshold since I want to avoid blocking regular users at all costs, I think the detection can be improved a lot by using more data and not a single division to calculate the score, in this case it's a somewhat simple PoC.
Thanks for taking the time to test it, I really appreciate it!
It's a super cool tool, I've been wondering about an open source tool doing this since reading about the technique in one of Nikolai Tschacher's blog posts years ago (https://incolumitas.com/pages/about/).
There's a few ways to work around this, but I think it's one of the best signals available to detect low-effort/common proxy providers.
Oh I haven't seen that before, it's really cool, thank you for showing me that!
I want to clarify that the approaches are a bit different, they use IP intelligence too and this approach doesn't use any kind of websockets, which is a really good idea, and I have to admit I didn't think of that, but sadly it's not really possible to do it with Fastly.
Another big difference is that this could work with any TCP application, not only HTTP, and if you do it with HTTP/S you can know if it's a proxy or not on a request basis and totally passively, without adding any delay or changing the code of the app.
This feels like something that’s a neat claim and will work against simple setups, but less accurate for more complicated scenarios (eg Tor). Then you’re really just relying on how accurate your knowledge of the proxies are.
Also, the readme has slightly incorrect logic I think:
> According to Special Relativity, information cannot travel faster than the speed of light. Therefore, if the round trip time (RTT) is 4ms, it's physically impossible for them to be farther than 2 light milliseconds away, which is approximately 600 kilometers.
It calls out the 33% for fiber but ignores that there’s not a straightline path between two points on the network and there could be wireless, cable, and DSL links somewhere on that hop.
Also, the controlled variable here is latency, not distance. Thus you can always increase latency through buffering and therefor you could be made to appear further than you are. And that buffering need not even be intentional - your perceived distance estimate will vary based upon queuing delays in intermediary depending on time of day (itself a fingerprint if you incorporate time-aware measurements, but a source of error if you don’t).
Fingerprinting is hard and I dislike the framing that it’s absolutely impossible to mask or that there’s not false positive and false negative error rates with the fingerprint.
About the straightline path I did think of that but apparently I forgot to address it when writing the README :p
The point I was trying to make is that if the RTT is low enough you can know the connection is being made from close, it's an upper bound, and making some assumptions you can get it lower, so it's not a way of knowing the exact distance but rather the max distance the connection can be made from. If someone is in Spain but they can't be more than 400km from Australia, something went terribly wrong somewhere hehe
In hindsight I think the issue with my explanation is that I was trying to explain the differences when fingerprinting two different protocols, but ended up going for a TCP-only approach since Fastly wouldn't expose to me the data I needed for the TLS and HTTP RTT. But in theory fingerprinting with protocol RTT difference where one protocol is proxied and the other is impossible to bypass, but this is only the theory.
I think I will edit the README in the future since I don't like how it turned out too much. Thanks for the feedback!
By the way, it detects Tor, I tested it ;D
This is the core concept of how proxies are detected via services like https://layer3intel.com/tripwire or https://spur.us/monocle/
The difference in min TCP RTT and min RTT to respond to a websocket payload is a dead giveaway that there's a middlebox terminating TCP somewhere along the path. You can bypass this by sourcing your request within 30ms of wherever TCP is being terminated, anything under that threshold could be caused by regular noise and isn't a reliable fingerprint. Due to how many gateway's there are between you and a residential proxy exit node this makes fingerprinting them extremely easy.
I expect it won't be long until someone deploys the first proxy service that handles the initial CONNECT payload in the kernel before offloading packet forwarding to an eBPF script that will proxy packets between hosts at layer 3, making this fingerprinting technique obsolete. The cat and mouse game continues.
Every TCP proxy (that doesn't thwart this) is detectable :)
Countermeasure: pick some min-RTT >= the actual client RTT (you can do this as a TCP proxy by measuring client ping). Measure server RTT and artificially delay responses to be >= min-RTT. This will require an added delay during the handshake and ACKs, but no added delay for the response payloads.
Counter-countermeasure: the above may lead to TCP message types that don't make sense given a traditional TCP client state machine (e.g., delayed ACK would bundle ACK and PUSH but the system shows separate/simultaneous ACK and PUSH packets. Counter-counter-countermeasure is left to the reader.
I think you could also compare with TLS handshake timings, delay for client hello among other things. And you could also compare it with HTTP RTT, not to mention that you can do TCP fingerprinting and compare it with the TLS and HTTP fingerprint of the browser, you can also measure the IP TTL and ping, among many other things... What I mean is that there are a ton of things that can be done on both sides, but any company with enough people working at this and enough servers will surely make something miles away from my proof of concept, and they also have a lot of traffic to know what's baseline data and what isn't.
It's a complex but fun world we live in hehe
Just in case someone tries to use it to make some kind of judgement about the traffic - there's a whole world behind legit or enforced proxies. Especially corporate environments will often tunnel all the traffic for compliance and audit reasons.
Yes, it's important to keep this in mind, thanks for your comment!
Very clever, I like it.
When deployed on a popular server, one bit of "IP intelligence" this detector itself can gather is keep database of lowest-seen RTT per given source IP, maybe with some filtering - to cut out "faster-than-light" datapoints, gracefully update when actual network topology changes, etc.
That would establish a baseline, and from there, additional end-to-end RTT should become much more visible.
First of all, thanks!
I imagine any big CDN implementing something like this could keep a database of all of this, combined with the old kind of IP intelligence and collecting not only RTT on other protocols like TLS, HTTP, IP (aka ping, and traceroutes too), TCP fingerprint, TLS fingerprint, HTTP fingerprint...
And with algorithms that combine and compare all these data points, I think very accurate models of the proxy could be made. And for things like credit card fraud this could be quite useful.
> collecting RTT on [...] ping
Do you mean originating traffic back to the IP where HTTP request came from? That's way too risky for a reputable server/CDN. There is a reason why driftnet, shodan, paloalto, and the likes, only ever originate scans from separately-registered, separately-announced subnets. Those end up in lists like [1] in no time at all.
[1] https://github.com/UninvitedActivity/UninvitedActivity
I like this. I could see this being extra useful for people not using CDN's if they could easily plug it into nginx, haproxy and such. Currently for proxies I look for the proxy headers and also use a list of known proxy IP's but that is obviously nowhere near as complete as what you built. It might also be interesting to test assorted configurations of SSH forwards and MitM TLS caching proxies such as Squid SSL Bump.
I guess for this to work best you'd build your own CDN and have as many servers as possible. I have always dreamed of an Open Source CDN managed by a nonprofit and dedicated to offering CDN services for free or for a reasonable cost.
If you did the timings by comparing to other protocols, like TLS or HTTP you could do this with a single server, but that's a bit more complex than doing it on the same protocol since you have to account for more stuff, but it could be done, at the end of the day, my idea with Aroma was mostly to prove that it's possible, thanks for the feedback btw!
Neat demo. The unsettling part is how little signal you actually need: big CDNs and fraud teams already run much richer timing models than a simple min_rtt / rtt ratio. You can’t spoof away the speed of light, only add latency or jitter, and that itself becomes a fingerprint once you have enough traffic and a few global PoPs to compare from. So this doesn’t magically break L3 VPNs, but anyone relying on “just stick a TCP proxy in front and I’m anonymous/in-region” has been living with a pretty outdated threat model.
Thank you! There are other ways of detecting L3 VPNs, but I wanted to start with proxies since they do most of the damage.
Why would one want this? Are there particular situation(s) that it's desirable to detect a TCP proxy? Does presence of a TCP proxy indicate some adverserial behaviour? E.g. surveillance, censorship, a particular attack?
curl -x http://xxxxx:xxxxx@geo.iproyal.com:11202 -L https://aroma.global.ssl.fastly.net/
<html><body><h1>You don't seem to be using a TCP Proxy!</h1><p>(If you are using a VPN or any other kind of proxy that is not a TCP Proxy, this will not detect it)</p></body></html>
That's strange, could you try with "https://aroma.global.ssl.fastly.net/score"?
Also, something I haven't included on the README is that apart from testing with Tor, WARP and some other proxies. I did some testing with the free one-week trial of Brightdata's residential proxies, and it does detect them too!!!
so will this detect residential proxies? how is that being done, I am getting hammered and its all legitimate normal ISP traffic.
It's done by checking the difference between the initial TCP RTT and the subsequent TCP RTTs, both of which can be retrieved from the Linux Kernel easily without the need for PCAPing. There is more info about how it is done on the README
To answer your first question, in my tests its around 50% of requests making it through.
Are you using a proxy? If you aren't that would be concerning, since false positives are way worse than false negatives.
If you are then it means the score is sometimes a bit lower and sometimes a bit higher than 0.1, which is the threshold for getting blocked.
If you want to know the exact score, you can check https://aroma.global.ssl.fastly.net/score
It's set at a low threshold since I want to avoid blocking regular users at all costs, I think the detection can be improved a lot by using more data and not a single division to calculate the score, in this case it's a somewhat simple PoC.
Thanks for taking the time to test it, I really appreciate it!
I'm testing using our residential proxies.
It's a super cool tool, I've been wondering about an open source tool doing this since reading about the technique in one of Nikolai Tschacher's blog posts years ago (https://incolumitas.com/pages/about/).
There's a few ways to work around this, but I think it's one of the best signals available to detect low-effort/common proxy providers.
Oh I haven't seen that before, it's really cool, thank you for showing me that!
I want to clarify that the approaches are a bit different, they use IP intelligence too and this approach doesn't use any kind of websockets, which is a really good idea, and I have to admit I didn't think of that, but sadly it's not really possible to do it with Fastly.
Another big difference is that this could work with any TCP application, not only HTTP, and if you do it with HTTP/S you can know if it's a proxy or not on a request basis and totally passively, without adding any delay or changing the code of the app.
But yeah, it's a really cool demo, thanks again!