To everyone building these things: Please add a disclaimer to say something like:
"This is not a vulnerability: eBPF currently requires root access to do this. Also, eBPF makes this easy but does not make it possible, as debuggers, interposers/shims, and other tools can also attach to pre-encryption points, and therefore banning eBPF (as some people want to do after seeing projects like this) would not actually improve security, but it would instead _reduce_ security as it would prevent eBPF-based security solutions from being used."
On an unrelated note, your work has inspired most of my career in Solaris/Illumos/Linux systems and honestly this project likely wouldn't have happened if it wasn't for all of your books/blogs/projects to help me along the way. Thank you!
Does this work for Go binaries? My understanding is that Go programs do all the encryption "in the process" so the data is encrypted before eBPF can intercept it. I'd love to be wrong about that!
We have Go support, but it is not open sourced yet. Go is a bit more complicated but we were able to get it after some cave diving in the ELF formats. To give you a little insight on how this works, because Go is statically linked, we need to pull several different offsets of the functions we are going to hook into.
We do this by scanning every version of Go that is released to find offsets in the standard library that won't change. Then when we detect a new Go process, we use an ELF scanner to find some function offsets and hook into those with uprobes. Using both of these, we have all the information we need to see Go pre-encryption content as well as attribute it to connections and processes.
Great approach. I love the choice of practicality over generalization.
Are these offsets consistent across compilation targets, and they vary only by version of the Go binary? Or do you need to do this scan for every architecture?
The short answer is that we only have to calculate the offset per go version, no expensive runtime scanning is required.
The long answer is that the offsets are the byte alignment offsets for the go structs containing the pointers to the file descriptor and buffers. Fortunately we only have to calculate these for each version where the TLS structs within go actually change, so not even for every version. For instance, if a field is added, removed, or changes type then the location in memory where those pointers will be found changes. We can then calculate the actual offset at runtime where we know which architecture (amd64, arm64, etc) with a simple calculation. Within the eBPF probe, when the function is called, it uses pointer arithmetic to extract the location of the file descriptor and buffer directly.
Is the precomputation sufficiently resilient to cross-compilation where the system may be composed of significantly different versions than something mainstream like Debian might ship? I'm thinking of embedded targets built using the Yocto Project tooling.
I think you only need to use the eBPF approach for statically linked programs.
ISTR, at some point in the far past, using LD_PRELOAD with my own shims to capture TLS traffic before encryption/after decryption. I might have it lying around somewhere here.
Qtap scans binaries of processes as well known locations for OpenSSL on startup, then passes the offsets to eBPF where it hooks into the SSL_read and SSL_write to get the content before or after it's been encrypted.
`ecapture` has been around for a while and do a lot of great stuff and a lot of functionality overlaps.
Our aim is to make Qtap extensible and via a plugin system. We have http1/2 streaming capabilities and a plugin engine to run these in what we call a stack. Our goal is to add more protocols, like gRPC in the near future.
We have a few example plugins that do things like report request/response's and push access information to standard out in a console or structure log format. Our Pro version has a few more plugins like the ability to report errors (eg. an AI agent is getting HTTP 429 errors). These can be pushed to a service or log aggregator.
To summarize, we do a lot of the same things that ecapture does. We'd like to be less of a tool and more of a "always running" that ops, opsec, and devs use to answer tough questions. We look forward to open sourcing more of plugins as they mature!
Have been following this project for a while, cool stuff!
I work a bunch with vpn-like networking on Android phones and it would be cool to have a bit of info on how I might get something like working on phones. I guess its probably not your typical usecase.
Currently since the project is a VPN client, I already intercept all of the packets, I have a pcap writer and can write to files or a tcp sockets and connect wireshark to it - but it needs a bunch of complication to setup the keys so that I can see through encryption, so anything that would make that process easier would be great.
It can do multi-hop. It can also automatically authenticate across any number of phones, sort of like how radius works with fixed wifi (android hotspots don't support radius - you can only connect with radius as a client). You can also earn money from sharing your internet.
I know that arguing that SSLKEYLOGFILE is all you need will just be a different version of the rsync/dropbox comment, but I do wonder under what circumstances is one able to strace a binary and isn’t able to make it dump session keys? I read the headline and set high hopes on finding a nifty way to mitm apps on Android - alas, I’m not sure this would work there necessarily.
Mostly that SSLKEYLOGFILE has only been an (disabled by default) OpenSSL feature for a few weeks (literally), apart from that it's something implemented by some other libraries (notably libcurl) on top. But it's very far from "just set this env var and the keys will pop out of any app using TLS".
The big usecase for me would be if you could attach the trace after starting the binary. The idea of coming into a production system that's behaving unexpectedly and getting a network sniff without having to fiddle with certificates is very attractive.
There's an alternative implementation where SSLKEYLOGFILE is more "dynamic" and permits being toggled on an off during runtime, but that doesn't currently exist.
My big use case is watching on the SERVER side, my coworkers will be asking me to help them debug something and I just want to see the HTTP plaintext, I don't really want to try running Apache under SSLKEYLOGFILE or something, I just want to see the data. ;-)
The main benefit is complete coverage. In production systems there are many different workloads with many different binaries, each with different build processes. Leveraging eBPF enables seeing everything on a system without having to adjust the build pipeline.
To hook into OpenSSL, don't you either need dynamic linking or userspace programs to compile your hooks in? Go and many Rust and C++ binaries tend to prefer static linking, so I wonder if this solution is workable there.
Great point! Yes it supports both scenarios. Qtap scans the binary ELF (curl, rust, etc) and looks for the TLS symbols. If they were statically compiled the eBPF probes will be attached directly to the binary, if dynamically linked the probes will be attached to the symbols in the library (.so).
Fair point on -O2 and -O3 optimized bins. We've approached this by building custom bin utils that are optimized for blazingly fast symbol recognition. Traditional ELF tools that focus on providing comprehensive context for debugging, we are strip away everything that is not the symbol locations we need.
We've also added caching so frequently used bins don't require multiple scans. Shared libraries as well. This has proven effective with optimized binaries, especially bins that are optimized, start, make a super quick network call, then exit, which was the bane of our existence for a little while.
There are many independent implementations of the same idea (given how easy it is to implement) but all suffer from similar shortcomings:
1. uprobes can be expensive and add latency (they force a context switch and copy data), especially when the hooked functions are called a lot
2. EBPF is not widely available outside of Linux, requires
elevated privileges (compared to a MITM proxy that requires no privileges and works with every OS)
3. Doesn't work with JVM, Rust, any runtime that doesn't use the hooked functions
These are all great callouts. We've worked hard to address some of them, some are future endeavours.
To address your points:
1. In our testing, uprobes add a statistically insignificant amount of latency and in comparison to a MITM proxies it's nearly identical to native.
2. True, we're focused on Linux right now. I'm looking forward to exploring Microsoft's eBPF implementation and exploring how we can support the Windows ecosystem.
3. You're right that the technique we are using for OpenSSL will not work for other runtimes. That said, there are other techniques that we've implemented in our Pro offering for the JVM, Go, and NodeJS. Rust is in the works!
What does the usage pattern look like for this. Will I need to be root to run it, and can it run from inside a container without "real" host root?
I'm always looking for a way to make sniffing traffic from inside a container easier, and if I could attach a debug sidecar with something like an eBPF based SSL pre-master key extractor (both on incoming and outgoing requests) it starts to feel a lot like having network JTAG.
Qtap does require root privileges to function as it uses eBPF to hook into kernel and userspace program functions. The good news is it can also be run within a container.
There are some important flags when spinning it up in docker: `--privileged`, `--cap-add CAP_BPF`, `--cap-add CAP_SYS_ADMIN`, and `--pid=host`. These provide access to load eBPF programs, and monitor traffic.
Many deployments use Kubernetes daemonsets where Qtap runs in a container, but monitors all of the traffic on the node. The Qpoint paid offering comes with a Control Plane that produces context specific dashboards so seeing what's happening from a specific container, or pod namespace can provide a lot of insights into your deployments.
I'm honestly not that interested in constant logging or central collection. I think it's a perfectly useful product, but I find it hard to get buy in for that sort of system in the short term.
Supposedly `kubectl debug` does allow you to set a `sysadmin` profile and grant the debug sidecar `privileged` access. I think that would be a neat low cost way to get value out of your product quickly, right when I have an issue, which would maybe help build some organizational trust and goodwill to make the rest of the stack easier to buy.
It would also solve an issue for me that I would really like solving :P
As of today, we don't output pcap or har files though these are additions I'd like to make in the future, they aren't currently on our near term roadmap.
Kinda related, anyone know of something similar for Windows? This is definitely going in my toolkit, but I need something similar for Windows client traffic inspection (tls 1.2+) to get the full picture. Working with proprietary client/server coms over tls. Can use a special debug build, but requires shutting down and replacing. Need something in-sutu.
Thanks! We're really excited about Qcontrol and what it will be able to provide! The rules in that doc are powered by our Rulekit project https://github.com/qpoint-io/rulekit if you're curious about seeing more.
As far as log sinks, we have stdout right now. We have been working on Fluentbit and will eventually add a bunch more. If you have a request, drop them here!
We also have a services concept which support an "event store" and "object store", where the object store handles artifacts that may contain sensitive data and don't need to be indexed for search/aggregation (this is an S3 compliant store). The event store handles all of the events from connection audit logs (these cover the ip protocol level) to individual http request/response pairs. The event store is a custom API we use and need to write some proper documentation for, stay tuned!
Standard output is all you need. Let a consumer on the other end do the rest of the work. Don’t make supporting every log collector under the sun your problem. Add OTel support if you really feel the need to.
I'm a heavy forward proxy user. Whatever the performance hit, I don't notice it. I do notice the performance hit of HTTPS versus HTTP.
Modifying response bodies in the forward proxy is less than ideal. The proxy must wait for the full response body to be received before making modifications.
With the minimal perf impact, does that mean it is not 100% guaranteed to catch all traffic? I’d think you’d have to insert yourself synchronously into the comms or allow some to get past unseen (eg when systems are heavily loaded).
That was one of our biggest motivators when dreaming up Qtap.
How can we remove the impact that proxies have on connections, AND see the content without having to manage a custom certificate authority, AND not have to instrument all of our code.
Just run the qtap agent on whatever Linux machine has apps running on it and it will see everything through the kernel vs eBPF.
You can customize config and/or integrate with existing observability pipelines, but initially you just need to turn it on for it to work. No app instrumentation required.
Java is supported, but currently in the pro version. Since JavaSSL is implemented in Java code, which runs in the Java VM and not exported as static symbols that can be uprobe'd, there is a bit more involved to generate a bridge between the JVM bytecode and static symbols that can be probed.
This is a great point, and Qtap itself does need to be used with care. The company behind Qtap (Qpoint.io) provides full inventory and alerting for this sort of scenario.
That said, the eBPF verifier has robust security guarantees and runs on every load. So arbitrary mem access for example isn't possible. Qtap runs exclusively on your nodes, so you control what it captures and where that data goes. Our paid offering provides more functionality with a Control Plane solutions that provides dashboards, alerting, and live config updates. However, all sensitive information, like captured http bodies, are uploaded to a S3 compliant bucket that you control. This could be S3, Minio, or anything else that supports the S3 API. We never see this information.
It's intentionally designed for deployment within your infra and abides by the security policies you set within your org.
What's the recommended way of locking down communications for this application? With a MITM based solution it's fairly clear you can lock its egress down to precisely what it needs at the network policy level at least, whereas it's a bit trickier with this as it necessarily shares an instance with other processes.
Is uploading clear text encrypted in flight data to another system even a good idea in most cases? In some cases that won't even be allowed because you'd end up storing regulated information in a way that auditors won't approve of (e.g. sometimes there is a requirement for field level encryption when data is in storage/at rest)
You've definitely hit on a point that we've talked about at length and have come to terms that different organizations have different requirements, especially when it comes to regulatory and compliance.
Qtap can be locked down with local firewalls or perimeter firewalls like other applications running within a network. The TLS inspection can also be disabled with a `--tls-probes=none` flag on startup.
Even without inspection enabled, Qtap provides rich context when it comes to connections to processes. For example, source/destination information, bandwidth usage, SNI information, container meta, even Kubernetes pod and namespace meta. All of this can paint a thorough picture of what's happening with zero instrumentation.
When it comes down to it, some orgs may not be able to use the TLS inspection or require specific methods of persisting data. If we can't support this today, our goal is to address these as they come up and hopefully help devs and ops folks working in these constrained environments get what they need while maintaining compliance.
I imagine one challenge you’ll face is how to keep it configurable and flexible prior to any data leaving the system. For example, you’d like to default to “intercept everything,” but also allow the user to define rulesets for which packets to ignore or include. That would be a nicer UX with an application-level tool, but at that point you’ve already exfiltrated the data they want to filter. So you’ll need a rules engine that can execute locally in the eBPF program, but now that program is becoming much more complicated.
Also worth noting this is very similar to the code path that got Crowdstrike in trouble when they crashed every device on the internet because of a bug in the parser of their rules engine.
>The company behind Qtap (Qpoint.io) provides full inventory and alerting for this sort of scenario.
Could you expand on this? I haven't seen anything on your company website that suggests detection of this kind of stuff. Also, could you explain how this could be detected? Through another eBPF program?
Yes, since NixOS runs on a standard Linux kernel, Qtap will work. I realized that we weren't scanning the nix store for shared libs (eg. OpenSSL) and added that this morning.
To everyone building these things: Please add a disclaimer to say something like:
"This is not a vulnerability: eBPF currently requires root access to do this. Also, eBPF makes this easy but does not make it possible, as debuggers, interposers/shims, and other tools can also attach to pre-encryption points, and therefore banning eBPF (as some people want to do after seeing projects like this) would not actually improve security, but it would instead _reduce_ security as it would prevent eBPF-based security solutions from being used."
Great idea!
On an unrelated note, your work has inspired most of my career in Solaris/Illumos/Linux systems and honestly this project likely wouldn't have happened if it wasn't for all of your books/blogs/projects to help me along the way. Thank you!
Thanks, glad it's useful!
Does this work for Go binaries? My understanding is that Go programs do all the encryption "in the process" so the data is encrypted before eBPF can intercept it. I'd love to be wrong about that!
We have Go support, but it is not open sourced yet. Go is a bit more complicated but we were able to get it after some cave diving in the ELF formats. To give you a little insight on how this works, because Go is statically linked, we need to pull several different offsets of the functions we are going to hook into.
We do this by scanning every version of Go that is released to find offsets in the standard library that won't change. Then when we detect a new Go process, we use an ELF scanner to find some function offsets and hook into those with uprobes. Using both of these, we have all the information we need to see Go pre-encryption content as well as attribute it to connections and processes.
Great approach. I love the choice of practicality over generalization.
Are these offsets consistent across compilation targets, and they vary only by version of the Go binary? Or do you need to do this scan for every architecture?
The short answer is that we only have to calculate the offset per go version, no expensive runtime scanning is required.
The long answer is that the offsets are the byte alignment offsets for the go structs containing the pointers to the file descriptor and buffers. Fortunately we only have to calculate these for each version where the TLS structs within go actually change, so not even for every version. For instance, if a field is added, removed, or changes type then the location in memory where those pointers will be found changes. We can then calculate the actual offset at runtime where we know which architecture (amd64, arm64, etc) with a simple calculation. Within the eBPF probe, when the function is called, it uses pointer arithmetic to extract the location of the file descriptor and buffer directly.
Is the precomputation sufficiently resilient to cross-compilation where the system may be composed of significantly different versions than something mainstream like Debian might ship? I'm thinking of embedded targets built using the Yocto Project tooling.
I think you only need to use the eBPF approach for statically linked programs.
ISTR, at some point in the far past, using LD_PRELOAD with my own shims to capture TLS traffic before encryption/after decryption. I might have it lying around somewhere here.
Ok, that's exciting, and thanks for the insight!
Most programs do encryption without syscalls! eBPF can intercept userspace execution, which they do as mentioned in the post:
> The key idea is to hook into common TLS libraries (like OpenSSL) before encryption and after decryption
I saw that, but Go doesn't use dynamically linked libraries for encryption, so I don't think it helps in this particular case.
If I want to do something similar, do you know where the relevant parts of the eBPF docs are?
Qtap scans binaries of processes as well known locations for OpenSSL on startup, then passes the offsets to eBPF where it hooks into the SSL_read and SSL_write to get the content before or after it's been encrypted.
This is the eBPF side: https://github.com/qpoint-io/qtap/blob/main/bpf/tap/openssl....
The Go side which indicates what we are scanning for is here: https://github.com/qpoint-io/qtap/blob/main/pkg/ebpf/tls/ope...
For more docs on the topic: - https://docs.ebpf.io/ is a must read - https://eunomia.dev/en/tutorials/30-sslsniff/ has a tutorial on cracking OpenSSL open and getting the content as well. The tutorials they have are fantastic in general
There's a similiar tool https://github.com/gojue/ecapture
Grafana's Beyla also basically works the same way https://github.com/grafana/beyla
Thanks for sharing. I didn't know about this one.
> Beyla supports a wide range of programming languages (Go, Java, .NET, NodeJS, Python, Ruby, Rust, etc.)
Although "gRPC and HTTP2 are not supported at the moment"
https://grafana.com/docs/beyla/latest/distributed-traces/
I was just about to ask what the difference is here with `ecapture`
`ecapture` has been around for a while and do a lot of great stuff and a lot of functionality overlaps.
Our aim is to make Qtap extensible and via a plugin system. We have http1/2 streaming capabilities and a plugin engine to run these in what we call a stack. Our goal is to add more protocols, like gRPC in the near future.
We have a few example plugins that do things like report request/response's and push access information to standard out in a console or structure log format. Our Pro version has a few more plugins like the ability to report errors (eg. an AI agent is getting HTTP 429 errors). These can be pushed to a service or log aggregator.
To summarize, we do a lot of the same things that ecapture does. We'd like to be less of a tool and more of a "always running" that ops, opsec, and devs use to answer tough questions. We look forward to open sourcing more of plugins as they mature!
Qtap works with Java too, which ecapture hasn't figured out yet
Have been following this project for a while, cool stuff!
I work a bunch with vpn-like networking on Android phones and it would be cool to have a bit of info on how I might get something like working on phones. I guess its probably not your typical usecase.
Currently since the project is a VPN client, I already intercept all of the packets, I have a pcap writer and can write to files or a tcp sockets and connect wireshark to it - but it needs a bunch of complication to setup the keys so that I can see through encryption, so anything that would make that process easier would be great.
I'm curious what your product does
I've seen that type of behavior for apps that inject ads and add affiliate marketing links
Its an internet sharing app that uses Wi-Fi direct and BLE
The wireshark stuff is only for when I'm debugging
That sounds like a hotspot with extra steps, what does it do differently?
It can do multi-hop. It can also automatically authenticate across any number of phones, sort of like how radius works with fixed wifi (android hotspots don't support radius - you can only connect with radius as a client). You can also earn money from sharing your internet.
I know that arguing that SSLKEYLOGFILE is all you need will just be a different version of the rsync/dropbox comment, but I do wonder under what circumstances is one able to strace a binary and isn’t able to make it dump session keys? I read the headline and set high hopes on finding a nifty way to mitm apps on Android - alas, I’m not sure this would work there necessarily.
Mostly that SSLKEYLOGFILE has only been an (disabled by default) OpenSSL feature for a few weeks (literally), apart from that it's something implemented by some other libraries (notably libcurl) on top. But it's very far from "just set this env var and the keys will pop out of any app using TLS".
The big usecase for me would be if you could attach the trace after starting the binary. The idea of coming into a production system that's behaving unexpectedly and getting a network sniff without having to fiddle with certificates is very attractive.
There's an alternative implementation where SSLKEYLOGFILE is more "dynamic" and permits being toggled on an off during runtime, but that doesn't currently exist.
My big use case is watching on the SERVER side, my coworkers will be asking me to help them debug something and I just want to see the HTTP plaintext, I don't really want to try running Apache under SSLKEYLOGFILE or something, I just want to see the data. ;-)
Isn't there already mechanisms for patching specific SSL libraries to view encrypted requests (e.g. frida)? What is the benefit of using eBPF?
The main benefit is complete coverage. In production systems there are many different workloads with many different binaries, each with different build processes. Leveraging eBPF enables seeing everything on a system without having to adjust the build pipeline.
To hook into OpenSSL, don't you either need dynamic linking or userspace programs to compile your hooks in? Go and many Rust and C++ binaries tend to prefer static linking, so I wonder if this solution is workable there.
Great point! Yes it supports both scenarios. Qtap scans the binary ELF (curl, rust, etc) and looks for the TLS symbols. If they were statically compiled the eBPF probes will be attached directly to the binary, if dynamically linked the probes will be attached to the symbols in the library (.so).
Yeah, so -O2 and -O3 are likely to be problems for you, and the ELF surgery is very invasive.
Fair point on -O2 and -O3 optimized bins. We've approached this by building custom bin utils that are optimized for blazingly fast symbol recognition. Traditional ELF tools that focus on providing comprehensive context for debugging, we are strip away everything that is not the symbol locations we need.
We've also added caching so frequently used bins don't require multiple scans. Shared libraries as well. This has proven effective with optimized binaries, especially bins that are optimized, start, make a super quick network call, then exit, which was the bane of our existence for a little while.
Just found out about a related things: https://github.com/cle-b/httpdbg
Anyone have any experience with it?
this looks like it hooks into python libs, not all openssl traffic system-wide.
There are many independent implementations of the same idea (given how easy it is to implement) but all suffer from similar shortcomings:
1. uprobes can be expensive and add latency (they force a context switch and copy data), especially when the hooked functions are called a lot
2. EBPF is not widely available outside of Linux, requires elevated privileges (compared to a MITM proxy that requires no privileges and works with every OS)
3. Doesn't work with JVM, Rust, any runtime that doesn't use the hooked functions
These are all great callouts. We've worked hard to address some of them, some are future endeavours.
To address your points:
1. In our testing, uprobes add a statistically insignificant amount of latency and in comparison to a MITM proxies it's nearly identical to native.
2. True, we're focused on Linux right now. I'm looking forward to exploring Microsoft's eBPF implementation and exploring how we can support the Windows ecosystem.
3. You're right that the technique we are using for OpenSSL will not work for other runtimes. That said, there are other techniques that we've implemented in our Pro offering for the JVM, Go, and NodeJS. Rust is in the works!
What does the usage pattern look like for this. Will I need to be root to run it, and can it run from inside a container without "real" host root?
I'm always looking for a way to make sniffing traffic from inside a container easier, and if I could attach a debug sidecar with something like an eBPF based SSL pre-master key extractor (both on incoming and outgoing requests) it starts to feel a lot like having network JTAG.
Qtap does require root privileges to function as it uses eBPF to hook into kernel and userspace program functions. The good news is it can also be run within a container.
There are some important flags when spinning it up in docker: `--privileged`, `--cap-add CAP_BPF`, `--cap-add CAP_SYS_ADMIN`, and `--pid=host`. These provide access to load eBPF programs, and monitor traffic.
Many deployments use Kubernetes daemonsets where Qtap runs in a container, but monitors all of the traffic on the node. The Qpoint paid offering comes with a Control Plane that produces context specific dashboards so seeing what's happening from a specific container, or pod namespace can provide a lot of insights into your deployments.
I'm honestly not that interested in constant logging or central collection. I think it's a perfectly useful product, but I find it hard to get buy in for that sort of system in the short term.
Supposedly `kubectl debug` does allow you to set a `sysadmin` profile and grant the debug sidecar `privileged` access. I think that would be a neat low cost way to get value out of your product quickly, right when I have an issue, which would maybe help build some organizational trust and goodwill to make the rest of the stack easier to buy.
It would also solve an issue for me that I would really like solving :P
Can it output pcap files or anything similar I can import onto Wireshark or a similar tool? Haven't found anything checking the docs...
As of today, we don't output pcap or har files though these are additions I'd like to make in the future, they aren't currently on our near term roadmap.
Nice. Thanks for the response!
Kinda related, anyone know of something similar for Windows? This is definitely going in my toolkit, but I need something similar for Windows client traffic inspection (tls 1.2+) to get the full picture. Working with proprietary client/server coms over tls. Can use a special debug build, but requires shutting down and replacing. Need something in-sutu.
Was going to ask if it was only passive monitoring or active controlling and found https://docs.qpoint.io/appendix/qcontrol-beta
> Security enforcement: Allowing or denying traffic based on precise conditions
Very cool. What are your supported log sinks?
Thanks! We're really excited about Qcontrol and what it will be able to provide! The rules in that doc are powered by our Rulekit project https://github.com/qpoint-io/rulekit if you're curious about seeing more.
As far as log sinks, we have stdout right now. We have been working on Fluentbit and will eventually add a bunch more. If you have a request, drop them here!
We also have a services concept which support an "event store" and "object store", where the object store handles artifacts that may contain sensitive data and don't need to be indexed for search/aggregation (this is an S3 compliant store). The event store handles all of the events from connection audit logs (these cover the ip protocol level) to individual http request/response pairs. The event store is a custom API we use and need to write some proper documentation for, stay tuned!
Standard output is all you need. Let a consumer on the other end do the rest of the work. Don’t make supporting every log collector under the sun your problem. Add OTel support if you really feel the need to.
My vote would be https://vector.dev/ or https://nanomq.io/ over Fluent Bit for performance reasons. That said, S3/MinIO is fairly universal so that along with stdout should be fine. I'd be interested to learn more about your custom local Pulse Service: https://docs.qpoint.io/readme/data-flow#fully-self-hosted-ma...
I'm a heavy forward proxy user. Whatever the performance hit, I don't notice it. I do notice the performance hit of HTTPS versus HTTP.
Modifying response bodies in the forward proxy is less than ideal. The proxy must wait for the full response body to be received before making modifications.
Can eBPF be any better in this regard.
With the minimal perf impact, does that mean it is not 100% guaranteed to catch all traffic? I’d think you’d have to insert yourself synchronously into the comms or allow some to get past unseen (eg when systems are heavily loaded).
I like the fact this doesn't impact performance like MITM solutions do.
That was one of our biggest motivators when dreaming up Qtap.
How can we remove the impact that proxies have on connections, AND see the content without having to manage a custom certificate authority, AND not have to instrument all of our code.
If installed in a router, can it see traffic of all devices connected to it or only traffic that originates in the router itself?
Perhaps with `direction: all`
https://github.com/qpoint-io/qtap/issues/29#issuecomment-286...
Does it also work on android? Afaik ebpf is also available there.
Not today, maybe one day!
Edit: Another user posted a link to https://github.com/gojue/ecapture which looks like it supports android and has some overlapping functionality.
Look into HTTP Toolit for android (not affiliated with it at all, it's just that cool a tool)
I don't have any answers/questions, but reading through the discussion, all I can say at this point is — Super impressive guys!
Does this support other libraries besides OpenSSL? Such as BoringSSL as used in Chrome/chromium?
How easy is the set up, does this need to be deeply integrated in each step of the life-cycle?
Just run the qtap agent on whatever Linux machine has apps running on it and it will see everything through the kernel vs eBPF.
You can customize config and/or integrate with existing observability pipelines, but initially you just need to turn it on for it to work. No app instrumentation required.
Do you support Java? If so, how do you do this for Java?
Java is supported, but currently in the pro version. Since JavaSSL is implemented in Java code, which runs in the Java VM and not exported as static symbols that can be uprobe'd, there is a bit more involved to generate a bridge between the JVM bytecode and static symbols that can be probed.
Is there anything like this but for MacOS?
I think the closest is an app with a full GUI called LittleSnitch. It's pretty impressive.
https://www.obdev.at/products/littlesnitch/index.html
sounds like a security breach. how you ensure this does not become link in some next complex CVE?
This is a great point, and Qtap itself does need to be used with care. The company behind Qtap (Qpoint.io) provides full inventory and alerting for this sort of scenario.
That said, the eBPF verifier has robust security guarantees and runs on every load. So arbitrary mem access for example isn't possible. Qtap runs exclusively on your nodes, so you control what it captures and where that data goes. Our paid offering provides more functionality with a Control Plane solutions that provides dashboards, alerting, and live config updates. However, all sensitive information, like captured http bodies, are uploaded to a S3 compliant bucket that you control. This could be S3, Minio, or anything else that supports the S3 API. We never see this information.
It's intentionally designed for deployment within your infra and abides by the security policies you set within your org.
What's the recommended way of locking down communications for this application? With a MITM based solution it's fairly clear you can lock its egress down to precisely what it needs at the network policy level at least, whereas it's a bit trickier with this as it necessarily shares an instance with other processes.
Is uploading clear text encrypted in flight data to another system even a good idea in most cases? In some cases that won't even be allowed because you'd end up storing regulated information in a way that auditors won't approve of (e.g. sometimes there is a requirement for field level encryption when data is in storage/at rest)
You've definitely hit on a point that we've talked about at length and have come to terms that different organizations have different requirements, especially when it comes to regulatory and compliance.
Qtap can be locked down with local firewalls or perimeter firewalls like other applications running within a network. The TLS inspection can also be disabled with a `--tls-probes=none` flag on startup.
Even without inspection enabled, Qtap provides rich context when it comes to connections to processes. For example, source/destination information, bandwidth usage, SNI information, container meta, even Kubernetes pod and namespace meta. All of this can paint a thorough picture of what's happening with zero instrumentation.
When it comes down to it, some orgs may not be able to use the TLS inspection or require specific methods of persisting data. If we can't support this today, our goal is to address these as they come up and hopefully help devs and ops folks working in these constrained environments get what they need while maintaining compliance.
I imagine one challenge you’ll face is how to keep it configurable and flexible prior to any data leaving the system. For example, you’d like to default to “intercept everything,” but also allow the user to define rulesets for which packets to ignore or include. That would be a nicer UX with an application-level tool, but at that point you’ve already exfiltrated the data they want to filter. So you’ll need a rules engine that can execute locally in the eBPF program, but now that program is becoming much more complicated.
Also worth noting this is very similar to the code path that got Crowdstrike in trouble when they crashed every device on the internet because of a bug in the parser of their rules engine.
Your writing style reminds LLM for some reason.
>The company behind Qtap (Qpoint.io) provides full inventory and alerting for this sort of scenario.
Could you expand on this? I haven't seen anything on your company website that suggests detection of this kind of stuff. Also, could you explain how this could be detected? Through another eBPF program?
My first thought was this is cool. My second thought was that this is going to be impossible to securely manage and administer.
Does this work in NixOS?
Yes, since NixOS runs on a standard Linux kernel, Qtap will work. I realized that we weren't scanning the nix store for shared libs (eg. OpenSSL) and added that this morning.
Out of curiosity, do you use NixOS in production? If so, how large of an installation do you have?
Stream any good movies lately?