Bhyve bridges are inefficient: every packet traverses NIC → CPU → bridge → VM, adding unnecessary copies that kill throughput. Switching to SR-IOV eliminated that overhead and I saturated the 10 GbE link.
I do not see how that follows. Memory bandwidth is measured in the hundreds of Gb/s. You can issue tens of unnecessary full memory copies before you bottleneck at a paltry 10 Gb/s.
It is much more likely there is something else terribly wrong in a network stack if it can not even drive a measly 10 Gb/s.
When this settles down, I look forward to all of jail/iojail, Sylve, Bastille, Bhyve documenting this in a mutually consistent manner. As it stands, I have managed to completely knot my brain over the abstractions, what is happening. It's me, not the systems, but I think there is a little bit of "meh, I understand it, so it must be obvious to anyone smart" going on, and alas, I am not smart, and I get confused easily.
I'm in bastille atm, but have been in all of them and TrueNAS core. and libvirt over on the other unix.
I cor one welcome and applaud any progress on the bsd front,and this seems to be huge.
> -tso4 -tso6 -vlanhwfilter -vlanmtu -vlanhwtso -vlanhwtag -vlanhwcsum -lro
Whys the author disabling tso and lro? Whats the motivation?
I'm not familiar with the other flags.
Bhyve bridges are inefficient: every packet traverses NIC → CPU → bridge → VM, adding unnecessary copies that kill throughput. Switching to SR-IOV eliminated that overhead and I saturated the 10 GbE link.
I do not see how that follows. Memory bandwidth is measured in the hundreds of Gb/s. You can issue tens of unnecessary full memory copies before you bottleneck at a paltry 10 Gb/s.
It is much more likely there is something else terribly wrong in a network stack if it can not even drive a measly 10 Gb/s.
That assumes memory bandwidth is the issue, and not latency and/or CPU.
You used the new optimized bridges on FreeBSD 15?
On Linux?
When this settles down, I look forward to all of jail/iojail, Sylve, Bastille, Bhyve documenting this in a mutually consistent manner. As it stands, I have managed to completely knot my brain over the abstractions, what is happening. It's me, not the systems, but I think there is a little bit of "meh, I understand it, so it must be obvious to anyone smart" going on, and alas, I am not smart, and I get confused easily.
I'm in bastille atm, but have been in all of them and TrueNAS core. and libvirt over on the other unix.