How to use Linux vsock for fast VM communication
popovicu.com
How to use Linux vsock for fast VM communication
1–10 of 22 posts
Re: How to use Linux vsock for fast VM communication
#2Says it is fast, but presents zero benchmarks to demonstrate it is actually fast or even “faster”. It is shameful to make up adjectives just to sound cool.
Re: How to use Linux vsock for fast VM communication
#3Says it is fast, but presents zero benchmarks to demonstrate it is actually fast or even “faster”. It is shameful to make up adjectives just to sound cool.
vsock is pretty widely used, and if you're using virtio-vsock it should be reasonably fast. Anyway if you want to do some quick benchmarks and have an existing Linux VM on a libvirt host:
(1) 'virsh edit' the guest and check it has '' in the section of the XML.
(2) On the host:
$ nbdkit memory 1G --vsock -f
$ nbdinfo 'nbd+vsock://2'
And then you can try using commands like nbdcopy to copy data into and out of the host RAM disk over vsock. eg:
$ time nbdcopy /dev/urandom 'nbd+vsock://2' -p
$ time nbdcopy 'nbd+vsock://2' null: -p
To compare it to regular TCP:
host $ nbdkit memory 1G -f -p 10809
vm $ time nbdcopy /dev/urandom 'nbd://host' -p
vm $ time nbdcopy 'nbd://host' null: -p
I had to kill the firewall on my host to do the TCP test (as trying to reconfigure nft/firewalld was beyond me), which actually points to one advantage of vsock, it bypasses the firewall. It's therefore convenient for things like guest agents where you want them to "just work" without reconfiguration hassle.
Re: How to use Linux vsock for fast VM communication
#4Given how slow protobufs and grpc is, I wonder if the socket transport would ever be the bottleneck to throughput here.
Changing transports means if you want to move your grpc server process to a different box you now have new runtime configuration to implement/support and new performance characteristics to test.
I can see some of the security benefits if you are running on one host, but I also don't buy the advantages highlighted at the end of the article about using many different OS's and language environments on a single host. Seems like enabling and micro-optimising chaos instead of trying to tame it.
Particularly in the ops demo: Statically linking a C++ grpc binary, and standardising on host OS and gcc-toolset, doesn't seem that hard. On the other hand, if you're using e.g. a python rpc server are you even going to be able to feel the impact of switching to vsock?
Re: How to use Linux vsock for fast VM communication
#5Says it is fast, but presents zero benchmarks to demonstrate it is actually fast or even “faster”. It is shameful to make up adjectives just to sound cool.
vsock is pretty widely used, and if you're using virtio-vsock it should be reasonably fast. Anyway if you want to do some quick benchmarks and have an existing Linux VM on a libvirt host: (1) 'virsh edit' the guest and check it has ' ' in the section of the XML. (2) On the host: $ nbdkit memory 1G --vsock -f (3) Inside the guest: $ nbdinfo 'nbd+vsock://2' (You should see the size being 1G) And then you can try using…
Is nbdcopy actually touching the data consumer side or is splicing to /dev/null ?
Re: How to use Linux vsock for fast VM communication
#6Says it is fast, but presents zero benchmarks to demonstrate it is actually fast or even “faster”. It is shameful to make up adjectives just to sound cool.
vsock is pretty widely used, and if you're using virtio-vsock it should be reasonably fast. Anyway if you want to do some quick benchmarks and have an existing Linux VM on a libvirt host: (1) 'virsh edit' the guest and check it has ' ' in the section of the XML. (2) On the host: $ nbdkit memory 1G --vsock -f (3) Inside the guest: $ nbdinfo 'nbd+vsock://2' (You should see the size being 1G) And then you can try using…
Ah, thanks. That is a much better example than the one in the article.
Re: How to use Linux vsock for fast VM communication
#7This is interesting. I wasn't familiar with vsock.
I think the use of Bazel and gRPC in the article distracts from understanding how vsock works. I found this article[1] much more approachable.
[1]: https://medium.com/@F.DL/understanding-vsock-684016cf0eb0
Re: How to use Linux vsock for fast VM communication
#8Earlier quoted context omitted.
vsock is pretty widely used, and if you're using virtio-vsock it should be reasonably fast. Anyway if you want to do some quick benchmarks and have an existing Linux VM on a libvirt host: (1) 'virsh edit' the guest and check it has ' ' in the section of the XML. (2) On the host: $ nbdkit memory 1G --vsock -f (3) Inside the guest: $ nbdinfo 'nbd+vsock://2' (You should see the size being 1G) And then you can try using…
Is nbdcopy actually touching the data consumer side or is splicing to /dev/null ?
It's actually copying the data. Splicing wouldn't be possible (maybe?), since NBD is a client/server protocol.
The difference between nbdcopy ... /dev/null and nbdcopy ... null: is that in the second case we avoid writing the data anywhere and just throw it away inside nbdcopy.
Re: How to use Linux vsock for fast VM communication
#9Says it is fast, but presents zero benchmarks to demonstrate it is actually fast or even “faster”. It is shameful to make up adjectives just to sound cool.
vsock is pretty widely used, and if you're using virtio-vsock it should be reasonably fast. Anyway if you want to do some quick benchmarks and have an existing Linux VM on a libvirt host: (1) 'virsh edit' the guest and check it has ' ' in the section of the XML. (2) On the host: $ nbdkit memory 1G --vsock -f (3) Inside the guest: $ nbdinfo 'nbd+vsock://2' (You should see the size being 1G) And then you can try using…
Is that a typo? TCP was 2.5x faster?
I presume this is down to much larger buffers in the TCP stack.
Re: How to use Linux vsock for fast VM communication
#10Earlier quoted context omitted.
vsock is pretty widely used, and if you're using virtio-vsock it should be reasonably fast. Anyway if you want to do some quick benchmarks and have an existing Linux VM on a libvirt host: (1) 'virsh edit' the guest and check it has ' ' in the section of the XML. (2) On the host: $ nbdkit memory 1G --vsock -f (3) Inside the guest: $ nbdinfo 'nbd+vsock://2' (You should see the size being 1G) And then you can try using…
Is that a typo? TCP was 2.5x faster? I presume this is down to much larger buffers in the TCP stack.
Not a typo & yes quite likely. I haven't tuned nbd/vsock at all.
Edit: I patched both ends to change SO_SNDBUF and SO_RCVBUF from the default (both 212992) to 4194304, and that made no difference.