Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Comparing Filesystem Performance in Virtual Machines

mitchellh.com

Post reply on HN
Print view View original

31–40 of 46 posts

Re: Comparing Filesystem Performance in Virtual Machines

#31

Post by hnriot »

weddpros wrote: post #26
mitchellh wrote: post #24

Earlier quoted context omitted.

You're assuming the writes are actually going to a physical disk. As I mentioned in the post, the hypervisors are very likely just writing to RAM and not ever committing it to disk. Even when you `fsync()` from a VM, there is no guarantee the hypervisor puts that to disk. If you look at the graphs, they corroborate this. The "native" disk never really exceeds 500 to 600 MB/s, which is about as fast as my SSD goes. Th…

I meant to say you've not benchmarked disk access, and we have no idea what each filesystem is actually doing. Caching performance is "nice", but it says nothing about the actual performance we'd get in real use. Maybe the "slow" fs just exhibits less aggressive caching, which might prove just as efficient depending on the workload. It's definitely interesting to note the huge differences, but I'd really like to see…

how are these not real conditions?

Re: Comparing Filesystem Performance in Virtual Machines

#32

Post by cgbystrom »

With our team, we also found shared folders performance to be too low. Our Python framework/app is very read-heavy and stat() a lot of files (the Python module loading system isn't your friend)

We ended up using the synchronization feature in PyCharm to continually rsync files from native FS into the VirtualBox instance. Huge perf improvement but a little more cumbersome for the developers. But so far it has been working good, PyCharm's sync feature does what it is supposed to.

Re: Comparing Filesystem Performance in Virtual Machines

#33

Post by stefanha »

This benchmark is bogus because the iozone -I flag is missing. -I uses O_DIRECT to avoid the page cache.

Due to page cache usage it's hard to say what this benchmark is comparing. The I/O pattern seen by the actual disk, shared folder, or NFS may be different between benchmark runs. It all depends on amount of RAM available, state of cache, readahead, write-behind, etc.

Please rerun the benchmark with -I to get an apples-to-apples comparison.

Re: Comparing Filesystem Performance in Virtual Machines

#34

Post by gizmo686 »

rdtsc wrote: post #23
weddpros wrote: post #10

3GB/s writes on a single SSD should raise more eyebrows. I dunno what was actually benchmarked, but there's a problem somewhere...

Writing to /dev/null is even faster I bet.

  >dd if=/dev/zero of=/dev/null
  ^C32646111+0 records in
  32646110+0 records out
  16714808320 bytes (17 GB) copied, 17.881 s, 935 MB/s
Running on the host OS of my laptop.

Re: Comparing Filesystem Performance in Virtual Machines

#35

Post by wila »

mitchellh wrote: post #24
weddpros wrote: post #10

3GB/s writes on a single SSD should raise more eyebrows. I dunno what was actually benchmarked, but there's a problem somewhere...

You're assuming the writes are actually going to a physical disk. As I mentioned in the post, the hypervisors are very likely just writing to RAM and not ever committing it to disk. Even when you `fsync()` from a VM, there is no guarantee the hypervisor puts that to disk. If you look at the graphs, they corroborate this. The "native" disk never really exceeds 500 to 600 MB/s, which is about as fast as my SSD goes. Th…

The problem is that the tests are flawed and the native speed being slower as virtual is a pretty big red herring for that. That's OK because benchmarking that tests what you are after is actually very difficult. Small assumptions can create big differences. If you can't guarantee that the data has actually been written to the disk then you're testing caching mechanisms, something you already point out in your article, but then you're no longer testing file system performance as the article claims it is benchmarking. The problem is that we don't even know which caching mechanisms (guest OS, hypervisor, hard disk driver) or that the conditions are always the same.

A typical thing that performance benchmarks do to negate guest OS caching is to process significantly more data as what the available RAM is set to. For example, if your guest OS RAM is set 512MB, process 10GB of random data. Of course then the question is how-to get random data as you don't want to end up testing the random generator ;) or your host OS caching.

Another way to make sure you test data committed to disk could be to include a "shutdown guest OS" part and measure total time until the guest has been fully shut down.

I know that at least VMware has the ability to turn off disk caching (in Fusion, select Settings, advanced "Hard Disk Buffering" Even while you tested the same guest OS, we don't even know if the hard disk adapters where both using the same hard disk drivers. Performance differs between IDE/Sata/SCSI drivers. SCSI drivers have queue depths, IDE drivers have not.

Re: Comparing Filesystem Performance in Virtual Machines

#36

Post by buster »

hnriot wrote: post #31
weddpros wrote: post #26

Earlier quoted context omitted.

I meant to say you've not benchmarked disk access, and we have no idea what each filesystem is actually doing. Caching performance is "nice", but it says nothing about the actual performance we'd get in real use. Maybe the "slow" fs just exhibits less aggressive caching, which might prove just as efficient depending on the workload. It's definitely interesting to note the huge differences, but I'd really like to see…

how are these not real conditions?

I don't know what you are doing in real conditions but i am for sure not only running everything in my RAM (that'd be great, i wouldn't even need a harddisk anymore). The benchmark needs to be revised to account for the VM caching or it's just useless. Or rename it to "benchmark of memory access and caching algorithms".

Re: Comparing Filesystem Performance in Virtual Machines

#38

Post by mitchellh »

wila wrote: post #35
mitchellh wrote: post #24

Earlier quoted context omitted.

You're assuming the writes are actually going to a physical disk. As I mentioned in the post, the hypervisors are very likely just writing to RAM and not ever committing it to disk. Even when you `fsync()` from a VM, there is no guarantee the hypervisor puts that to disk. If you look at the graphs, they corroborate this. The "native" disk never really exceeds 500 to 600 MB/s, which is about as fast as my SSD goes. Th…

The problem is that the tests are flawed and the native speed being slower as virtual is a pretty big red herring for that. That's OK because benchmarking that tests what you are after is actually very difficult. Small assumptions can create big differences. If you can't guarantee that the data has actually been written to the disk then you're testing caching mechanisms, something you already point out in your articl…

While the article uses only 64 KB/MB files for the analysis, the full Excel workbook contains data that had up to 512 MB files. The VMs only had 256 MB of RAM, so I did indeed test the RAM-spillover cases. The results were very similar, though unsurprisingly the VMs didn't perform quite as well (though they did still beat over native).

I never tested going over the native's RAM.

Re: Comparing Filesystem Performance in Virtual Machines

#39

Post by rdtsc »

gizmo686 wrote: post #34
rdtsc wrote: post #23

Earlier quoted context omitted.

Writing to /dev/null is even faster I bet.

>dd if=/dev/zero of=/dev/null ^C32646111+0 records in 32646110+0 records out 16714808320 bytes (17 GB) copied, 17.881 s, 935 MB/s Running on the host OS of my laptop.

Hint: always use a non-default block size (unless you have a reason not to)

   $ dd if=/dev/zero of=/dev/null bs=1M
   ^C55034+1 records in
   55034+0 records out
   57707331584 bytes (58 GB) copied, 6.04479 s, 9.5 GB/s
And this is a 7 year old laptop. I could get blistering fast speeds writing to /dev/null on my latest 6 core i7 ;-)

Re: Comparing Filesystem Performance in Virtual Machines

#40

Post by Nux »

liuw wrote: post #28
Nux wrote: post #20

Would love to see in there KVM and Xenserver; you know, stuff that actual clouds run on.

I think you mean KVM and Xen . Xen hypervisor is open source project just like KVM while XenServer is a product that uses Xen hypervisor. Just think of Linux kernel and Linux distributions.

I actually meant Xenserver (which is open source, too), but you do have a valid point. Xen's in use a lot.
Post reply on HN