Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-volatile Storage: CPUs no longer more performant than I/O devices

queue.acm.org

Post reply on HN
Print view View original

1–10 of 121 posts

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#3

Post by rdtsc »

Yeah per packet processing at 40Gbps and higher is problematic on regular kernels, OS stack and CPUs. A lot of really can be cache hits -- hundreds of nanoseconds. Article mentions that too:

--- To put these numbers in context, acquiring a single uncontested lock on today's systems takes approximately 20ns, while a non-blocking cache invalidation can cost up to 100ns, only 25x less than an I/O operation. ---

It also depends if workload is throughput sensitive or latency sensitive. If it is latency, can do things like tie processes and interrupts to cores, isolate those cores, etc. For throughput can processes more than one packet at a time perhaps.

Then there is dpdk and even unikernels.

> CPU has responsibilities beyond simply servicing a device—at the very least, it must process a request and act as either a source or a sink for the data linked to it. In the case of data parallel frameworks such as Hadoop and Spark,7,17 the CPU

That's why you get more CPUs and explicitly isolate them if you can. But now depending on how they share data with other CPUs there will be invalidated cache lines so will pay that way as well.

In general if you run on RHEL / CentOS ( a lot banks, military and enterprise deployments do ), there is this helpful guide as an overview:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#4

Post by dogma1138 »

So NVDIMM's... Is any one actually making those except Viking and is anyone actually supporting them in servers except SuperMicro?

These are basically DDR3/DDR4 DIMM's with onboard flash and a supercap/battery pack to provide persistence incase of system reboots and power failures.

They are also a bit odd as they would ignore various system event calls from the BIOS/UEFI and then have to be specifically managed by various software hacks that create RAM drives and access the memory directly rather than working with OS virtual memory. Since NVDIMM's are basically treated as system memory by both the server and the OS they pretty much only work for very very boutique applications I'ts a bit odd that these are presented as the next step in storage evolution while being effectively an overpriced hack. I've only seen them actually been use in weird server setups like the overclocked watercooled servers that are used for HFT where they strip everything even the OS as possible and bypass anything that adds even a few NS of latency and don't mind running their own code for everything from a bastardized TCP stack that isn't even remotely compliant but works to their own in memory custom database.

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#10

Post by emcq »

I like the article, but not your title. It implies that this trend of I/O becoming highly performant has occurred recently, when in fact has been observed and studied for quite some time [0, 1]. Even before SSDs Gigabit ethernet was saturating CPUs needing to do more than DMA a packet, and I'm sure this trend continued for some time. The original title seems more accurate: "Implications of the Datacenter's Shifting Center", and references the existing trends in an insightful article.

[0] http://ucsdnews.ucsd.edu/archive/newsrel/supercomputer/11-09...

[1] http://nvsl.ucsd.edu/index.php?path=pubs

Post reply on HN