Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-volatile Storage: CPUs no longer more performant than I/O devices

queue.acm.org

Post reply on HN
Print view View original

61–70 of 121 posts

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#61

Post by dunkelheit »

Animats wrote: post #18
ricw wrote: post #16

Earlier quoted context omitted.

Just to clarify: this is to enable programs to have more a more direct and faster interface to data, while retaining data consistency and safety that typically would be managed by the OS?! Is this not just another form of DMA (direct memory access)? And if so, how would it differ from current implementations? Sometimes DMA only refers to ram, though on many systems this is fluid between different data storage types (…

"this is to enable programs to have more a more direct and faster interface to data, while retaining data consistency and safety that typically would be managed by the OS?" No, managed by the database engine. The idea is to put the data-intensive operations of a database engine into a highly parallel SPU. The application would see an interface much like an SQL or NoSQL database.

Maybe that's a dumb question but why should this chip be highly parallel? In the case of GPU computations are by their nature embarrassingly parallel but I don't see why this should be the case for SPU. Why not just add a few more CPU cores and let this specialized controller handle the problem of concurrent accesses?

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#62

Post by Animats »

matt_d wrote: post #52
mikehollinger wrote: post #35

Earlier quoted context omitted.

Being up front: this is what I work on for IBM Systems. A buddy wrote this blog ( https://www.ibm.com/developerworks/community/blogs/fe313521-... ) with a little more info. What we have is an IO offload accelerator that knows how to drive high bandwidth IOs to some external storage device. A user app doesn't interact with the device - they make shared library calls to read or write data from a particular buffer, and…

Interesting! Incidentally (since this may be somewhat related), I'm wondering, what are your thoughts on the Persistent Memory Manager approach, as in the following: Justin Meza, Yixin Luo, Samira Khan, Jishen Zhao, Yuan Xie, and Onur Mutlu: "A Case for Efficient Hardware/Software Cooperative Management of Storage and Memory." Workshop on Energy-Efficient Design, 2013. Context: "emerging high-performance NVM technolo…

One giant flat address space is not the answer. Hardware people tend to come up with approaches like that because flat address spaces and caching are well understood hardware. It's the same thinking that leads to "storing into device registers" as an approach to I/O control, even when the interface is really packets over a serial cable as in FireWire or USB or PCI Express.

File systems and databases are useful abstractions, from an ease of use, security, and robustness perspective. The challenge is to make them go faster. Pushing the machinery behind them out to special-purpose hardware can do that.

The straightforward thing to do first is to to take some FPGA part and use it to implement a large key/value store using non-volatile solid state memory. That's been done at Stanford[1], Berkeley[2], and MIT[3], and was suggested on YC about six years ago.[4] One could go further, and implement more of an SQL database back end. It's an interesting data structure problem; the optimal data structures are different when you don't have to wait for disk rotation, but do need persistence and reliability.

[1] http://csl.stanford.edu/~christos/publications/2014.hwkvs.nv... [2] https://www.cs.berkeley.edu/~kubitron/courses/cs262a-F14/pro... [3] https://dspace.mit.edu/handle/1721.1/91829 [4] https://news.ycombinator.com/item?id=1628550

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#63

Post by yongjik »

mikeash wrote: post #23
ape4 wrote: post #13

Earlier quoted context omitted.

eg "faster"

See also: "utilize" instead of "use."

Actually, "fast" doesn't mean "performant" (OK, maybe that's not a word, in which case I mean "having high performance"), unless you want to also say "speed" means "performance".

Also "utilize" is more specific than "use": it implies "using for a desirable purpose". Consider:

(1) This daemon process is utilizing all the available bandwidth.

(2) This daemon process is using all the available bandwidth.

Sometimes people making up new words do come across as lazy, but you can easily go too far in the other direction, too.

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#64

Post by sgt101 »

rdtsc wrote: post #28
sgt101 wrote: post #22

Earlier quoted context omitted.

I have just bought a machine with one of these drives and a bunch of GPU's with the intent of running tensor-flow on it. Do you think that Fedora will allow for the kind of tuning you describe or should I stick with Centos? I was thinking that I would need Fedora because these new devices need new kernels and drivers and so on, but maybe I am just confused?

I would guess give Fedora a try or whatever GPU drivers support better. CentOS / RHEL as a rule is behind as far as package versions in order to be more stable. But they do bring in new drivers and back-port many fixes and packages. Most of the stuff that applies to CentOS will apply to Fedora as well as a rule.

thank you.

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#65

Post by CyberDildonics »

marcosdumay wrote: post #41

> and the performance of an SCM (hundreds of thousands of I/O operations per second) is such that one or more entire many-core CPUs are required to saturate it. So, we are getting a lot of data, but latency is still killing. (Even more taking into account that this thing has a few stages of pipeline inside.) Anyway, our CPU is getting distributed nearer IO and memory. We are going to get NUMA machines, everything poi…

Knights Landing from Intel is already NUMA (I think). I'm not sure if it can be bought yet, but it should be very close to release.

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#66

Post by woodchuck64 »

shimon wrote: post #19
Animats wrote: post #15

I've made that point before on YC.[1] We need to view fast storage as something other than a disk accessed through the OS, and other than slow RAM accessed as raw memory. Access through the OS is too slow, and access as raw memory is too risky. What's probably needed is something like a GPU sitting between the CPU and the fast persistent storage. Call this an SPU, or "storage processing unit." What would such a devic…

This sounds interesting, but why should this be a new piece of hardware as opposed to a new OS service? Are these functions simply so specialized that implementing them in the OS would be a bottleneck (even though the CPU has plenty of free cycles)?

Well the article does say: "Our own experience has been that efforts to saturate PCIe flash devices often require optimizations to existing storage subsystems, and then consume large amounts of CPU cycles. "

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#67

Post by mikeash »

yongjik wrote: post #63
mikeash wrote: post #23

Earlier quoted context omitted.

See also: "utilize" instead of "use."

Actually, "fast" doesn't mean "performant" (OK, maybe that's not a word, in which case I mean "having high performance"), unless you want to also say "speed" means "performance". Also "utilize" is more specific than "use": it implies "using for a desirable purpose". Consider: (1) This daemon process is utilizing all the available bandwidth. (2) This daemon process is using all the available bandwidth. Sometimes peopl…

I have no idea what difference is implied between (1) and (2) there.

What's the difference between speed and performance here?

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#68

Post by thescriptkiddie »

erichocean wrote: post #58

My sense is this is only true today because OS kernels are ridiculously slow relative to what the hardware can achieve. Most of my recent designs treat RAM as if it were (what we used to considered to be) disks, i.e. all computation and in-process data is in cache exclusively, and "going to RAM" requires the use of a B-tree-like structure to amortize the cost. For example, once you've opened a RAM page line on a norm…

Not that is some interesting thinking. You got a blog post elaborating on this?

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#69

Post by thescriptkiddie »

CyberDildonics wrote: post #65
marcosdumay wrote: post #41

> and the performance of an SCM (hundreds of thousands of I/O operations per second) is such that one or more entire many-core CPUs are required to saturate it. So, we are getting a lot of data, but latency is still killing. (Even more taking into account that this thing has a few stages of pipeline inside.) Anyway, our CPU is getting distributed nearer IO and memory. We are going to get NUMA machines, everything poi…

Knights Landing from Intel is already NUMA (I think). I'm not sure if it can be bought yet, but it should be very close to release.

Aren't AMD parts already (cache-coherent) NUMA?

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#70

Post by mikehollinger »

i336_ wrote: post #57
mikehollinger wrote: post #35

Earlier quoted context omitted.

Being up front: this is what I work on for IBM Systems. A buddy wrote this blog ( https://www.ibm.com/developerworks/community/blogs/fe313521-... ) with a little more info. What we have is an IO offload accelerator that knows how to drive high bandwidth IOs to some external storage device. A user app doesn't interact with the device - they make shared library calls to read or write data from a particular buffer, and…

This is pretty interesting. It's possible that people might confuse this bit... > The performance of SCMs means that systems must no longer "hide" them via caching and data reduction in order to achieve high throughput. ...in the original article with your mention of caching; by "cache coherency," I assume you're referring that your addon (card?) can introspect into the CPU cache? That's pretty awesome if that's what…

> I assume you're referring that your addon (card?) can introspect into the CPU cache? That's pretty awesome if that's what's happening.

Y - see page 5, section 3.1.1 of http://www-304.ibm.com/webapp/set2/sas/f/capi/CAPI_POWER8.pd... for some info. Also - it doesn't have to be a card. ;-) It just is that today...

> RAM Emulation

This is hard. Telling a program (and the OS) that different pages are fundamentally different will require some pretty drastic changes. For example- how does one malloc memory from an NVDIMM vs a regular DIMM, and differentiate between the two?

> Performance

Yes - as an example we can show that it takes ~26 threads on the CPU to drive ~450k IOPs to some external storage. Doing the same thing with the accelerated IO path requires about 4 HW threads on the main CPU. This kind of lines up with the point of the article.

> Are you aware of any alternative offerings for x86 that offer similar performance?

To my knowledge no one else has a similar architecture that's shipping today.

>- What does this thing (I have no idea if it's a card, a module...) look like? Being able to see "the thing" is generally really cool :)

http://www.nallatech.com/solutions/openpower-capi-developer-... or http://www.alpha-data.com/dcp/capi.php are your choices for Altera or Xilinx FPGA support (as of today).

> a) re: ATX

see http://www.enterprisetech.com/2014/10/08/tyan-ships-first-no... from last year. Go talk to Tyan if you want to buy one.

> b) if/when full-scale POWER8 systems will be available in the style of Heroku/OpenShift.

https://ptopenlab.com/cloudlabconsole/index.html has some boxes with CAPI cards...

Post reply on HN