Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-volatile Storage: CPUs no longer more performant than I/O devices

queue.acm.org

Post reply on HN
Print view View original

111–120 of 121 posts

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#111

Post by fungos »

mikehollinger wrote: post #70
i336_ wrote: post #57

Earlier quoted context omitted.

This is pretty interesting. It's possible that people might confuse this bit... > The performance of SCMs means that systems must no longer "hide" them via caching and data reduction in order to achieve high throughput. ...in the original article with your mention of caching; by "cache coherency," I assume you're referring that your addon (card?) can introspect into the CPU cache? That's pretty awesome if that's what…

> I assume you're referring that your addon (card?) can introspect into the CPU cache? That's pretty awesome if that's what's happening. Y - see page 5, section 3.1.1 of http://www-304.ibm.com/webapp/set2/sas/f/capi/CAPI_POWER8.pd... for some info. Also - it doesn't have to be a card. ;-) It just is that today... > RAM Emulation This is hard. Telling a program (and the OS) that different pages are fundamentally diffe…

Thanks for the links! ptopenlab seems very interesting! Question, just for curiosity, how much does it cost to have something like this?

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#112

Post by dogma1138 »

wmf wrote: post #11
dogma1138 wrote: post #4

So NVDIMM's... Is any one actually making those except Viking and is anyone actually supporting them in servers except SuperMicro? These are basically DDR3/DDR4 DIMM's with onboard flash and a supercap/battery pack to provide persistence incase of system reboots and power failures. They are also a bit odd as they would ignore various system event calls from the BIOS/UEFI and then have to be specifically managed by va…

Intel will be pushing them hard starting with Skylake-EP, so some people are getting themselves ready.

Did Intel created a new interface for NVDIMM's? because to work with the ones Viking makes you pretty much need to hack your linux kernel to ensure that it doesn't access physical memory over a certain address range, and I don't even know if or how can you use them on Windows based applications.

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#113

Post by wmf »

dogma1138 wrote: post #112
wmf wrote: post #11

Earlier quoted context omitted.

Intel will be pushing them hard starting with Skylake-EP, so some people are getting themselves ready.

Did Intel created a new interface for NVDIMM's? because to work with the ones Viking makes you pretty much need to hack your linux kernel to ensure that it doesn't access physical memory over a certain address range, and I don't even know if or how can you use them on Windows based applications.

Yes, Intel basically controls UEFI/ACPI and has been submitting patches to Linux for a while. http://pmem.io/

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#114

Post by CamperBob2 »

xxs wrote: post #86
CamperBob2 wrote: post #59

Earlier quoted context omitted.

Sounds like multiple caches are an obvious solution. No code in the OS needs access to the user code or data cache, and vice versa. Smart cache management might be one of Intel's goals for the Altera buyout.

>> No code in the OS needs access to the user code or data cache This is not true for the data. How do you pass any data structures outside CPU registers then, say the path to a file to open. Normally it's a char*[0] (indeed passed in a register) but then the OS actually reads the data off the process memory (L1 data cache usually) [0]: http://linux.die.net/man/3/open

I'd say a cache isn't the right structure for passing around data (or references to data) that you know will be accessed very soon by completely different code.

As userland code, you'd like to grant the OS access to a particular subset of lines in D$ while keeping it out of your C$ altogether. Traditional implementations fail in both respects.... and at the same time, the OS probably can't take advantage of its own historical locality because userland has evicted it since the last call.

From what other people are saying it sounds like these problems are being worked on, though.

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#115

Post by matt_d »

hadagribble wrote: post #90
rsp1984 wrote: post #54

Please excuse my ignorance on this matter but will this technology have any impact on the hierarchy levels below disk (i.e. RAM and CPU caches)? Compared to Register, L1 and L2 access RAM access is still really slow. Will non-volatile storage latencies rival or exceed those of standard RAM? From how I understand the article it's primarily disk IO speed that's affected, correct?

Yes. Even the persistent memories that attach to the memory bus are currently quite a bit slower than DRAM (5-7x from estimates I've seen), while the difference with PCIe-attached ones is even more. I'm not sure what the future holds in terms of latencies for non-volatile storage but sub-DRAM levels aren't within reach yet.

On a side note, it's interesting to me that emerging memory technologies currently seem to be mainly focused on addressing the "from-DRAM-to-disk" part of the memory hierarchy.

That is, as you mentioned, not directly competing with DRAM, and consistently on the same side of the 1 microsecond dividing line between memory and storage; as in:

http://www.rambusblog.com/2015/10/15/mid-when-memory-and-sto... (note SCM placed between DRAM and SSD)

http://semiengineering.com/the-memory-and-storage-hierarchy/

As far as the other side of the line is concerned, I think I've only seen proposals for hybrid-cache architectures (HCA) -- other than http://link.springer.com/chapter/10.1007%2F978-1-4419-9551-3... -- with a hybrid approach (e.g., combining SRAM/eDRAM/STT-RAM/PCRAM) probably making sense due to latency/endurance/bandwidth trade-offs.

If anything, there seems to be more development on the DRAM interface itself -- with multiple candidates for the (or a) DDR4's successor, so far involving Wide I/O (Samsung), Hybrid Memory Cube (Intel, Micron), High Bandwidth Memory (SK Hynix, AMD, Nvidia): http://www.extremetech.com/computing/197720-beyond-ddr4-unde...

(Latency and bandwidth improvements seem promising, http://semiengineering.com/which-memory-type-should-you-use/)

One interesting development I've seen involves reducing SRAM's footprint, by moving from 6T (6-transistors) cell to a 1T (one-transistor) one: http://www.eetimes.com/document.asp?doc_id=1328453

It's fairly recent development, though, and it remains to be seen how is it going to fare.

Other than the above, there doesn't really seem to be much progress around competing with/improving SRAM. However, this may become increasingly important, since some of the technological process scaling issues apply to SRAM, too.

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#116

Post by pmehra »

Both Tandem DP2 and IBM Coupling Facility on zSeries Sysplexes worked exactly like how you envisage an SPU would work. Therefore, when we developed RDMA-attached persistent memory at Tandem in 2002, we put it under the control of DP2/ADP process pairs. Later, we ported it to HP-UX and InfiniBand RDMA. There is one paper at IPDPS'04 and several published patents you can look up in my Google Scholar page. pmem.io crowd is reinventing some of this wheel. If any of you work at HPE, you can find much more detailed internal papers, source code, drivers, firmware, and other stuff that the outside world cannot get to.

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#117

Post by i336_ »

fungos wrote: post #111
mikehollinger wrote: post #70

Earlier quoted context omitted.

> I assume you're referring that your addon (card?) can introspect into the CPU cache? That's pretty awesome if that's what's happening. Y - see page 5, section 3.1.1 of http://www-304.ibm.com/webapp/set2/sas/f/capi/CAPI_POWER8.pd... for some info. Also - it doesn't have to be a card. ;-) It just is that today... > RAM Emulation This is hard. Telling a program (and the OS) that different pages are fundamentally diffe…

Thanks for the links! ptopenlab seems very interesting! Question, just for curiosity, how much does it cost to have something like this?

These are my own hazy opinions, but I suspect it starts at "Wallet vaporizes from shock" and goes up from there.

I remember reading about how old IBM mainframes used to have a couple ThinkPads (literally two, for redundancy) bolted just inside the cabinet door, just to change low-level configuration settings. It seems to me that these POWER8 boxen are aimed toward that end of the market.

YouTube's history interface is terrible, but I managed to dig this out - https://www.youtube.com/watch?v=jOzPTopt7HE - which shows the different discrete components in a POWER8 system and how they're put together. I should probably do a bit more research on this, that video is quite basic (and 2 years old now).

POWER8 systems seem to necessarily take up a lot of space, and not does this contributes to the raw material cost, it's also a factor in renting, considering that you can pack a basic but decent punch with 1U or 2U of x86. My guess is that IBM isn't trying to be competitive here, but aim for a specific market. That'll influence the price too.

It's thanks to market factors and the state of education (which sometimes produces wins like these!) that places like PTOpenLab exist, I think (again, this is an [un]educated guess), and I'm super appreciative that they do. I haven't figured out how the "blue points" system works yet though (you get 500, and use 10/day for running a VM); I can at least say that the number doesn't increase each day. I vaguely recall reading something to the effect of creating HDD images for the platform would give you points based on how many other people downloaded them (there's somewhere you can upload to), but I can't find that documentation now.

Another fun tidbit: the dashboard UI is based on SmartAdmin (a premium jQuery plugin, apparently), which comes with Chrome-compatible voice control (note the mic button at the top-right). The voice command list doesn't show because of a 404, but you can find the list in app.config.js (F12 -> Network -> reload page) - scroll to the pile of "show"s. Useless, and horribly flaky, but extremely cool. :D

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#118

Post by jleahy »

erichocean wrote: post #58

My sense is this is only true today because OS kernels are ridiculously slow relative to what the hardware can achieve. Most of my recent designs treat RAM as if it were (what we used to considered to be) disks, i.e. all computation and in-process data is in cache exclusively, and "going to RAM" requires the use of a B-tree-like structure to amortize the cost. For example, once you've opened a RAM page line on a norm…

Right, modern CPUs can do 50 gigaflops per core. There's absolutely no chance we're going to have non-volatile storage that can do hundreds of billions of IOPS any time soon (if only because you won't be able to get that much data over PCI-express).

Further given you can saturate 16 lanes of PCIe when talking to a GPU there's no reason you shouldn't be able to do the same for storage, it's just a matter of having the right abstractions and the right kind of thinking like you're saying.

It sounds more like storage and RAM are going to converge (and people are still learning to deal with how slow RAM is compared to the CPU these days).

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#119

Post by mikehollinger »

fungos wrote: post #111
mikehollinger wrote: post #70

Earlier quoted context omitted.

> I assume you're referring that your addon (card?) can introspect into the CPU cache? That's pretty awesome if that's what's happening. Y - see page 5, section 3.1.1 of http://www-304.ibm.com/webapp/set2/sas/f/capi/CAPI_POWER8.pd... for some info. Also - it doesn't have to be a card. ;-) It just is that today... > RAM Emulation This is hard. Telling a program (and the OS) that different pages are fundamentally diffe…

Thanks for the links! ptopenlab seems very interesting! Question, just for curiosity, how much does it cost to have something like this?

SuperVessel's free! http://www.zdnet.com/article/ibm-launches-supervessel-a-free...

Re: Non-volatile Storage: CPUs no longer more performant than I/O devices

#120

Post by mikehollinger »

i336_ wrote: post #117
fungos wrote: post #111

Earlier quoted context omitted.

Thanks for the links! ptopenlab seems very interesting! Question, just for curiosity, how much does it cost to have something like this?

These are my own hazy opinions, but I suspect it starts at " Wallet vaporizes from shock " and goes up from there. I remember reading about how old IBM mainframes used to have a couple ThinkPads (literally two, for redundancy) bolted just inside the cabinet door, just to change low-level configuration settings. It seems to me that these POWER8 boxen are aimed toward that end of the market. YouTube's history interface…

> "wallet vaporizes"

The SuperVessel lab's free, and there are several resources to rent time on a P8 VM if SuperVessel isn't appropriate.

Also, the video you found is the E8xx product line, which are at the high end of the enterprise / scale-up product line, and (incidentally) different from Mainframes.

Here's some links about the 2U / 2socket boxes if space of each node is important:

S822LC - https://www.youtube.com/watch?v=OdlLszagnos

S822L - https://www.youtube.com/watch?v=xF_fw_NJ5nI (not IBM's, but a reasonable unboxing video)

And if you're interested in videos, check out the IBM Power Systems youtube channel: https://www.youtube.com/user/ibmpowersystems/videos

Post reply on HN