Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does my data fit in RAM?

yourdatafitsinram.net

Post reply on HN
Print view View original

161–167 of 167 posts

Re: Does my data fit in RAM?

#161

Post by wtallis »

ClumsyPilot wrote: post #159
wtallis wrote: post #102

Earlier quoted context omitted.

It can, and their fastest enterprise SSD can write at that speed too, or do sequential reads at 7-8GB/s, or random reads at over 4 GB/s. I just ran `time cp /dev/nvme0n1 /dev/null` on the 1TB 970 Pro. The result: real 4m50.724s user 0m2.001s sys 3m10.282s So with literally zero optimization effort, we've hit the spec (and saturated a PCIe 3.0 x4 link).

That's impressive and all, but any fragmentation or non-linear access and performance will fall off a cliff

You'd probably be surprised. For reads, there are tons of drives that will saturate PCIe 3.0 x4 with 4kB random reads. Throughput is a bit lower because of more overhead from smaller commands, but still several GB/s. Fragmentation won't appreciably slow you down any further, as long as you keep feeding the drive a reasonably large queue of requests (so you do need your software to be working with a decent degree of parallelism).

What will cause you serious and unavoidable trouble is if you cannot structure things to have any spatial locality. If you only want one 64-bit value out of the 4kB block you've fetched, and you'll come back later another 511 times to fetch the other 64b values in that block, then your performance deficit relative to DRAM will be greatly amplified (because your DRAM fetches would be 64B cachelines fetch 8x each instead of 4kB blocks fetched 512x each).

Re: Does my data fit in RAM?

#162

Post by lmilcin »

Dylan16807 wrote: post #155
ghc wrote: post #113

Earlier quoted context omitted.

There's a world outside of web apps and SV tech companies. There's a lot of big datasets out there, most of which never hit the cloud at all. Story time: I worked on one project where a single (large) building's internal sensor data (HVAC, motion, etc. 100k sensors) would fill a 40TB array every year. They had a 20 year retention policy. So Dell would just add a new server + array every year. I worked with another co…

> Story time: I worked on one project where a single (large) building's internal sensor data (HVAC, motion, etc. 100k sensors) would fill a 40TB array every year. They had a 20 year retention policy. So Dell would just add a new server + array every year. That's a lot of data, but what do you even do with it other than take minuscule slices or calculate statistics? And for those uses, I'd put whether it fits in RAM a…

This is another problem. Is really 20 year retention policy necessary for ALL sensor data? Can it be somehow aggregated and only then the aggregated data to be subject to retention policy? Can the retention policy be made to make it possible to lose some fidelity gradually (the way RRDtool is used by Nagios, for example)?

Re: Does my data fit in RAM?

#163

Post by ghc »

Dylan16807 wrote: post #155
ghc wrote: post #113

Earlier quoted context omitted.

There's a world outside of web apps and SV tech companies. There's a lot of big datasets out there, most of which never hit the cloud at all. Story time: I worked on one project where a single (large) building's internal sensor data (HVAC, motion, etc. 100k sensors) would fill a 40TB array every year. They had a 20 year retention policy. So Dell would just add a new server + array every year. I worked with another co…

> Story time: I worked on one project where a single (large) building's internal sensor data (HVAC, motion, etc. 100k sensors) would fill a 40TB array every year. They had a 20 year retention policy. So Dell would just add a new server + array every year. That's a lot of data, but what do you even do with it other than take minuscule slices or calculate statistics? And for those uses, I'd put whether it fits in RAM a…

They paid us $600K every six months to analyze the data and suggest adjustments to their control systems (it's called continuous commissioning, but it's not really continuous due to laws in many places about requiring a person in the loop on controls). They saved millions of dollars every year doing this, because large, complex buildings drift out of optimized airflow and electricity use very quickly.

Agreed that 20 year retention is silly. We thought it was silly, but the policies reflected the need for historical analysis for audit purposes.

It does in fact matter what you can fit in RAM though. We had to adapt all our systems to a janky SQL Server setup that was horrible for time series data and make our software run on those servers. RAM availability for working sets was a huge bottleneck (hence the cost of analysis).

Re: Does my data fit in RAM?

#164

Post by FridgeSeal »

malux85 wrote: post #140
FridgeSeal wrote: post #129

Earlier quoted context omitted.

I don't mean "let your production systems spool up to point where you're maxing out a single machine" - that would be exceedingly silly. I mean "when you've proven that the application you've written can fully, or near fully utilise the available power on a single machine, and that when running production-grade workloads, actually does so, then you may scale to additional machines. What this means is not getting a 9-…

And what about redundancy in case of node failure?

Fail over?

I’m not against redundancy/HA in production systems, I’m opposing clusters of machines to perform data workloads that could more efficiently handled by single machines. Also note here that I’m talking about data science and machine learning workloads, where node failure simply means the job isn’t marked as done, a replacement machine gets provisioned and we resume/restart.

I’m not suggesting running your web servers and main databases on a single machine.

Re: Does my data fit in RAM?

#165

Post by satanspastaroll »

Dylan16807 wrote: post #156
satanspastaroll wrote: post #153

Earlier quoted context omitted.

If it doesen't have to be available, i could sell one stick for $0.01 and that'd be the new floor

You sure are giving any benefit of the doubt there. Lowest massively-available price, please and thank you.

If it's lowest massively-available price then this

>most of the time DRAM has not been available at these prices.

should make it not the floor. If the floor doesen't have to be available, then what's the exact point it becomes relevant? Otherwise the price is simply misleading

Re: Does my data fit in RAM?

#166

Post by Dylan16807 »

satanspastaroll wrote: post #165
Dylan16807 wrote: post #156

Earlier quoted context omitted.

You sure are giving any benefit of the doubt there. Lowest massively-available price, please and thank you.

If it's lowest massively-available price then this >most of the time DRAM has not been available at these prices. should make it not the floor. If the floor doesen't have to be available, then what's the exact point it becomes relevant? Otherwise the price is simply misleading

Any price that is massively-available becomes relevant and stays relevant forever.

A price has to be massively available at a point in time to matter. It doesn't have to be available forever to matter. It feels like you're conflating the two.

The price is on a downward trend, but there are hitches and setbacks. One fair way to measure it is to use some kind of average. Another also-fair way to measure it is to go by the lowest "real" price, where "real" means you can buy something like a million sticks on the open market.

When we're talking about whether we should be impressed by a price, using the lowest historical price for comparison makes sense.

(And just to be absolutely clear, you would need to adjust the metric for a product that goes up in price over time. But for something on a downward trend, this metric works fine.)

Re: Does my data fit in RAM?

#167

Post by 1MachineElf »

I've no idea if the intended audience here would ever run their workloads on Solaris, but like the IBM POWER systems, Oracle & Fujitsu SPARC servers also max out at 64TB of RAM. I didn't see those included here.
Post reply on HN