Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does my data fit in RAM?

yourdatafitsinram.net

Post reply on HN
Print view View original

151–160 of 167 posts

Re: Does my data fit in RAM?

#151

Post by jdc »

mrmrcoleman wrote: post #121
sevensor wrote: post #116

Earlier quoted context omitted.

There's also the "not all memory is RAM" trick: plan ahead with enough swap to fit all the data you intend to process, and just pretend that you have enough RAM. Let the virtual memory subsystem worry about whether or not it fits in RAM. Whether this works well or horribly depends on your data layout and access patterns.

Interesting. Can you provide some examples of where this is the correct approach?

With the Varnish HTTP cache the authors started out with a very "mmap or bust" type of approach, but later added a malloc-based backend.

Re: Does my data fit in RAM?

#152

Post by manigandham »

arnon wrote: post #148
manigandham wrote: post #147

Earlier quoted context omitted.

You don't need to store everything into RAM to get fast results. Data warehouse relational databases are designed exactly for this kind of fast SQL analysis over extremely large datasets. They use a variety of techniques like vectorized processing on compressed columnar storage to get you quick results. Google's BigQuery, AWS Redshift, Snowflake (are all hosted), or MemSQL, Clickhouse (to run yourself). Other options…

Some of those aren't designed for data that doesn't fit in RAM. They even have it in their name (MemSQL for example).

All of those support datasets that don't fit in RAM, or else they would be useless at data warehousing.

MemSQL uses rowstores in memory combined with columnstores on disk. Both can be joined together seamlessly and the latest release will automatically choose and transition the table type for you as data size and access patterns change.

Re: Does my data fit in RAM?

#153

Post by satanspastaroll »

ksec wrote: post #114
keanzu wrote: post #57

Earlier quoted context omitted.

Everyone knows that DRAM prices have been in a collapse since early this year, but last week DRAM prices hit a historic low point on the spot market. Based on data the Memory Guy collected from spot-price source InSpectrum, the lowest spot price per gigabyte for branded DRAM reached $2.59 last week. https://thememoryguy.com/dram-prices-hit-historic-low/ You've selected out the low points on the graph: 2012, 2016, and…

>the lowest spot price per gigabyte for branded DRAM reached $2.59 last week. It would be better to reference this as quoted from the article which was written in November 2019. So not really last week >most of the time DRAM has not been available at these prices. I did said price floor.

If it doesen't have to be available, i could sell one stick for $0.01 and that'd be the new floor

Re: Does my data fit in RAM?

#154

Post by Dylan16807 »

malux85 wrote: post #144
malux85 wrote: post #140

Earlier quoted context omitted.

And what about redundancy in case of node failure?

> I mean "when you've proven that the application you've written can fully, or near fully utilise the available power on a single machine, and that when running production-grade workloads, actually does so, then you may scale to additional machines How is that any different? You just backed off a tiny amount by saying “fully or near fully” - you still shouldn’t burden a single host to “fully or near Fully” because: I…

> How is that any different? You just backed off a tiny amount by saying “fully or near fully” - you still shouldn’t burden a single host to “fully or near Fully”

You're missing the word "can". It's a very important part of that sentence.

If your software can't even use 80% of one node, it has scaling problems that you need to address ASAP, and probably before throwing more nodes at it.

> It puts more strain on the hardware and will cause it to fail a LOT faster

Unless you're hammering an SSD, how does that happen? CPU and RAM should be at a pretty stable amount of watts anywhere from 'moderate' load and up, which doesn't strain anything or overheat.

> redundancy

Sure.

Re: Does my data fit in RAM?

#155

Post by Dylan16807 »

ghc wrote: post #113
lmilcin wrote: post #19

Earlier quoted context omitted.

You know what else costs? Humongous amount of servers to run silly stuff to orchestrate other silly stuff to autoscale yet else silly stuff to do stuff on your stuff that could fit into memory and be processed on a single server (+ backup, of course). Add to that small army of people, because, you know, you need specialists of variety of professions just to debug all integration issues between all those components th…

There's a world outside of web apps and SV tech companies. There's a lot of big datasets out there, most of which never hit the cloud at all. Story time: I worked on one project where a single (large) building's internal sensor data (HVAC, motion, etc. 100k sensors) would fill a 40TB array every year. They had a 20 year retention policy. So Dell would just add a new server + array every year. I worked with another co…

> Story time: I worked on one project where a single (large) building's internal sensor data (HVAC, motion, etc. 100k sensors) would fill a 40TB array every year. They had a 20 year retention policy. So Dell would just add a new server + array every year.

That's a lot of data, but what do you even do with it other than take minuscule slices or calculate statistics?

And for those uses, I'd put whether it fits in RAM as not applicable. It doesn't, but can you even tell the difference?

Re: Does my data fit in RAM?

#156

Post by Dylan16807 »

satanspastaroll wrote: post #153
ksec wrote: post #114

Earlier quoted context omitted.

>the lowest spot price per gigabyte for branded DRAM reached $2.59 last week. It would be better to reference this as quoted from the article which was written in November 2019. So not really last week >most of the time DRAM has not been available at these prices. I did said price floor.

If it doesen't have to be available, i could sell one stick for $0.01 and that'd be the new floor

You sure are giving any benefit of the doubt there.

Lowest massively-available price, please and thank you.

Re: Does my data fit in RAM?

#157

Post by satanspastaroll »

cldellow wrote: post #80
dijit wrote: post #58

Earlier quoted context omitted.

How long does it take to copy your data in? And what’s the bandwidth cost involved? People like to talk about the elasticity or compute, but startup is not free (or even cheap in most cases).

If your data is in S3, my experience is that you can push ~20-40MB/core/sec on most instances. OP is probably talking about an x1e.32xlarge. According to Daniel Vassalo's S3 benchmark [1], it can do about 2.7GB/sec. So your 4TB DB might take ~30min to fetch. Bandwidth is free, you'd pay $2 for the 30 min of compute, and some fractions of pennies for the few hundred S3 requests. [1]: https://github.com/dvassallo/s3-be…

It's to note that any data exported out of AWS will be billed at $0.09/GB, or $90/TB

Re: Does my data fit in RAM?

#158

Post by xchaotic »

tmountain wrote: post #125
mrmrcoleman wrote: post #121

Earlier quoted context omitted.

Interesting. Can you provide some examples of where this is the correct approach?

This is how mongodb originally managed all its data. It used memory mapped files to store the data and let the underlying OS memory management facilities do what they were designed to do. This saved the mongodb devs a ton of complexity in building their own custom cache and let them get to market much faster. The downside is that since virtual memory is shared between processes, other competing processes could potent…

Except nowadays with Docker / Kubr you can safely assume the db engine will be the only tenant of a given vm /pod whatever so I think it’s better to let OS do memory management than fight it

Re: Does my data fit in RAM?

#159

Post by ClumsyPilot »

wtallis wrote: post #102
thedance wrote: post #67

Earlier quoted context omitted.

It can't though.

It can, and their fastest enterprise SSD can write at that speed too, or do sequential reads at 7-8GB/s, or random reads at over 4 GB/s. I just ran `time cp /dev/nvme0n1 /dev/null` on the 1TB 970 Pro. The result: real 4m50.724s user 0m2.001s sys 3m10.282s So with literally zero optimization effort, we've hit the spec (and saturated a PCIe 3.0 x4 link).

That's impressive and all, but any fragmentation or non-linear access and performance will fall off a cliff

Re: Does my data fit in RAM?

#160

Post by saint_abroad »

globular-toast wrote: post #145
yongjik wrote: post #105

Even more importantly, does your data have to fit in RAM? There are tons of problems that need to process large data, but touch each item just once (or a few times). You can go a really long way by storing them in disk (or some cloud storage like S3) and writing a script to scan through them. I know, pretty obvious, but somehow escapes many devs.

Escapes many devs? Really? I used to work with biologists who thought they needed to run their scripts on a supercomputer because the first line read their entire file into an array. But if I saw someone who calls themselves a "dev" doing this I'd consider them incompetent.

I once got into an argument with a senior technical interviewer because he wanted a quick solution of an in-memory sort of an unbounded set of log files.

Needless to say I wasn't recommended for the job, and it taught me a valuable lesson: if you don't first give them what they want, you can't give them what they actually need.

Post reply on HN