Live data from Hacker News

Your data fits in RAM

yourdatafitsinram.com

201–210 of 230 posts

Re: Your data fits in RAM

#201
post #195
post #193

Earlier quoted context omitted.

I'd also toss in ease of correctness if you don't have a really well-known problem with a good test suite: I've seen a number of cases where people developed “tight” C code which gave results which appeared reasonable and then ran it for months before noticing logic errors which would either have been impossible or easier to find in a higher-level language or, particularly, one which had better numeric error handling…

> one which had better numeric error handling characteristics Which language do you suggest which matches this? I don't know many people who use the language "numeric error handling characteristic" but I know the benefit of already having good written high-level primitives (or libraries).

The first example which comes to mind are integer overflows, which are notoriously easy to miss in testing with small / insufficiently-representative data. Languages like Python which use a arbitrary-precision system will avoid incorrect results at the expense of performance and I believe a few other languages will raise an exception to force the developer to decide the correct way to handle them.

The other area I've seen this a lot is caused by the way floating point math works not matching non-specialists’ understanding – unexpectedly-failing equality checks, precision loss making the results dependent on the order of operations, etc. That's a harder problem: switching to a Decimal type can avoid the problem at the expense of performance but otherwise it mostly comes down to diligent testing and, hopefully, some sort of linting tool for the language you're using.

Re: Your data fits in RAM

#202
post #64

Earlier quoted context omitted.

Ditto for computationally intensive work: if it is CPU dominated, more CPU's calculating in parallel will be of advantage, even if the data could fit some RAM. There's no a single simple answer, but sure, whenever less computers are enough, less should be used. The recent problem is, some people love "clouds" so much today that they push there the work that could really be done locally.

The problem is getting 100% of the work to be parallel. If I have to do that last 10% in one machine there isn't much point to more than 10 machines.

You are talking about https://en.wikipedia.org/wiki/Amdahl's_law , am I right?

Re: Your data fits in RAM

#203
post #88

Earlier quoted context omitted.

Untrue about the speed of R. R and Python are always around the same speed, but there are always other options specially with R, where there is always more than one way to do anything. We have data.tables and dplyr which data.tables is maybe on average 50% faster and on some points multiple faster than Python [ http://datascience.la/dplyr-and-a-very-basic-benchmark/ ]

> mm system.time(eigen(mm)) user system elapsed 5.26 0.00 5.25 IPy [1] >>> xx = np.random.rand(1000000).reshape(1000, 1000) IPy [2] >>> %timeit(np.linalg.eig(xx)) 1 loops, best of 3: 1.28 s per loop But where R really stinks is memory access: > system.time(for(x in 1:1000) for(y in 1:1000) mm[x, y] >> def do(): ...: for x in range(1000): ...: for y in range(1000): ...: xx[x, y] = 1 ...: IPy [10] >>> %timeit do() 10 l…

That's why you never ever grow lists with R. do.call('rbind',...) or even better data.table::rbindlist(). You can't blame R for being slow if you don't know how to write fast R code.

Re: Your data fits in RAM

#204

Earlier quoted context omitted.

A step between RAM and SSD could be "Your data fits in RAM in compressed form". LZ4 compression is takes 3-4x longer than memcpy. LZ4 decompression is only 50% [1] slower. 2-3 GB/s per core. [1]: Your mileage may vary.

If it's slower both in and out what's the benefit? To guy below me: Ah, thanks. I thought the guy above was trying to say it's slower than paging to disk. : )

Basically, parsing the percentages, using compressed data in RAM is ~1 order of magnitude slower to get into memory, and less than 1 order of magnitude slower to read from RAM. That is still around 2 (or more depending on setup) orders of magnitude faster than the hit to go to SSD / SSD RAID.

Useful knowledge on latency: https://gist.github.com/jboner/2841832

Re: Your data fits in RAM

#205
Before core, there was tape. Tape used to be backup medium, then disk became the new tape. Bubble memory begat SSD, so memory has in some sense become the new disk.

RAM is the new disk: now for some, later for others.

Re: Your data fits in RAM

#206
post #53
post #38

If you are programming in R, you sure better hope it does!

After reading the title I was sure there was something about R in the comments. You can program R in Spark you can now program in R http://blog.revolutionanalytics.com/2015/01/a-first-look-at-... Now you can work directly with SQL Server as announced this week by MS. http://www.computerworld.com/article/2923214/big-data/sql-se... I have had a ton of arguments about R's "biggest weakness" being that it uses RAM. I hav…

For my workloads, R has always choked on its single thread long before it choked on memory. And the parallelism options are terrible hacks.

Re: Your data fits in RAM

#207

So this seems to use 6.144TiB as the limit that will fit in RAM. That's 1.536TiB x 4 when using the latest Xeon I could find[1]. According to the specs though you should be able to use 8, so the total limit should actually be 1.536 x 8 = 12.288 TiB. 12TiB of RAM, that's quite amazing. [1] http://ark.intel.com/products/84688/Intel-Xeon-Processor-E7-...

It seemed to use 6.000000000000000444089209850...??? TiB when I tried values.

It seems to use different values of cutoff depending on if you are using MiB/GiB/TiB/etc. I tested with GiB and 6144 is OK, 6145 is not.

Re: Your data fits in RAM

#208
post #192

Earlier quoted context omitted.

I own a ASUS gaming laptop... It has two flaws: One, it has nVidia Optimus (that just suck, whoever implemented it should be shot). Two, the I/O is not that good, even with a 7200 RPM disk, and Windows 8.1 make it much worse (windows for some reason keep running his anti-virus, superfetch, and other disk intensive stuff ALL THE TIME). This is noticeable when playing emulated games: games that use emulator, even cartr…

I can bet in your case it's not the disk I/O actually, unless you have some very strange emulator the file access should still go through the OS disk cache. VMware for example surely benefits from it. How many GB do you have on the machine? How much is still left free when you run the emulator and the other software you need?

I noticed it as disk i/o because I would leave task manager running on second screen and every time the game lagged memory and CPU use were below 30% and disk was 100%, and if I left sorting by disk usage, the first places are the windows stuff, and after them, the emulator.

Re: Your data fits in RAM

#209
post #17

Yes! As someone who frequently runs memory-intensive algorithms on large(ish) datasets, I have a hard time explaining to many technical people that moving from a single server to a cluster increases complexity and cost by an incredible amount. It affects key decisions like algorithm and language, and generally requires a lot of tweaking. When a problem becomes big enough, moving to a cluster is absolutely the right d…

Complexity, sure. But cost? I thought a single 1 TB RAM server is more expensive than 10x 100 GB RAM servers. And many people don't want to deal with physical hardware. Dealing with physical hardware increases operational complexity too. They want to rent a virtual/cloud server. Which provider allows you to rent a virtual server with 1 TB RAM?

SoftLayer lets you rent a server with 512 GB ram directly from their order form. (Monthly price for the ram is $1,444.00; a dual xeon 2620 server you can put it in is $380/month). It's baremetal, not virtual, but you can file tickets with them for any hardware stuff that comes up.

If you work outside the order form, you can get 768 GB, too. 1 TB is possible with their haswell servers, but availability seems limited.

Re: Your data fits in RAM

#210
post #120

Earlier quoted context omitted.

Comparison still holds, because if you buy a cluster with X amount of RAM the price will be roughly the same as a single server with X amount of RAM. Except that for some large X there won't be any off the shelf servers you can buy with that amount of RAM (let's say 2000GB), but lets be honest here, 99% of companies needs are under that X especially if we're talking about startups.

> if you buy a cluster with X amount of RAM That's not the only option. You can rent a cluster for a lot cheaper.

...
Post reply on HN