RAM Is the New Disk • Hacker News Board

Re: RAM Is the New Disk

#71

Post by ars » Mon, Jul 31, 2017, 6:45 PM UTC

majewsky wrote: post #69

ars wrote: post #54

Earlier quoted context omitted.

> Not sure if ramdisks are pinned though. Ramdisks will go to swap. A memory leak will force the entire ramdisk into swap, and reading it back into memory afterward is 10 to 100 times slower than reading normal files off of a disk.

> Ramdisks will go to swap. Assuming that you have swap. I don't; I want my SSD to stay alive.

If you have a memory leak, and a ramdisk, and no swap then the OOM killer will trigger.

Hopefully it will target the program with the memory leak, but this is not guaranteed.

Swap is useful because you can shift unused memory onto disk. There are many programs that allocate (and write) a lot of memory that they never afterward use.

By having swap you make more room for cache in memory.

SSD doesn't matter here - this is not swap thrashing, but rather occasional writes.

Re: RAM Is the New Disk

#72

Post by AndyNemmity » Mon, Jul 31, 2017, 9:15 PM UTC

manigandham wrote: post #67

AndyNemmity wrote: post #48

Earlier quoted context omitted.

I've used MemSQL, and it's rapid ingest by the default isn't ACID compliant, so it sort of depends on how you compare it. The results of the columnstore data was pretty fast, and it's even faster in memory. Depends on what you're doing, and what the requirements are. Was really impressed by MemSQL, and loved the wire compatibility with mysql, so don't take this as just a knock on MemSQL in anyway.

> by the default isn't ACID compliant What do you mean?

I'm surprised you wouldn't understand this as it's an absolutely requirement given you are a user of MemSQL.

But then I went to their docs to link you to the details, and it feels like they intentionally avoid stating clearly the problem.

Essentially, they allow committed transactions to hit memory and not disk. They allow you to configure it so that's not the case, but it isn't the default, and looking over the current documentation they certainly aren't clear about it like an open source project would be.

transaction-buffer needs to be set to 0 for durability, but the way the docs are explaining it is trying to confuse not being durable, as a different kind of durability.

I'm not interested in getting into a long discussion about this though, but it's difficult to explain the literal issue when they do such a marketing job of trying to hide the specifics.

Now I'm far less surprised a user wouldn't know this. Apologies for my forward initial statement.

Re: RAM Is the New Disk

#73

Post by manigandham » Tue, Aug 01, 2017, 4:07 AM UTC

AndyNemmity wrote: post #72

manigandham wrote: post #67

Earlier quoted context omitted.

> by the default isn't ACID compliant What do you mean?

I'm surprised you wouldn't understand this as it's an absolutely requirement given you are a user of MemSQL. But then I went to their docs to link you to the details, and it feels like they intentionally avoid stating clearly the problem. Essentially, they allow committed transactions to hit memory and not disk. They allow you to configure it so that's not the case, but it isn't the default, and looking over the curr…

It's not a surprise, that's normal. Durability just means writes are safe, how it does so doesn't matter. All databases use a write-ahead log with small batch/async flushes and this is the same with MemSQL.

The enterprise edition has HA so with data on 2 nodes for safety. Otherwise you lose the whole point of in-memory performance (for writes) if you're going to write every single bit to disk immediately.

They explain it clearly on the durability page: http://docs.memsql.com/docs/using-durability-and-recovery

Re: RAM Is the New Disk

#74

Post by kthejoker2 » Thu, Aug 03, 2017, 6:47 PM UTC

icebraining wrote: post #64

jlebrech wrote: post #50

with "serverless" applications you can read the whole app into memory and run it and then clear it for the next app, which i'm sure speeds things up.

That's not how it works; programs are kept "warm" for some time after each requests, or indefinitely (e.g. in App Engine you can choose dynamic or resident instances).

Just chiming in to say this is also true for Lambda and Azure Functions.

What Id really like is for them to scale up to full fledged VMs once some usage or performance threshold was hit.