Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MemSQL is now free to use for databases with up to 128GB of RAM usage

memsql.com

Post reply on HN
Print view View original

61–70 of 121 posts

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#61

Post by rickn »

lbriner wrote: post #7

The bit I can't see on the FAQ is about how it is as fast as RAM but protects against data loss using disk. I think most engines guarantee Durability by assuming that once on disk, it won't go anywhere but if it's in RAM, it is susceptible to power outage? If it gets written to disk, it's not as fast as RAM?

[Director of Product Management for MemSQL]

MemSQL has two storage modes Rowstore and Columnstore. The Rowstore is "in-memory" and the columnstore is "on-disk" but those are oversimplifications. The rowstore data is stored in memory but we keep a snapshot of the data on disk. We also keep the transaction log (a record of all changes since the snapshot was taken) also on disk. So queries can be satisfied fully from memory (because that is where the current data lives) but writes go to memory and to the transaction log on disk. If the machine reboots then the snapshot is loaded from disk back into memory and the transaction log is replayed. When that is complete you are back to where you were when the machine rebooted with no loss of committed data. Columnstore data is always stored on disk although we use a row store in front of the column store that is hidden from the user but acts as a buffer of sorts so that writes in the column store can be pretty fast. More details on how the columnstore works can be found here: https://docs.memsql.com/concepts/v6.7/columnstore/#how-the-m...

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#62

Post by manigandham »

shin_lao wrote: post #54
hendzen wrote: post #46

Earlier quoted context omitted.

No real HFT system is doing a * database transaction * in the critical trading path. HFT systems are not built like web applications. They are typically built as a tight event loop, reading market data packets directly from the network card, doing a tiny bit of computation and then writing to a userspace TCP stack for order entry. I guess you could be using MemSQL for post-order or trade analysis but then it would pr…

Additionally 128 GB is just about 20 minutes worth of data.

That's just the free edition RAM limit...

But yes, actual HFT is not an accurate use-case, for any database product.

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#63

Post by polskibus »

ruw1090 wrote: post #53
polskibus wrote: post #26

Earlier quoted context omitted.

What are the specific operations? Will another batch of INSERTs / UPDATEs work? Does it matter if I use the analytical part of MemSQL ?

I mean that an operation that needs more memory than is available will fail, but the server itself will remain operational. An simple example would be loading more data into an in-memory table (row store) than there is available memory. If you're using the columnstore then your storage won't be limited to memory.

That sucks, it's similar to Hekaton though. I was hoping there was some kind of a fallback mode when the DB has to juggle memory buffers at a performance cost.

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#64

Post by elvinyung »

manigandham wrote: post #44
elvinyung wrote: post #35

How is MemSQL for OLTP nowadays? Has anyone had any success consistently/scalably using it for true HTAP-ish use cases?

It's always been good at OLTP and great at OLAP. We used it for years as a single HTAP solution during the 5.x versions for heavy adtech applications.

Thanks! Just curious, do you use the same tables for both OLTP and OLAP, or do you have some kind of pseudo-ETL process to transform between OLTP and OLAP schemas?

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#65

Post by zedpm »

hanson5b wrote: post #37
starik36 wrote: post #28

Earlier quoted context omitted.

It says that he 128GB version comes with High Availability features. Does that mean if I have 2 nodes, each one is limited to 64GB? Or each one can have 128?

A typical setup for HA would be 2 aggregator nodes and 2 leaf nodes. You can allocate the memory however you want. For example, you could give 32GB to all 4. Or give 16GB to each aggregator and 48GB to each leaf. (I'm a MemSQL Product Manager by the way.)

Do you have recommendations on provisioning memory in aggregators vs leaves? That is, would one of those configurations make for sense than the other for typical workloads?

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#67

Post by dangoor »

noir_lord wrote: post #24
nikita wrote: post #14

Earlier quoted context omitted.

MemSQL CEO here. There are a few: - MemSQL is transactional and writes transactions on disk - MemSQL has an excellent implementation of SQL with mature query optimization and query execution. And it get better every release. This is from 6.5 https://www.memsql.com/blog/6.5-performance/ - MemSQL has in-memory and on-disk data storage so you can use MemSQL to store petabytes - MemSQL has columnstores and vectorized que…

Given the availability of open source solutions why would I in 2018 build a critical part of my application on a closed platform? Genuinely not intended as snark, I'm just curious why memsql is so compelling that I would consider it.

Isn't that what the previous answer was trying to say? Paraphrasing "MemSQL has these features which we think Apache Ignite and others do not."

All things being equal, I agree that open source solutions are the best. Things are just not always equal.

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#68

Post by manigandham »

elvinyung wrote: post #64
manigandham wrote: post #44

Earlier quoted context omitted.

It's always been good at OLTP and great at OLAP. We used it for years as a single HTAP solution during the 5.x versions for heavy adtech applications.

Thanks! Just curious, do you use the same tables for both OLTP and OLAP, or do you have some kind of pseudo-ETL process to transform between OLTP and OLAP schemas?

The schemas are the same. Tables behave the same way logically, and its just performance and physical semantics that are different.

We would have recent data in rowstore and move older data into columnstore. You can easily join/union between both for queries. Also some constantly changing data (like budget counters) would always remain in rowstore with many lookups and updates per-second.

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#70

Post by gacba »

CuriouslyC wrote: post #33
setheron wrote: post #2

is memSQL a replacement for a full RDBMS in your stack ? Or just for analytics -- siphoning off the data from your master.

Personally, I would not use memsql as a first line RDBMS. It lacks too many useful features, and it doesn't return consistent results for all queries (I've found inconsistencies when using group by with having, for instance).

Been using MemSQL for a enterprise-level financial services client since mid-2017. We have this in production and are running it in a multi-TB cluster. We've not seen ANY of these issues here and are heavy, crazy query users. Their support has been nothing but on-the-spot and very helpful.
Post reply on HN