Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MemSQL is now free to use for databases with up to 128GB of RAM usage

memsql.com

Post reply on HN
Print view View original

81–90 of 121 posts

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#81

Post by [deleted] »

CuriouslyC wrote: post #71
neildahlke wrote: post #36

Earlier quoted context omitted.

I've seen you mention these inconsistent results twice here in this this thread, but have worked at MemSQL for 5 years and never heard of such an issue. Have you reached out to see if maybe your query / data is not what you expect? I've seen inconsistent results only once, and it was because the default date formats across RDMBSs were different (and was not anticipated).

How would the query/data not be what I expect if If I'm writing the query myself, and looking directly at the sql table definition to create it? Beyond those considerations, why would the same exact same query (executed several times in rapid succession from the console) produce vastly different results? Also, I should clarify, rewriting the query from "select ... from xyz group by ... having ..." to "select ... from…

[deleted]

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#82

Post by manigandham »

riku_iki wrote: post #77
manigandham wrote: post #48

Earlier quoted context omitted.

Distributed. Scalable. OLTP + OLAP queries. High availability. Very fast performance for reads and writes. They are entirely different systems. Sqlite is meant for self-contained applications that need some relational data persistence with a single file for storage, not for accessing as a central database with many clients storing TBs and scaling across nodes.

> Distributed is MemSQL shared nothing or shared everything, or can be mix of both?

Shared nothing. It has leaf nodes that store data and do local processing, and aggregator nodes that run queries spread to the leaf nodes and return the results.

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#83

Post by manigandham »

majidazimi wrote: post #75
manigandham wrote: post #29

Earlier quoted context omitted.

Apache Ignite is an in-memory data grid that supports persistence and overflow-to-disk. It primarily started as a cache and now has a full key/value store with SQL-92 on top, but isn't a full relational database. Instead it has other features like distributed data structures, messaging, and is more about connecting your applications together. Easier deployment model with all nodes being identical. MemSQL is a distrib…

Presto on top of Apache Ignite is basically an OLAP database (It just takes some effort to write the connector). It doesn't support transactional workloads though.

"Basically like" is very different from "engineered for"

Column-oriented storage itself is many times faster for analytical queries, even if on disk, and combined with the other optimizations of MemSQL will get you far better performance. Along with all the data being able to constantly undergo transactional updates.

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#84

Post by furicane »

I tried to come up with a smart and polite comment but I can't. The target audience for memsql aren't developers or engineers. It's the management that has no idea about IT. I don't like closed-source solutions. I don't want to book a demo. I want to be able to read the source. I want to install it, use it, benchmark it, be sure that the results are 100% accurate. Sadly, I see this post as marketing ploy and I can't find any nice words for this product.

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#85

Post by manigandham »

CuriouslyC wrote: post #72
gacba wrote: post #70

Earlier quoted context omitted.

Been using MemSQL for a enterprise-level financial services client since mid-2017. We have this in production and are running it in a multi-TB cluster. We've not seen ANY of these issues here and are heavy, crazy query users. Their support has been nothing but on-the-spot and very helpful.

I totally admit that the inconsistency might be a result of misconfiguration by maintainers at my company. That being said, being able to shoot yourself in the foot so subtly and badly via configuration seems like a pretty strong anti-feature.

Since you aren't sharing any actual details and are switching between blaming the product to blaming your devs and config (where no product can magically keep you from breaking settings), your comments come across as rather disingenuous.

Why not share a clear example of exactly what happened, or post on their forum with details, so we can all judge for ourselves?

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#86

Post by manigandham »

furicane wrote: post #84

I tried to come up with a smart and polite comment but I can't. The target audience for memsql aren't developers or engineers. It's the management that has no idea about IT. I don't like closed-source solutions. I don't want to book a demo. I want to be able to read the source. I want to install it, use it, benchmark it, be sure that the results are 100% accurate. Sadly, I see this post as marketing ploy and I can't…

You can install it, use it, benchmark it, and check everything yourself.

It's not open-source, but there is plenty of closed-source proprietary software, and plenty of buyers who care about solving their problems and paying money to get that done (and ensure the vendor stays alive).

If you don't want to use a closed-source product then that's your prerogative, but I don't see how you're making a dev/engineering decision by ignoring a product because of that.

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#87

Post by hermitdev »

PeterisP wrote: post #32
shin_lao wrote: post #12

I don't understand the high frequency use case they describe. High frequency trading is something very different from "12,000 transactions a minute". What is exactly the use case? Pre-deal checks? Post-deal checks? Book replay? Or is it just a simulation? It's not very clear.

HFT can easily be 12000 transactions a minute with a requirement for each of these transactions to be very fast. It's not (necessarily) about high throughput, but about low latency.

I work for a dark pool ATS that is hit by HFT firms, and we routinely see flows greater than 12k transactions per minute. Ive been benchmarking a variety of compilers, db libs, drivers and platforms. So far, best perfomance ive gotten, single threaded, is I can write a single order to a man store table in MSQL in about 500 microsecs (that was from a .net core app running directly on the same server as MS SQL, ive been able to get comparable performance from a C++ app running on Linux with kernel bypass network IO). Mind, ive not tried to optimize the DB at all, this is purely comparing DB APIs. Worst Ive seen, all other things being equal is about 800 micros.

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#88

Post by brd529 »

zedpm wrote: post #65
hanson5b wrote: post #37

Earlier quoted context omitted.

A typical setup for HA would be 2 aggregator nodes and 2 leaf nodes. You can allocate the memory however you want. For example, you could give 32GB to all 4. Or give 16GB to each aggregator and 48GB to each leaf. (I'm a MemSQL Product Manager by the way.)

Do you have recommendations on provisioning memory in aggregators vs leaves? That is, would one of those configurations make for sense than the other for typical workloads?

In general more memory should go to the leaves. All data is stored there and MemSQL will push processing to the leaves whenever possible.

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#89

Post by capkutay »

majidazimi wrote: post #75
manigandham wrote: post #29

Earlier quoted context omitted.

Apache Ignite is an in-memory data grid that supports persistence and overflow-to-disk. It primarily started as a cache and now has a full key/value store with SQL-92 on top, but isn't a full relational database. Instead it has other features like distributed data structures, messaging, and is more about connecting your applications together. Easier deployment model with all nodes being identical. MemSQL is a distrib…

Presto on top of Apache Ignite is basically an OLAP database (It just takes some effort to write the connector). It doesn't support transactional workloads though.

This sounds like one of those things that sounds fun on a whiteboard but will be hell to implement and actually make perform well at all.

Re: MemSQL is now free to use for databases with up to 128GB of RAM usage

#90

Post by nikita »

nikita wrote: post #80
CuriouslyC wrote: post #78

Earlier quoted context omitted.

I really appreciate the offer. I got the go-ahead to share this information, where should I direct it?

Here, or https://www.memsql.com/forum/ , or memsql-public.slack.com.

To close the loop on this one. We looked at the query and strictly speaking we should be rejecting it b/c HAVING clause is referencing a column that's NOT in group by and NOT an aggregate expression. The query shape is:

  select count(*), a from T group by a having b > 0
In this case b is not allowed to be part of having by ANSI standard.

We let it run b/c some customers migrate from MySQL and MySQL allows this query. You can set MemSQL to be strict about it by setting this variable:

  set session sql_mode = only_full_group_by;
Post reply on HN