Eric, one of the cofounders, here. happy to answer any questions on MemSQL 4 and the community edition. Some new features in MemSQL 4: - fully distributed joins - native geospatial index and datatypes - lots of new SQL surface area - concurrency improvements - analytic optimizer - Spark, HDFS, and S3 connectors
What is the replication picture for the community version? I can see that Enterprise has HA features but I have to guess that there is some form of safety if one node goes down in Community.
You mean like Oracle with MySQL? At least in that case the 'community' could move to MariaDB, which is not an option for non-Free databases like MemSQL.
you can do the very same - MemSQL uses the MySQL-wire protocol so it works with any MySQL driver and tool.
If MySQL was a drop-in replacement for MemSQL, you wouldn't need MemSQL in the first place. The reason you chose MemSQL is probably because it offers something that MySQL doesn't. If I can't take the source and continue using the product, it's a very different situation from MySQL/Oracle.
Interesting. We've implemented a metadata layer for HDFS and YARN using NDB (MySQL Cluster) - that also supports READ COMMITTED transactions. Do you support: - row-level locking - independent transaction coordinators at data nodes - pruned index scans - network-aware transactions (with user-defined partition keys for tables) - any asynchronous/event API ?
- row-level locking -> yes we use MVCC and take a row level write lock when necessary for consistency - independent transaction coordinators at data nodes -> we have a tier called "aggregators" that act as transaction coordinators. These are the nodes you connect to. Under the hood leaf nodes in memsql also manage transactions. - pruned index scans -> Do you mean information retrieval? Our indexes support seeks and r…
Great. Lots of good stuff there.
Pruned index scans are index scans where the data is located on a single shard and the index scan doesn't flood all nodes in the DB.
I'll definitely be looking into MemSQL.
Interesting. We've implemented a metadata layer for HDFS and YARN using NDB (MySQL Cluster) - that also supports READ COMMITTED transactions. Do you support: - row-level locking - independent transaction coordinators at data nodes - pruned index scans - network-aware transactions (with user-defined partition keys for tables) - any asynchronous/event API ?
MemSQL partitions data across nodes by hash, not by range, so partition prunning is less applicable. However, in a case when it can be applied MemSQL does apply it. [1] Within each node, for column store tables in MemSQL we do use segment elimination very aggressively, which is effectively the same thing as partition pruning. [2] [3] [1] http://docs.memsql.com/latest/concepts/distributed_sql/#inde... [2] http://docs.…
Shard key matching is effectively partitioned pruning - which is great. This is a feature not many people are aware of, but is super important when scaling to large clusters and when you have "session-oriented" (or in our case inode-oriented) data spread across different tables.
Could you please elaborate? Do you mean approximate joins as in this talk? http://www2.research.att.com/~divesh/papers/ks2005-aj-tutori...
No no, sorry, it's much simpler (at least in how it works, no guarantees on implementation complexity, of course). It's an issue that comes up in timeseries databases pretty often. Say I have a table full of quotes and a table full of trades. I want to know what the quote price was at the time the trade occurred. In no-frills SQL, that translates into something like: select * from t left outer join q on q.time=(selec…
If you just use the no-frills SQL query, we're able to optimize it to do a fast index seek (instead of scan) on q.time, because we know we only have to get the max row. This optimization isn't specific to proximate joins, a simple query like:
select max(a) from t where a < 42;
will be optimized by memsql.