Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Crate: Distributed SQL Database for the Age of Docker

crate.io

Post reply on HN
Print view View original

21–30 of 45 posts

Re: Crate: Distributed SQL Database for the Age of Docker

#21

Post by jodok »

naiv wrote: post #19

Is just SQL query syntax supported? The beauty of the Elasticsearch query syntax is that you can dynamically create complex JSON dsl objects as you drill down just using push and other methods. With the crate SQL syntax it looks like it would be a messier dynamic query generation using string functions? So my question is: Is it possible to query Crate with Elasticsearch syntax?

yes, you can enable the elasticsearch API ("es.api.enabled: true" in the crate.yml config file). however, it's not officially supported and we recommend to use it read-only as we store additional information about tables,...

Re: Crate: Distributed SQL Database for the Age of Docker

#22

Post by jodok »

cephaslr wrote: post #18

Where does it keep its data? Docker volumes?

we recommend to expose a host directory to crate ('docker run -d -p 4200:4200 -p 4300:4300 -v :/data crate') and configure replicas. if one of the crate containers disappears, replicas will be promoted as primary shard and new replicas created on the fly. it's also possible to expose multiple directories (e.g. on multiple disks for more performance), you can configure crate to use them in parallel.

Re: Crate: Distributed SQL Database for the Age of Docker

#23

Post by spanktar »

prawnsalad wrote: post #17

So the website mentions that a Crate cluster is a share-nothing system; That each node can act independently. This would mean that the data is limited to one machines storage size? Doesn't really fit in with the claim of a distributed database unless I'm missing something.

By shared-nothing, we mean that no node is special, no node is indispensable. Unlike other popular NoSQL databases, we don't have different classes of nodes. The persistent data is distributed across the cluster, and nodes communicate amongst each other about which data are where.

http://en.wikipedia.org/wiki/Shared_nothing_architecture

Re: Crate: Distributed SQL Database for the Age of Docker

#24

Post by spanktar »

cephaslr wrote: post #18

Where does it keep its data? Docker volumes?

Like Jodok says, we don't recommend this, but you could do it. What this would mean that if you destroy a container, you also destroy the data. This would cause your cluster to have to rebalance itself adding IOPs overhead.

When you map a volume into a container as suggested, the data can persist through a container restart/replacement. When the container is instantiated, the volume is read, the node checksums the shards it finds to make sure they're not stale. If so, they're brought up to date. By tuning the recovery settings you can avoid extraneous shard movement and therefore leverage containers as you would expect.

Re: Crate: Distributed SQL Database for the Age of Docker

#25

Post by ddorian43 »

spanktar wrote: post #14
arthursilva wrote: post #10

Earlier quoted context omitted.

Seems like it's an layer over ElasticSearch using Presto SQL parser. SQL queries are translated into ES queries.

That's part of it, but not the whole picture. For example, we mostly bypass the ES query engine and go directly to Lucene. Queries are not simply translated to ES query syntax. Also, we've done a lot more work than simply pasting an SQL layer over the top. We've built streaming BLOB support, a distributed SQL layer with real-time MapReduce, and a distributed aggregation engine that gives accurate results for aggregat…

Why can't you aggregate on non-indexed fields? I know lucene doesn't allow that, but why? It seems to work on normal-rdbms ?

Re: Crate: Distributed SQL Database for the Age of Docker

#26

Post by naiv »

jodok wrote: post #21
naiv wrote: post #19

Is just SQL query syntax supported? The beauty of the Elasticsearch query syntax is that you can dynamically create complex JSON dsl objects as you drill down just using push and other methods. With the crate SQL syntax it looks like it would be a messier dynamic query generation using string functions? So my question is: Is it possible to query Crate with Elasticsearch syntax?

yes, you can enable the elasticsearch API ("es.api.enabled: true" in the crate.yml config file). however, it's not officially supported and we recommend to use it read-only as we store additional information about tables,...

Read-Only should be fine.

Is there specific "Crate" query syntax for selects that is not supported by the Elasticsearch DSL?

Re: Crate: Distributed SQL Database for the Age of Docker

#27

Post by naiv »

spanktar wrote: post #14
arthursilva wrote: post #10

Earlier quoted context omitted.

Seems like it's an layer over ElasticSearch using Presto SQL parser. SQL queries are translated into ES queries.

That's part of it, but not the whole picture. For example, we mostly bypass the ES query engine and go directly to Lucene. Queries are not simply translated to ES query syntax. Also, we've done a lot more work than simply pasting an SQL layer over the top. We've built streaming BLOB support, a distributed SQL layer with real-time MapReduce, and a distributed aggregation engine that gives accurate results for aggregat…

do you have a field type that indexes in real time? or are you bound to the (default 1s) index delay from es?

this is one thing that bothers me with elasticsearch, that I can not define eg "type": "cart","index":"realtime", "not-analyzed" so if an item gets added to a cart, the subsequent count would directly return the correct number of items in the cart.

Re: Crate: Distributed SQL Database for the Age of Docker

#29

Post by fizx »

spanktar wrote: post #23
prawnsalad wrote: post #17

So the website mentions that a Crate cluster is a share-nothing system; That each node can act independently. This would mean that the data is limited to one machines storage size? Doesn't really fit in with the claim of a distributed database unless I'm missing something.

By shared-nothing, we mean that no node is special, no node is indispensable. Unlike other popular NoSQL databases, we don't have different classes of nodes. The persistent data is distributed across the cluster, and nodes communicate amongst each other about which data are where. http://en.wikipedia.org/wiki/Shared_nothing_architecture

Given that this is based on Elasticsearch, there will be a master node, which should be chosen explicitly in configuration to be away from the data nodes, because ES responds poorly to garbage collection on the master node. So you want to keep the load down on your master.

So you can run in a sort-of-shared-nothing configuration, but its not recommended.

Re: Crate: Distributed SQL Database for the Age of Docker

#30

Post by fizx »

spanktar wrote: post #15
arthursilva wrote: post #11

Elastic Search is known to drop your data in some occasions. I think it's fair to advise against using it as your source of truth (or anything built on top of it like Crate).

Yes, and while there are still cases (see the link @jodok supplied below) a ton of work was done in ES v1.4.x (which Crate uses) and even more issues were closed in ES v1.5 which we're working on supporting. With the amount of attention and support this has received from both Crate and ES, coupled with the amount of progress that's been made against these issues, I don't think it's a fair to advise against using eith…

The Lucene team has been promising an end to corruption since the 2.9 days and before (~10 years). If you believe them that the problems are all fixed, and you need to be serious about it, I'd wait more than a month before declaring the problem solved.

That said, they are checksumming a hell of a lot more than before, so there's a chance.

Post reply on HN