Elastic Search is known to drop your data in some occasions. I think it's fair to advise against using it as your source of truth (or anything built on top of it like Crate).
Crate: Distributed SQL Database for the Age of Docker
11–20 of 45 posts
Re: Crate: Distributed SQL Database for the Age of Docker
#12Elastic Search is known to drop your data in some occasions. I think it's fair to advise against using it as your source of truth (or anything built on top of it like Crate).
the elastic team is working hard in fixing the issues discovered by aphyr (jepsen test). you can find the current status here: http://www.elastic.co/guide/en/elasticsearch/resiliency/curr... - we're doing our best to support them. (btw. here's what aphyr thinks about that document: https://twitter.com/aphyr/status/525712547911974913)
Re: Crate: Distributed SQL Database for the Age of Docker
#13I spent a bunch of time reading the website and I can't tell what this product does: Is it a database engine or an abstraction on top of other products? If it's a database engine, then here are my thoughts: How is this database engine built to replace SQL and No-SQL? If it doesn't support JOINS why would I replace my SQL with it? Are transactions ACID? Why would anyone use this if there are no built-in user / group s…
Seems like it's an layer over ElasticSearch using Presto SQL parser. SQL queries are translated into ES queries.
arthur, that's how we started two years ago. meanwhile the queries are being handled by our own engine. https://github.com/crate/crate
Re: Crate: Distributed SQL Database for the Age of Docker
#14I spent a bunch of time reading the website and I can't tell what this product does: Is it a database engine or an abstraction on top of other products? If it's a database engine, then here are my thoughts: How is this database engine built to replace SQL and No-SQL? If it doesn't support JOINS why would I replace my SQL with it? Are transactions ACID? Why would anyone use this if there are no built-in user / group s…
Seems like it's an layer over ElasticSearch using Presto SQL parser. SQL queries are translated into ES queries.
That's part of it, but not the whole picture. For example, we mostly bypass the ES query engine and go directly to Lucene. Queries are not simply translated to ES query syntax. Also, we've done a lot more work than simply pasting an SQL layer over the top. We've built streaming BLOB support, a distributed SQL layer with real-time MapReduce, and a distributed aggregation engine that gives accurate results for aggregations rather than HLL estimates.
If you'd like we're happy to answer any questions in IRC or our Google Group: https://groups.google.com/forum/#!forum/crateio IRC Freenode #crate: irc://irc.freenode.net/crate @mention anyone with Voice
Re: Crate: Distributed SQL Database for the Age of Docker
#15Elastic Search is known to drop your data in some occasions. I think it's fair to advise against using it as your source of truth (or anything built on top of it like Crate).
Yes, and while there are still cases (see the link @jodok supplied below) a ton of work was done in ES v1.4.x (which Crate uses) and even more issues were closed in ES v1.5 which we're working on supporting.
With the amount of attention and support this has received from both Crate and ES, coupled with the amount of progress that's been made against these issues, I don't think it's a fair to advise against using either as a source of truth.
Re: Crate: Distributed SQL Database for the Age of Docker
#16Earlier quoted context omitted.
Hi, Spanky from Crate.io here. We still maintain the PyPI portion of the site at https://crate.io/packages As for crates.io, I guess their entitled to choose their own name & domain, but I admit it is confusing.
Might end up being a source of confusion for you guys, as the package manager for a major language will have much more mindshare than a database.
we're good friends with the original owner "dstufft" and took over the domain after he took over an official pypi position (more than a year ago). read his own blog post: https://caremad.io/2014/03/crate-io-new-ownership/
Re: Crate: Distributed SQL Database for the Age of Docker
#17So the website mentions that a Crate cluster is a share-nothing system; That each node can act independently. This would mean that the data is limited to one machines storage size? Doesn't really fit in with the claim of a distributed database unless I'm missing something.
Re: Crate: Distributed SQL Database for the Age of Docker
#18Where does it keep its data? Docker volumes?
Re: Crate: Distributed SQL Database for the Age of Docker
#19Is just SQL query syntax supported?
The beauty of the Elasticsearch query syntax is that you can dynamically create complex JSON dsl objects as you drill down just using push and other methods.
With the crate SQL syntax it looks like it would be a messier dynamic query generation using string functions?
So my question is: Is it possible to query Crate with Elasticsearch syntax?
Re: Crate: Distributed SQL Database for the Age of Docker
#20So the website mentions that a Crate cluster is a share-nothing system; That each node can act independently. This would mean that the data is limited to one machines storage size? Doesn't really fit in with the claim of a distributed database unless I'm missing something.
No, it can hold as much data as the sum of all the nodes in the cluster have (also depending on the amount of replicas you configure).
Some background: All data is being sharded by default. These shards are distributed evenly between the nodes. Nodes can hold multiple shards. Furthermore shards can have replicas, being hold by other nodes.
Memory is being utilized to hold indizes in RAM and to cache values that are being needed frequently for e.g. aggregations ("field caches").