Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Crate.io – Big Data SQL in real time

crate.io

Post reply on HN
Print view View original

31–40 of 83 posts

Re: Crate.io – Big Data SQL in real time

#32

Post by jandrewrogers »

icambron wrote: post #20
jandrewrogers wrote: post #16

Nicely done and thoughtfully executed. As a quibble on the suggested use cases, this platform will not work for the Internet of Things generally for two reasons. First, it lacks support for the spatial data types, including polygons, and operations, including spatial joins, that are typical of those types of data models. Second, typical commercial IoT data sources are often on the order of 100TB-1PB per day , which i…

> Second, typical commercial IoT data sources are often on the order of 100TB-1PB per day Interesting. Do you have a source on that?

These are the kinds of applications and workloads I have been working on for almost a decade now, spanning many industries, so it is my area of domain expertise. The above numbers are a pretty typical range for the myriad companies and organizations I work with. A small system or pilot project might be 10 TB per day. For some government systems, 1PB per day is a small fraction.

People are just beginning to take advantage of these data sources but table stakes is being able to continuously ingest and index many millions of complex spatial relationships every second, which by itself is something no popular Big Data platform supports.

The (fun!) computer science challenge of it is that the systems have to be unbelievably scalable, but you can't use hash partitioning or range partitioning, and many things you were taught about database engine design turn out to be completely wrong in this context. It is an area ripe for innovation and growth.

Re: Crate.io – Big Data SQL in real time

#33

Post by danpalmer »

Some initial thoughts from looking at the website and documentation...

1. The main reason I want SQL is for relational data. The lack of joins basically makes this a "NoSQL" database in every respect except the query language being something resembling SQL. I'm fairly sure ANSI SQL requires support for joins.

2. It does auto-sharding, but I don't know how. The documentation doesn't specify how the data is sharded, and this is quite important. Range-based sharding, like MongoDB uses by default, is often not what users want (depends on use-case), so if it's that, we need to know. Whatever it is, there are trade-offs with different approaches and that's something users need to take account of.

3. You can't change the shard cluster size after initially sharding. I assume this is a planned feature, but until then, it's probably not ready for production use.

4. You can only shard on the primary key, if you have a primary key.

5. The configuration for the number of replicas is confusing, and appears to not be very configurable.

A brief read makes this seem to be basically MongoDB from its early days, with many of the disadvantages, but some advantages like custom analysers which appear to replace Mongo's map reduce, that can be queried by an SQL-like syntax.

When I saw this, I thought it was going to be a relational database that did auto-sharding and replication. That would have been great. Unfortunately it's not. It might become that a few years down the line, but right now, I'm not inspired by it.

Re: Crate.io – Big Data SQL in real time

#34

Post by cevaris »

asdfologist wrote: post #30

"It includes solid established open source components (Presto, Elasticsearch, Lucene, Netty) and extends those with added core functionalities like read/write support, a SQL interface, a dashboard and a query console." This makes no sense. Presto is already a SQL query engine, so what does it mean to "extend" it with a SQL interface? Furthermore, the crate.io SQL is even more limited than Presto SQL! Presto allows op…

It probably uses Presto, just restricts the more complex functionality for now. Apparently they have some code here (http://goo.gl/3sgsA4) which is ready for joins.

Re: Crate.io – Big Data SQL in real time

#35

Post by rburhum »

danpalmer wrote: post #33

Some initial thoughts from looking at the website and documentation... 1. The main reason I want SQL is for relational data. The lack of joins basically makes this a "NoSQL" database in every respect except the query language being something resembling SQL. I'm fairly sure ANSI SQL requires support for joins. 2. It does auto-sharding, but I don't know how. The documentation doesn't specify how the data is sharded, an…

> I'm fairly sure ANSI SQL requires support for joins

Yup. The older SQL-92 spec BNF definition for SELECT clauses:

http://savage.net.au/SQL/sql-92.bnf.html#query%20specificati...

Re: Crate.io – Big Data SQL in real time

#39

Post by icambron »

jandrewrogers wrote: post #32
icambron wrote: post #20

Earlier quoted context omitted.

> Second, typical commercial IoT data sources are often on the order of 100TB-1PB per day Interesting. Do you have a source on that?

These are the kinds of applications and workloads I have been working on for almost a decade now, spanning many industries, so it is my area of domain expertise. The above numbers are a pretty typical range for the myriad companies and organizations I work with. A small system or pilot project might be 10 TB per day. For some government systems, 1PB per day is a small fraction. People are just beginning to take advan…

I was hoping you could give a source not because I was questioning the veracity of your statement, but because I wanted to learn more about it, specifically what kind of data that would be and what you do with it. I bet there are a lot of interesting challenges in that space.
Post reply on HN