Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Crate.io – Big Data SQL in real time

crate.io

Post reply on HN
Print view View original

51–60 of 83 posts

Re: Crate.io – Big Data SQL in real time

#51

Post by jandrewrogers »

rsvihla wrote: post #44
jandrewrogers wrote: post #41

Earlier quoted context omitted.

There are commercial database solutions focused on IoT that have no problems with these workloads (e.g. my company, SpaceCurve, or Pixia) but nothing open source. A single rack of servers arranged as a parallel system with a 10GbE switch fabric can support it if you design the software correctly. If you look at every company that is working in this space, one of the first things you will notice is that they all use c…

Cassandra is open source and used in a large number of IoT applications. Most famous one being Nest. Also as far as throughput Netflix is doing 1.5 trillion (yes trillion) transactions per day in production on Cassandra.

This is a great example of not understanding the scaling problem of IoT. Not only is the above (quasi-)transaction rate modest by IoT system standards today, but Cassandra is not doing real-time analysis or ad hoc querying of complex relationships across those transactions at the same time, which is usually a requirement.

I know of a production IoT system in the private sector that does 1.5 trillion (quasi-)transactions every 10 minutes, so almost three orders of magnitude higher throughput. Cassandra is an okay choice for storing IoT data but it isn't real-time in the sense that you can do immediate, fast queries about the relationships across those records as they are hitting the system.

Re: Crate.io – Big Data SQL in real time

#52

Post by syntern »

leccine wrote: post #26

Anybody who comes up with a new "Big Data" solution has to realize that the 90% of the data users are vested deeply in Hadoop and the ecosystem around it. Having tools like PrestoDB, Hive, Shark on the top of HDFS makes it really hard to newcomers to convince companies to invest in something else. If data was a greenfield territory these project would be more viable. Btw. on this note, PrestoDB just getting the right…

Maybe in your case they have Hadoop, but not everyone is satisfied with Hadoop or wants to create new systems with it. Choice is good, as long as we can differentiate the choices by the values they provide.

Re: Crate.io – Big Data SQL in real time

#53

Post by xhrpost »

rburhum wrote: post #4

Interesting. Nevertheless, just by browsing the documentation, it seems they only support very basic SQL. For example, joins are not supported. To me, "SQL support" basically entails taking advantage of the relational model. In here, "SQL Support" means "you can use something that looks like a SQL-like syntax to insert and fetch records from a table". Those are two very different things. I do understand why they did…

Not looking to down the work here but it sounds like in using this then, you're basically lowering DB admin complexity at the cost of increasing programming complexity. Does that sound right? I'm kind of torn as to when I would want to go that route. Perhaps on very simple data sets with limited relationships.

Re: Crate.io – Big Data SQL in real time

#54

Post by caioariede »

huxley wrote: post #22

If you were initially confused like I was, crate.io used to be the domain for an alternative Python Package Index. http://www.reddit.com/r/Python/comments/1wcp93/what_happened...

Confusing, but it seems to be moved to https://warehouse.python.org/

Edit: I'm not sure they are related

Re: Crate.io – Big Data SQL in real time

#55

Post by dobe »

hi, this is bernd from crate ... first - we are overwhelmed here at crate that we are mentioned at hacker news! by reading through the comments on this page, i thought it would make sense to give you some background, which should at least partly answer some questions mentioned here.

- we come from the service business and discovered that nearly every database design for applications which needed to scale somewhere reached a point where data needed to be de-normalized because joins where simply too expensive in terms of cost and latency when data does not fit on a single affordable machine. therefore we do not have join support yet. however we already planned to allow joins in the future which still makes sense for smaller datasets of course, but it is currently not a top priority, since many join use-cases could also be implemented by using nested objects which we support.

- we have chosen SQL as a query language, since this allows us to re-use existing ORMs and tools. but most of all SQL is still a great language to define queries, so we thought "why re-invent the wheel and crate yet another query syntax"

- regarding sharding: we use a hash/modulo based sharding mechanisms - actually the same as elasticsearch, since we use elasticsearch under the hood for cluster state, sharding and replication. we also added partitioned table support in our current development branch.

there are still a lot of features on our roadmap; and apparently also a lot of things we need to document and explain in our documentation. so if you are interested in our progress you might keep an eye on our github project page https://github.com/crate/crate

thx, bernd

Re: Crate.io – Big Data SQL in real time

#57

Post by jodok »

asdfologist wrote: post #30

"It includes solid established open source components (Presto, Elasticsearch, Lucene, Netty) and extends those with added core functionalities like read/write support, a SQL interface, a dashboard and a query console." This makes no sense. Presto is already a SQL query engine, so what does it mean to "extend" it with a SQL interface? Furthermore, the crate.io SQL is even more limited than Presto SQL! Presto allows op…

hi, i'm jodok from crate. yes, right now we only use the sql parser (and extended it)

Re: Crate.io – Big Data SQL in real time

#58

Post by jodok »

arjn wrote: post #56

Isn't this very similar to Apache Cassandra ? Perhaps with management-automation included.

cassandra is a super choice for write oriented workloads with really high performance. it's also super stable. we want to be in the spot where you also need advanced realtime search, high numbers of concurrent read/writes. and yes, we focus to make administration as easy as possible

Re: Crate.io – Big Data SQL in real time

#59

Post by jodok »

jandrewrogers wrote: post #16

Nicely done and thoughtfully executed. As a quibble on the suggested use cases, this platform will not work for the Internet of Things generally for two reasons. First, it lacks support for the spatial data types, including polygons, and operations, including spatial joins, that are typical of those types of data models. Second, typical commercial IoT data sources are often on the order of 100TB-1PB per day , which i…

spatial data types are high on our backlog. you'll be able to store geo points, query them by location, bounding box, polygon, sort them by geo distance, and also aggregate them based on geo-distance (e.g. 0-10km, 10-50km,..). we haven't thought about additional topics on top of lucene - but that's our list for a start. feedback, github issues welcome!

Re: Crate.io – Big Data SQL in real time

#60

Post by jodok »

leccine wrote: post #26

Anybody who comes up with a new "Big Data" solution has to realize that the 90% of the data users are vested deeply in Hadoop and the ecosystem around it. Having tools like PrestoDB, Hive, Shark on the top of HDFS makes it really hard to newcomers to convince companies to invest in something else. If data was a greenfield territory these project would be more viable. Btw. on this note, PrestoDB just getting the right…

we here at crate were living in the hadoop ecosystem for some time. but when we came across the beautiful architecture of netty.io (async, event-driven) and the way elasticsearch orchestrated it - since that time we know there will be room for newcomers :)
Post reply on HN