190k records is Big Data? For me big data starts from 9M+ rows
Big data starts at N+1 rows, where N is the number of rows the person your responding to manages.
Crate.io – Big Data SQL in real time
61–70 of 83 posts
Re: Crate.io – Big Data SQL in real time
#62Anybody who comes up with a new "Big Data" solution has to realize that the 90% of the data users are vested deeply in Hadoop and the ecosystem around it. Having tools like PrestoDB, Hive, Shark on the top of HDFS makes it really hard to newcomers to convince companies to invest in something else. If data was a greenfield territory these project would be more viable. Btw. on this note, PrestoDB just getting the right…
we here at crate were living in the hadoop ecosystem for some time. but when we came across the beautiful architecture of netty.io (async, event-driven) and the way elasticsearch orchestrated it - since that time we know there will be room for newcomers :)
Well sorry my buzzword filter removed half of your sentence. :) I understand that engineers can get excited about async and event driven but these mean absolutely nothing to your endusers. I could implement your service with blocking IO and non-event driven code and still get the same performance. It simply does not matter that much. The key to Hadoop's success is scalability and predictable performance. Don't get me wrong, I think Hadoop is one of the worst ecosystems I have ever seen in my entire life, the code quality makes me cry sometimes, but putting all these aside, projects like PrestoDB making Hadoop viable and keeping it alive in the long run. Your project looks interesting but without extensive performance testing and proving that your TCO is lower than Hadoop's, and having the same features as Hive, PrestoDB does will be really hard to break in to this market. Again, I would be the happiest person to see something more sane than Hadoop on the market.
Re: Crate.io – Big Data SQL in real time
#63Some initial thoughts from looking at the website and documentation... 1. The main reason I want SQL is for relational data. The lack of joins basically makes this a "NoSQL" database in every respect except the query language being something resembling SQL. I'm fairly sure ANSI SQL requires support for joins. 2. It does auto-sharding, but I don't know how. The documentation doesn't specify how the data is sharded, an…
its actually sql on top of elasticsearch
That is bad.
I've done a lot of work with Elasticsearch[1] and while it's a great search engine, it is NOT a primary source of truth or something you want to trust not to lose your data.
[1]: Built a startup's product on top of it and have written an open source client library for it.
Re: Crate.io – Big Data SQL in real time
#64Some initial thoughts from looking at the website and documentation... 1. The main reason I want SQL is for relational data. The lack of joins basically makes this a "NoSQL" database in every respect except the query language being something resembling SQL. I'm fairly sure ANSI SQL requires support for joins. 2. It does auto-sharding, but I don't know how. The documentation doesn't specify how the data is sharded, an…
thanks for your summary and valuable feedback. a few notes from the crate team:
1. JOINs are on our roadmap. we might never reach full JOIN support anytime soon (INNER, LEFT OUTER, RIGHT OUTER, FULL OUTER and CROSS), but simple use-cases (1:n), equal joins shouldn't be too hard.
2. after re-reading our documentation on sharding i need to admit that we need to improve it. we mention it briefly here https://crate.io/docs/current/sql/ddl.html#sharding, more will be added asap
3. if we have growing datasets we typically work with parted tables. we just pushed that change just a few days ago (https://github.com/crate/crate/blob/master/docs/sql/partitio...), not yet in the RPM build.
4. no, you can specify routing https://crate.io/docs/current/sql/ddl.html#routing
5. agreed, that the doc can be improved. and we're at the very beginning https://crate.io/docs/current/sql/ddl.html#replication - rack awareness,... there are quite some replica settings on our roadmap. i'd be happy to hear your most important additions to replica policies.
i promise, we're working hard and try to be there faster than in a few years :)
Re: Crate.io – Big Data SQL in real time
#65Earlier quoted context omitted.
Cassandra is open source and used in a large number of IoT applications. Most famous one being Nest. Also as far as throughput Netflix is doing 1.5 trillion (yes trillion) transactions per day in production on Cassandra.
This is a great example of not understanding the scaling problem of IoT. Not only is the above (quasi-)transaction rate modest by IoT system standards today, but Cassandra is not doing real-time analysis or ad hoc querying of complex relationships across those transactions at the same time, which is usually a requirement. I know of a production IoT system in the private sector that does 1.5 trillion (quasi-)transacti…
"Cassandra [...] isn't real-time in the sense that you can do immediate, fast queries about the relationships across those records as they are hitting the system."
That depends on how you are using Cassandra. Typically, you are expected to know your query patterns up front, and so you will lay your data out accordingly when ingesting. When done properly, this allows for ~1ms queries that return completely up-to-date results.
Re: Crate.io – Big Data SQL in real time
#66I don't understand how this can be called "Big Data SQL in real time" when that is not even their selling point. Their selling point is the scalability.
Their demo is only working against ~190,000 records. I don't now any databases that aren't going to perform the demonstrated operations quickly.
Re: Crate.io – Big Data SQL in real time
#67Any idea what app or library they used to make the retro green-screen terminal video?
As a bonus, I noticed they just released the iOS version in February. Pretty cool.
Re: Crate.io – Big Data SQL in real time
#68Earlier quoted context omitted.
we here at crate were living in the hadoop ecosystem for some time. but when we came across the beautiful architecture of netty.io (async, event-driven) and the way elasticsearch orchestrated it - since that time we know there will be room for newcomers :)
Well sorry my buzzword filter removed half of your sentence. :) I understand that engineers can get excited about async and event driven but these mean absolutely nothing to your endusers. I could implement your service with blocking IO and non-event driven code and still get the same performance. It simply does not matter that much. The key to Hadoop's success is scalability and predictable performance. Don't get me…
I like Hadoop tonnes so I would love to get an injection of reality and perspective: what are the insanities of Hadoop?
Re: Crate.io – Big Data SQL in real time
#69Earlier quoted context omitted.
its actually sql on top of elasticsearch
That is bad . I've done a lot of work with Elasticsearch[1] and while it's a great search engine, it is NOT a primary source of truth or something you want to trust not to lose your data. [1]: Built a startup's product on top of it and have written an open source client library for it.
why ?
at least i remember they didn't recommend it for database until the backup funcionality was done
Re: Crate.io – Big Data SQL in real time
#70If you were initially confused like I was, crate.io used to be the domain for an alternative Python Package Index. http://www.reddit.com/r/Python/comments/1wcp93/what_happened...
Confusing, but it seems to be moved to https://warehouse.python.org/ Edit: I'm not sure they are related
Warehouse is the re-reimplementation, also done by the same guy, which will eventually become the new pypi.python.org