Live data from Hacker News

Crate.io – Big Data SQL in real time

crate.io

71–80 of 83 posts

Re: Crate.io – Big Data SQL in real time

#71
post #23

It would be helpful to know the largest data set this has been used with and the performance of some example queries. Ideally > 1TB to qualify as "big data".

we've worked with upt to 25bn records (small size) in crate, that was about 3TB (without replicas). if you have larger records or blobs, you're soon in the 10th of TB.

Re: Crate.io – Big Data SQL in real time

#72
post #22

If you were initially confused like I was, crate.io used to be the domain for an alternative Python Package Index. http://www.reddit.com/r/Python/comments/1wcp93/what_happened...

Confusing, but it seems to be moved to https://warehouse.python.org/ Edit: I'm not sure they are related

we're unrelated. but we keep some redirects and subdomains from the "old" crate.io. see https://crate.io/thanks-dstufft.

Re: Crate.io – Big Data SQL in real time

#73
post #22

If you were initially confused like I was, crate.io used to be the domain for an alternative Python Package Index. http://www.reddit.com/r/Python/comments/1wcp93/what_happened...

It still is: https://simple.crate.io/ :-)

we took crate.io over in december. some people still haven't updated their links. that's why we keep some subdomains and redirects up and running (https://crate.io/thanks-dstufft)

Re: Crate.io – Big Data SQL in real time

#74
post #62
post #60

Earlier quoted context omitted.

we here at crate were living in the hadoop ecosystem for some time. but when we came across the beautiful architecture of netty.io (async, event-driven) and the way elasticsearch orchestrated it - since that time we know there will be room for newcomers :)

Well sorry my buzzword filter removed half of your sentence. :) I understand that engineers can get excited about async and event driven but these mean absolutely nothing to your endusers. I could implement your service with blocking IO and non-event driven code and still get the same performance. It simply does not matter that much. The key to Hadoop's success is scalability and predictable performance. Don't get me…

sorry for my ignorance, but Crate and "SQL on top of Hadoop" fulfil quite different needs. How would you handle tens of thousands concurrent queries on a Hadoop platform? Read and write at the same time? On the other hand side - crate will never be able to batch-oriented, complex map/reduce workload.

Re: Crate.io – Big Data SQL in real time

#75

Just a minor detail, the video show the load of the table with 190K rows, but then the distinct shows 194K results.. that is weird, because the table was empty before the load. :P

bummer! good catch. video was taken before this bugfix: https://github.com/crate/crate/blob/master/CHANGES.txt#L160 we'll re-captchure it: https://twitter.com/burndee/status/457256576345448449

Re: Crate.io – Big Data SQL in real time

#79

Earlier quoted context omitted.

That is bad . I've done a lot of work with Elasticsearch[1] and while it's a great search engine, it is NOT a primary source of truth or something you want to trust not to lose your data. [1]: Built a startup's product on top of it and have written an open source client library for it.

why ? at least i remember they didn't recommend it for database until the backup funcionality was done

Most people aren't good enough at backups for me to consider that enough to mitigate the risks.

The data needs to go somewhere other than ES before you considered it "saved".

Re: Crate.io – Big Data SQL in real time

#80
post #64

Some initial thoughts from looking at the website and documentation... 1. The main reason I want SQL is for relational data. The lack of joins basically makes this a "NoSQL" database in every respect except the query language being something resembling SQL. I'm fairly sure ANSI SQL requires support for joins. 2. It does auto-sharding, but I don't know how. The documentation doesn't specify how the data is sharded, an…

thanks for your summary and valuable feedback. a few notes from the crate team: 1. JOINs are on our roadmap. we might never reach full JOIN support anytime soon (INNER, LEFT OUTER, RIGHT OUTER, FULL OUTER and CROSS), but simple use-cases (1:n), equal joins shouldn't be too hard. 2. after re-reading our documentation on sharding i need to admit that we need to improve it. we mention it briefly here https://crate.io/do…

Even if you can get basic JOINs going, then I can see your tool becoming super useful in a lot of my use cases :)
Post reply on HN