Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Drill 1.0 – Schema-Free SQL Query Engine for Hadoop and NoSQL

drill.apache.org

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Apache Drill 1.0 – Schema-Free SQL Query Engine for Hadoop and NoSQL

#21

Post by vicenteg »

tdicola wrote: post #14

Wow this sounds pretty interesting. Is there any python library to use it in ipython/jupyter, etc. yet? I love pandas and the python data 'stack' but sometimes its syntax for joining and querying data is just extremely obtuse and odd. Many times I finding myself wishing I could query stuff like a SQL table.

Check this out: https://github.com/cjmatta/drill_ipython_notebook

Uses the Drill ODBC driver via pyodbc and includes some visualization through Pandas and matplotlib.

Re: Apache Drill 1.0 – Schema-Free SQL Query Engine for Hadoop and NoSQL

#22

Post by tdicola »

vicenteg wrote: post #21
tdicola wrote: post #14

Wow this sounds pretty interesting. Is there any python library to use it in ipython/jupyter, etc. yet? I love pandas and the python data 'stack' but sometimes its syntax for joining and querying data is just extremely obtuse and odd. Many times I finding myself wishing I could query stuff like a SQL table.

Check this out: https://github.com/cjmatta/drill_ipython_notebook Uses the Drill ODBC driver via pyodbc and includes some visualization through Pandas and matplotlib.

Very cool, thanks!

Re: Apache Drill 1.0 – Schema-Free SQL Query Engine for Hadoop and NoSQL

#23

Post by eranation »

How does it compare with spark SQL? Does the difference lie in where the query runs? In spark it loads all data into memory first and then uses SQL to query it (it parses the query and converts it to a set of spark transformations, kind of like hive). Does Drill actually translate SQL into, for example, mongo db native queries and run it on the source data store? If so this is a whooping amazing feat...

Re: Apache Drill 1.0 – Schema-Free SQL Query Engine for Hadoop and NoSQL

#24

Post by hurin »

eranation wrote: post #23

How does it compare with spark SQL? Does the difference lie in where the query runs? In spark it loads all data into memory first and then uses SQL to query it (it parses the query and converts it to a set of spark transformations, kind of like hive). Does Drill actually translate SQL into, for example, mongo db native queries and run it on the source data store? If so this is a whooping amazing feat...

How hard is it to translate syntaxes?

p.s. Will translate syntaxes, $250 per.

Re: Apache Drill 1.0 – Schema-Free SQL Query Engine for Hadoop and NoSQL

#25

Post by rxin »

eranation wrote: post #23

How does it compare with spark SQL? Does the difference lie in where the query runs? In spark it loads all data into memory first and then uses SQL to query it (it parses the query and converts it to a set of spark transformations, kind of like hive). Does Drill actually translate SQL into, for example, mongo db native queries and run it on the source data store? If so this is a whooping amazing feat...

Actually Spark SQL doesn't load everything into memory. Its data source API supports pushing predicates down, and if the data sources implement it, it even supports running aggregations and joins in the data sources!

Re: Apache Drill 1.0 – Schema-Free SQL Query Engine for Hadoop and NoSQL

#26

Post by ShirsenduK »

This will help save thousands of hours of developer time querying different sources and generating reports for biz analytics teams of startups who know only SQL. Kudos to the team and the project! The lonely junior developer firing queries late at night thanks you. 😊

Re: Apache Drill 1.0 – Schema-Free SQL Query Engine for Hadoop and NoSQL

#27

Post by IndianAstronaut »

polskibus wrote: post #5

How does Apache Drill compare to Power Pivot/ Power Query? As far as I understand, both aim to solve the problem if heterogeneous data sources. What I don't understand is how they differ.

I'm not sure power pivot deals well with heterogeneous data sources. It rests on OLAP services which rely on a fixed star schemas in a data warehouse. Power pivot is just a front end for MDX(multi-dimensional expression) queries.

Re: Apache Drill 1.0 – Schema-Free SQL Query Engine for Hadoop and NoSQL

#28

Post by m_mueller »

vicenteg wrote: post #20
m_mueller wrote: post #19

Sounds very interesting. Is there anything like this for CouchDB? I like couch because it's easily deployed on mobile as well.

There is a couchbase plugin for Drill: https://github.com/jacques-n/drill-couchbase-plugin

Thanks, very interesting!

Re: Apache Drill 1.0 – Schema-Free SQL Query Engine for Hadoop and NoSQL

#30

Post by MasterScrat »

eranation wrote: post #23

How does it compare with spark SQL? Does the difference lie in where the query runs? In spark it loads all data into memory first and then uses SQL to query it (it parses the query and converts it to a set of spark transformations, kind of like hive). Does Drill actually translate SQL into, for example, mongo db native queries and run it on the source data store? If so this is a whooping amazing feat...

I would really be interested in a comparison of BI tools performances using the JDBC/ODBC drivers from Spark vs Drill.
Post reply on HN