Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is a Dataframe Just a Table? (2019) [pdf]

plateau-workshop.org

Post reply on HN
Print view View original

71–80 of 120 posts

Re: Is a Dataframe Just a Table? (2019) [pdf]

#71

Post by TallGuyShort »

mumblemumble wrote: post #68
js8 wrote: post #6

> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…

OK, so, disclaimer, I haven't ever used GraphQL with live ammunition, but I really don't get the impression that it can be dismissed so easily. TFA does a good job of showing that, while SQL is a cleaner design and theoretically better than the hot mess that is Pandas dataframes in every way, the dataframes API offers a lot of conveniences and ergonomic affordances that can make it more pleasant to program against fo…

>> if you're talking APIs here, you may not be doing too much much hand-authoring of queries

No experience with GraphQL myself, but this is a very good point. A lot of the practical problems that come with SQL queries boil down to the code that constructs a string to be used as the query, so it can then be deconstructed by different code to execute the query. There's a lot of mistakes that happen right there.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#72

Post by mumblemumble »

TallGuyShort wrote: post #71
mumblemumble wrote: post #68

Earlier quoted context omitted.

OK, so, disclaimer, I haven't ever used GraphQL with live ammunition, but I really don't get the impression that it can be dismissed so easily. TFA does a good job of showing that, while SQL is a cleaner design and theoretically better than the hot mess that is Pandas dataframes in every way, the dataframes API offers a lot of conveniences and ergonomic affordances that can make it more pleasant to program against fo…

>> if you're talking APIs here, you may not be doing too much much hand-authoring of queries No experience with GraphQL myself, but this is a very good point. A lot of the practical problems that come with SQL queries boil down to the code that constructs a string to be used as the query, so it can then be deconstructed by different code to execute the query. There's a lot of mistakes that happen right there.

And this is the pain I'm thinking of. Doing dynamically generated SQL that isn't susceptible to SQL injection can get tricky, and ORM frameworks generally only partially solve the problem for a certain subset of possible schemata.

I do think SQL is a well-designed language. But it was designed, first and foremost, as a language for human analysts to bang into a terminal. Having computers flexibly communicate using it is an off-label use.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#74

Post by somurzakov »

kortex wrote: post #55
tanilama wrote: post #21

It is worse than a table. Dataframe don't really have a clear boundary as what it can do or cant do. I see no problem why you can't shoehorn some control flow logic into it which makes it pretty much just an executor of arbitrary computation graph specified in DSL, which relies on however the developer decides to implement it. I'd rather take SQL because I have a better understanding what it is doing.

Can you drop nans, parse strings to floats, apply arbitrary lambdas over a rolling window, take the cosine of those values, and plot a graph against time, in a single line of sane SQL? Easy in Pandas. Dataframes are not tables; tables are not dataframes. It's nearly as apples-to-bananas as comparing Python lists to C arrays.

everything you mentioned can be done easily through database schema. window functions work well in SQL. plots are easily done in any BI solution that hooks up to any database.

pandas is just poor man's SQL+BI. pandas stores everything in memory and has many limitations.

in SQL Server I can easily churn through terabyte sized database and get the data I need, because the schema is well designed with partitioned tables, clustered indexes and a well designed SQL takes less than a second to run against >1TB database. It even allows a team of more than 20 people to work with the same SQL and query it simultaneously.

i would love to see how you can analyze 1TB csv file, or a pile of 100 csv files totalling over 10TB where you have yet to discover the schema and how tables join together. and I am doing it with on a simple $700 workstation, not even using any hadoop nonsense

Re: Is a Dataframe Just a Table? (2019) [pdf]

#75

Post by tveita »

js8 wrote: post #6

> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…

> And yes, GraphQL is a poor reinvention of SQL.

No. GraphQL is an RPC specification. There is not a single comparison operator defined it the GraphQL spec. There is no way to join two separate collections. GraphQL was never meant to be an alternative to SQL, and people mainly try to compare them because they both end in "QL".

Re: Is a Dataframe Just a Table? (2019) [pdf]

#76

Post by rcar »

leto_ii wrote: post #63
bobbylarrybobby wrote: post #62

Earlier quoted context omitted.

Could you give some examples of this issue with pandas?

Some of the examples provided in the paper are eloquent: •df[df.a>3] •df[df["a"]>3] •df.loc[df.a>3] •df.loc[df["a"]>3]

Not sure that I'd consider those all that eloquent since it's just the product of 2 different pieces of syntactic sugar (df.a being shorthand for df["a"] and df[] shorthand for df.loc[]).

Re: Is a Dataframe Just a Table? (2019) [pdf]

#77

Post by rand_r »

tveita wrote: post #75
js8 wrote: post #6

> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…

> And yes, GraphQL is a poor reinvention of SQL. No. GraphQL is an RPC specification. There is not a single comparison operator defined it the GraphQL spec. There is no way to join two separate collections. GraphQL was never meant to be an alternative to SQL, and people mainly try to compare them because they both end in "QL".

The main similarity is the whole idea of declaratively saying what you want in a single request. In SQL, you use joins or subqueries, and in GraphQL you use use nested edge/node blocks.

Either way, you define what you want in a nested/tree-like manner and submit one big-ass request to the server.

The difference is that GraphQL is usually way less verbose and tedious to type out, but they’re fundamentally the same idea.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#78

Post by ezzzzz »

somurzakov wrote: post #74
kortex wrote: post #55

Earlier quoted context omitted.

Can you drop nans, parse strings to floats, apply arbitrary lambdas over a rolling window, take the cosine of those values, and plot a graph against time, in a single line of sane SQL? Easy in Pandas. Dataframes are not tables; tables are not dataframes. It's nearly as apples-to-bananas as comparing Python lists to C arrays.

everything you mentioned can be done easily through database schema. window functions work well in SQL. plots are easily done in any BI solution that hooks up to any database. pandas is just poor man's SQL+BI. pandas stores everything in memory and has many limitations. in SQL Server I can easily churn through terabyte sized database and get the data I need, because the schema is well designed with partitioned tables…

How about working with poorly designed schemas? I work with SQL-Server as well, dealing with legacy data designed around imperative t-sql programming. Our 'BI-Solution', SSRS, crawls on pretty simple queries, where 'hacks' need to be done, joining on same table, all kinds of dirty tricks...

I don't know... I honestly feel like 'BI-Solutions' are a poor-persons Python if you are doing anything more than simple dashboards. Something that can be done in 2 lines of code in a Notebook requires endless fiddling in an IDE, to produce something not easily reproducible.

Aside, I've no experience with Tableau or Power-BI, just know that Crystal Reports and SSRS which are pretty painful.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#79

Post by dang »

MrManatee wrote: post #53
tsbinz wrote: post #5

This doesn't seem to be from 2016 (some identifiers suggest that it is, but it cites papers up to 2018 and it says that a tweet from 2016 is two years old).

Indeed. "Conference on Very Important Topics 2016" is not a real conference, but placeholder from a template. Maybe it was left behind by accident? The paper is from the PLATEAU Workshop 2019.

Ok, we've added three years to the title above. Thanks to both of you!

Re: Is a Dataframe Just a Table? (2019) [pdf]

#80

Post by [deleted] »

somurzakov wrote: post #74
kortex wrote: post #55

Earlier quoted context omitted.

Can you drop nans, parse strings to floats, apply arbitrary lambdas over a rolling window, take the cosine of those values, and plot a graph against time, in a single line of sane SQL? Easy in Pandas. Dataframes are not tables; tables are not dataframes. It's nearly as apples-to-bananas as comparing Python lists to C arrays.

everything you mentioned can be done easily through database schema. window functions work well in SQL. plots are easily done in any BI solution that hooks up to any database. pandas is just poor man's SQL+BI. pandas stores everything in memory and has many limitations. in SQL Server I can easily churn through terabyte sized database and get the data I need, because the schema is well designed with partitioned tables…

[deleted]
Post reply on HN