Live data from Hacker News

Is a Dataframe Just a Table? (2019) [pdf]

plateau-workshop.org

91–100 of 120 posts

Re: Is a Dataframe Just a Table? (2019) [pdf]

#91
post #86
post #76

Earlier quoted context omitted.

Not sure that I'd consider those all that eloquent since it's just the product of 2 different pieces of syntactic sugar (df.a being shorthand for df["a"] and df[ ] shorthand for df.loc[ ]).

Well, that's kind of the point. What's the purpose of the syntactic sugar? Is it just that, or is there some hidden performance difference? This is not clear at first sight.

The point is to "Huffman encode" the API for expressing near-boilerplate. Like unix command names and flags.

The problem is that there is no simple logically coherent API to use when you haven't memorized all the shortcuts. And the author only allows "tax form" APIs (what he calls "Pythobic/Pandonic" where every parameter is a single atomic step, so it's laborious to express things like tree-structured queries that are more complex than parameter dictionaries.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#92
post #18

Earlier quoted context omitted.

But that was exactly how programming was taught 30 years ago. Relational data modeling was your starting point. Then object orientation came along and we suffered the dreaded object relational impedance mismatch when we couldn’t translate invoice header to invoice line-items in an object oriented way. Yet somehow the majority of development continued down this design philosophy, abstracting the data model further and…

I think NoSQL is only as popular as it is because a lot engineers don’t have a proper understanding of the relational model (which isn’t hard to understand at all, it just seems to have fallen out of popularity). I certainly think there are completely valid use cases for denormalized datastores, but I don’t think those use cases are what’s driving their adoption. You can see this in how so many mongo/dynamo... apps e…

NoSQL is popular because it's untyped/dynamically-typed. You don't need to do a schema migration every time you insert a new data type.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#93
post #55

Earlier quoted context omitted.

Can you drop nans, parse strings to floats, apply arbitrary lambdas over a rolling window, take the cosine of those values, and plot a graph against time, in a single line of sane SQL? Easy in Pandas. Dataframes are not tables; tables are not dataframes. It's nearly as apples-to-bananas as comparing Python lists to C arrays.

everything you mentioned can be done easily through database schema. window functions work well in SQL. plots are easily done in any BI solution that hooks up to any database. pandas is just poor man's SQL+BI. pandas stores everything in memory and has many limitations. in SQL Server I can easily churn through terabyte sized database and get the data I need, because the schema is well designed with partitioned tables…

Why aren't you or some other sql whiz bringing home all the kaggle prize money being won by people using inefficient dataframes?

Re: Is a Dataframe Just a Table? (2019) [pdf]

#94
post #54

> Having many different ways to express the same logic makes it hard for developers to understand programs of heterogeneous styles. Besides having varying ways to express the same simple logic, the sheer number of APIs (> 200) that are not only overloaded but also have default parameters that may change version to version, making it hard to remember the APIs. It's a bit tangential to the main point, but I do agree wi…

I feel like I'm constantly looking up SO or blog posts that benchmark Pandas methods while I'm coding with Pandas. You have to, since the inefficiency you add with a slower method is nontrivial.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#95
post #5

This doesn't seem to be from 2016 (some identifiers suggest that it is, but it cites papers up to 2018 and it says that a tweet from 2016 is two years old).

Indeed. "Conference on Very Important Topics 2016" is not a real conference, but placeholder from a template. Maybe it was left behind by accident? The paper is from the PLATEAU Workshop 2019.

Oh I thought "Conference on Very Important Topics" was Twitter

Re: Is a Dataframe Just a Table? (2019) [pdf]

#96
post #7

“Now GraphQL has almost 15K GitHub stars and a large and active developer community. What should we make of it?“ don’t know why, made me smile. Didnt know the DB world is so edgy - its cute. I say bygons about graphql, usability always trumps fad in the end and “nature will decide”, nature the masses and hordes of “US over Time”. Does GraphQL do warehousing well ? Well im not militant and think its cool. One could sa…

Thanks to GraphQL, the DB world is edgy and nodey.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#98
post #48

The article is a nice comparison of pros and cons of Pandas and SQL, but the title and ensuing comments are misleading and off the point, akin to favorite arguments like "is Haskell types just C++ objects"? They have similar and differences, in core semantics and in ergonomics. Pandas has a perlish API designed for abbreviating common tasks, combined with a Pythonic (per common practice, despite being against Putin's…

> being against Putin's stated principles

walk carefully

Re: Is a Dataframe Just a Table? (2019) [pdf]

#99
post #54

> Having many different ways to express the same logic makes it hard for developers to understand programs of heterogeneous styles. Besides having varying ways to express the same simple logic, the sheer number of APIs (> 200) that are not only overloaded but also have default parameters that may change version to version, making it hard to remember the APIs. It's a bit tangential to the main point, but I do agree wi…

I just finished a lengthy analysis of why pandas groupby operations ends up harder to use than R's dplyr or data.table.

For example, a grouped filter is very cumbersome in pandas.

Interested to hear if you think it gets at the heart of the problem.

https://mchow.com/posts/2020-02-11-dplyr-in-python/

Post reply on HN