Is a Dataframe Just a Table? (2019) [pdf]
plateau-workshop.org
Is a Dataframe Just a Table? (2019) [pdf]
1–10 of 120 posts
Re: Is a Dataframe Just a Table? (2019) [pdf]
#2[deleted]
Re: Is a Dataframe Just a Table? (2019) [pdf]
#3Yes
Re: Is a Dataframe Just a Table? (2019) [pdf]
#4Summary: no, because row order matters in dataframes. That's why matrix operations are a better computational framework for dataframes than relational algebra.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#5This doesn't seem to be from 2016 (some identifiers suggest that it is, but it cites papers up to 2018 and it says that a tweet from 2016 is two years old).
Re: Is a Dataframe Just a Table? (2019) [pdf]
#6> What should we make of it?
People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.)
Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing just analytics (i.e. pandas), you pretty much only need two types - number and (trimmed) string (and sometimes a datetime, but that's just conveniently formatted number). (I think SAS got that right.)
Anyway, I think the way out of this mess is to have a functional, Haskell-like language for processing tables (not necessarily Turing complete) that would subsume declarativness of SQL, and that could be compiled into different targets. The language would basically specify some (limited, so recursion in that language would not be allowed) processing of tables (of known types), and you could then compose these processing pieces, or integrate them with custom processing in other language.
I understand why people hate SQL, it is on some level hard to write and compose. I think a correctly designed functional language would help here. Or maybe just tear LINQ out of the .NET ecosystem.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#7“Now GraphQL has almost 15K GitHub stars and a large and active developer community. What should we make of it?“
don’t know why, made me smile. Didnt know the DB world is so edgy - its cute.
I say bygons about graphql, usability always trumps fad in the end and “nature will decide”, nature the masses and hordes of “US over Time”. Does GraphQL do warehousing well ?
Well im not militant and think its cool.
One could say, based on my experience at uni with random access machine calcs by hand, that everything a computer does reduces to a table. Anyone with examples of things that arnt essentially tables ?
Re: Is a Dataframe Just a Table? (2019) [pdf]
#8“Now GraphQL has almost 15K GitHub stars and a large and active developer community. What should we make of it?“ don’t know why, made me smile. Didnt know the DB world is so edgy - its cute. I say bygons about graphql, usability always trumps fad in the end and “nature will decide”, nature the masses and hordes of “US over Time”. Does GraphQL do warehousing well ? Well im not militant and think its cool. One could sa…
> Anyone with examples of things that arnt essentially tables ?
Well, my definition of table: Fixed number of columns of possibly heterogenous types. Variable number of rows that have all the same type.
So, for starters, a matrix is not a table. Nor is a list, a tree, or a hash table (KV store).
Re: Is a Dataframe Just a Table? (2019) [pdf]
#9> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…
In the re-inventing SQL department, I'd take a look at EdgeQL/EdgeDB. It's not perfect, but much closer to a functional language and composes well.
I would not consider GraphQL a poor reinvention of SQL, since its niche of decoupling and simplifying untrusted high latency clients is too different for flexible queries created by a trusted server. It competes with REST and RPC, not SQL.
GraphQL's native operations are limited to following a foreign-key link, which has predictable performance (quasi linear in the number of objects in the response) and selecting a subset of fields (reduce the response size and enable field level deprecation and usage tracking). These limitations prevent both malicious clients and less performance concerned front-end developers from putting excessive load on the database. These limitations also allow it to work with both data stored in a database and data generated by an application, while supporting SQL is pretty much limited to being processed by a database.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#10I liked the article, although it kinda conflates SQL and database and Pandas and dataframe.
It does acknowledge other dataframe implementation (Base R and Spark) but most of the arguments about what defines a dataframe are taken from the Pandas API.