Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is a Dataframe Just a Table? (2019) [pdf]

plateau-workshop.org

Post reply on HN
Print view View original

61–70 of 120 posts

Re: Is a Dataframe Just a Table? (2019) [pdf]

#61

Post by gmfawcett »

imtringued wrote: post #36
js8 wrote: post #15

Earlier quoted context omitted.

The way I see it, SQL and GraphQL are solving somewhat complementary problems. In SQL, I have a structure (all these tables that possibly have to be joined) in the database and I want to pick something out as a simple result table. In GraphQL, I create the more complex structure on the output. But I do consider GraphQL somewhat unnecessary, because if those REST APIs composed just like tables do in the database, then…

>you wouldn't need GraphQL, and you could run a normal query. You're making the assumption that there is something to run a normal query on. As soon as you write even a single line of server side code this assumption is broken. What if the GraphQL query doesn't actually use an SQL database and just reads something from a file or another service? What if the server is responsible for granting row level access under ve…

I'm not trying to move the goalposts -- you've mentioned SQL specifically, and not relational databases in general -- but RDBM systems offer these features, and in a way that's compatible with access via SQL. Foreign data wrappers can present external data sources as if they were SQL views, allowing cross-source relations to be constructed and queried. Complex row level access and business can be implemented in stored procedures and triggers, if a simpler method can't be found. People will debate the extent to which these features ought to be used, but they certainly exist.

My point is that it isn't "downright nonsensical" at all.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#62

Post by bobbylarrybobby »

leto_ii wrote: post #54

> Having many different ways to express the same logic makes it hard for developers to understand programs of heterogeneous styles. Besides having varying ways to express the same simple logic, the sheer number of APIs (> 200) that are not only overloaded but also have default parameters that may change version to version, making it hard to remember the APIs. It's a bit tangential to the main point, but I do agree wi…

Could you give some examples of this issue with pandas?

Re: Is a Dataframe Just a Table? (2019) [pdf]

#63

Post by leto_ii »

bobbylarrybobby wrote: post #62
leto_ii wrote: post #54

> Having many different ways to express the same logic makes it hard for developers to understand programs of heterogeneous styles. Besides having varying ways to express the same simple logic, the sheer number of APIs (> 200) that are not only overloaded but also have default parameters that may change version to version, making it hard to remember the APIs. It's a bit tangential to the main point, but I do agree wi…

Could you give some examples of this issue with pandas?

Some of the examples provided in the paper are eloquent:

•df[df.a>3]

•df[df["a"]>3]

•df.loc[df.a>3]

•df.loc[df["a"]>3]

Re: Is a Dataframe Just a Table? (2019) [pdf]

#64

Post by airstrike »

leto_ii wrote: post #54

> Having many different ways to express the same logic makes it hard for developers to understand programs of heterogeneous styles. Besides having varying ways to express the same simple logic, the sheer number of APIs (> 200) that are not only overloaded but also have default parameters that may change version to version, making it hard to remember the APIs. It's a bit tangential to the main point, but I do agree wi…

I mean, perhaps comparing R to panda is too low a bar.

If I read "Having many different ways to express the same logic makes it hard for developers to understand programs of heterogeneous styles" in a vacuum, the very first thing I would think of is R.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#65

Post by airstrike »

synthc wrote: post #27
AmericanChopper wrote: post #22

Earlier quoted context omitted.

I think NoSQL is only as popular as it is because a lot engineers don’t have a proper understanding of the relational model (which isn’t hard to understand at all, it just seems to have fallen out of popularity). I certainly think there are completely valid use cases for denormalized datastores, but I don’t think those use cases are what’s driving their adoption. You can see this in how so many mongo/dynamo... apps e…

I think a factor in this is that NoSql databases have nice API's that programmers can use to setup tables, do simple queries etc, which makes it much easier to get started. For RDBMS's you have to muck around with connections and SQL, which is more powerful but requires much more ceremony. (Connection pooling, prepared statements etc) The lack of understanding of the relational model is not the limiting factor in my…

Perhaps this is one reason why frameworks like Django are so popular

Re: Is a Dataframe Just a Table? (2019) [pdf]

#66

Post by leto_ii »

airstrike wrote: post #64
leto_ii wrote: post #54

> Having many different ways to express the same logic makes it hard for developers to understand programs of heterogeneous styles. Besides having varying ways to express the same simple logic, the sheer number of APIs (> 200) that are not only overloaded but also have default parameters that may change version to version, making it hard to remember the APIs. It's a bit tangential to the main point, but I do agree wi…

I mean, perhaps comparing R to panda is too low a bar. If I read "Having many different ways to express the same logic makes it hard for developers to understand programs of heterogeneous styles" in a vacuum, the very first thing I would think of is R.

Indeed, but I was more specifically referring to smth like data.table, which I found simpler to use than Pandas.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#68

Post by mumblemumble »

js8 wrote: post #6

> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…

OK, so, disclaimer, I haven't ever used GraphQL with live ammunition, but I really don't get the impression that it can be dismissed so easily.

TFA does a good job of showing that, while SQL is a cleaner design and theoretically better than the hot mess that is Pandas dataframes in every way, the dataframes API offers a lot of conveniences and ergonomic affordances that can make it more pleasant to program against for certain tasks. As someone who spends a heck of a lot of time using Pandas to prep data for loading into an RDBMS, I agree 1,000%. As someone who, once upon a time, did assembly language programming on both RISC and CISC architectures, and, once upon another time, bounced back and forth between Scheme and Objective-C, this sort of situation doesn't surprise me at all. It's just the standard "worse is better" story that we all know and love to hate.

I suspect it's similar for GraphQL. For example, the JSON basis for the language does make it awful to write by hand, but, if you're talking APIs here, you may not be doing too much much hand-authoring of queries. And it's going to be a lot easier than SQL to manipulate programmatically, since, particularly in a language like JavaScript or Python, its syntactic structure maps trivially to basic data structures that already exist in the language.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#69

Post by tylerl »

TeMPOraL wrote: post #24
AmericanChopper wrote: post #22

Earlier quoted context omitted.

I think NoSQL is only as popular as it is because a lot engineers don’t have a proper understanding of the relational model (which isn’t hard to understand at all, it just seems to have fallen out of popularity). I certainly think there are completely valid use cases for denormalized datastores, but I don’t think those use cases are what’s driving their adoption. You can see this in how so many mongo/dynamo... apps e…

"Serverless" is just "someone else's servers". I might be missing something big here, but I don't see why - in principle - a "serverless", SQL-over-API-as-a-Service couldn't work.

> "Serverless" is just "someone else's servers".

Clearing that bit up:

Cloud is "just someone else's servers".

Serverless is "someone else maintains the infrastructure." It's one step further in the same direction.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#70

Post by barrkel »

imtringued wrote: post #36
js8 wrote: post #15

Earlier quoted context omitted.

The way I see it, SQL and GraphQL are solving somewhat complementary problems. In SQL, I have a structure (all these tables that possibly have to be joined) in the database and I want to pick something out as a simple result table. In GraphQL, I create the more complex structure on the output. But I do consider GraphQL somewhat unnecessary, because if those REST APIs composed just like tables do in the database, then…

>you wouldn't need GraphQL, and you could run a normal query. You're making the assumption that there is something to run a normal query on. As soon as you write even a single line of server side code this assumption is broken. What if the GraphQL query doesn't actually use an SQL database and just reads something from a file or another service? What if the server is responsible for granting row level access under ve…

That's a distinction without a difference; you can execute SQL against CSV with the right tool, nothing you need to write yourself, likewise with programmatic data sources.

Implementing significant business logic on your data access path has other ramifications, but they're orthogonal to the query syntax chosen. And I say that as someone who thinks GraphQL is a fine idea, especially for reducing churn in API specs when you have separate teams for front and back end dev.

Post reply on HN