Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is a Dataframe Just a Table? (2019) [pdf]

plateau-workshop.org

Post reply on HN
Print view View original

101–110 of 120 posts

Re: Is a Dataframe Just a Table? (2019) [pdf]

#101

Post by somurzakov »

lonelappde wrote: post #93
somurzakov wrote: post #74

Earlier quoted context omitted.

everything you mentioned can be done easily through database schema. window functions work well in SQL. plots are easily done in any BI solution that hooks up to any database. pandas is just poor man's SQL+BI. pandas stores everything in memory and has many limitations. in SQL Server I can easily churn through terabyte sized database and get the data I need, because the schema is well designed with partitioned tables…

Why aren't you or some other sql whiz bringing home all the kaggle prize money being won by people using inefficient dataframes?

i think you are confusing xgboost that uses its own DMatrix that is winning kaggle, not dataframe

Re: Is a Dataframe Just a Table? (2019) [pdf]

#103

Post by TeMPOraL »

tylerl wrote: post #69
TeMPOraL wrote: post #24

Earlier quoted context omitted.

"Serverless" is just "someone else's servers". I might be missing something big here, but I don't see why - in principle - a "serverless", SQL-over-API-as-a-Service couldn't work.

> "Serverless" is just "someone else's servers". Clearing that bit up: Cloud is "just someone else's servers". Serverless is "someone else maintains the infrastructure." It's one step further in the same direction.

Correct.

If the cloud is akin to renting a flat instead of owning it, then serverless is like living in hotels.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#104

Post by leto_ii »

closed wrote: post #99
leto_ii wrote: post #54

> Having many different ways to express the same logic makes it hard for developers to understand programs of heterogeneous styles. Besides having varying ways to express the same simple logic, the sheer number of APIs (> 200) that are not only overloaded but also have default parameters that may change version to version, making it hard to remember the APIs. It's a bit tangential to the main point, but I do agree wi…

I just finished a lengthy analysis of why pandas groupby operations ends up harder to use than R's dplyr or data.table. For example, a grouped filter is very cumbersome in pandas. Interested to hear if you think it gets at the heart of the problem. https://mchow.com/posts/2020-02-11-dplyr-in-python/

I really think it does.

I also appreciate your idea of porting dplyr to python, keep up the good work :)

This table sums up some of it:

operation | time

apply score + 1 | 30s

apply score.values + 1 | 3s

transform score + 1 | 30s

transform score.values + 1 | 20s

It seems to me that pandas is simply a leakier abstraction than dplyr, data.table etc. As a user of the library in most instances you shouldn't have to profile your code to figure out why things behave the way they do (btw, thanks for pointing out snakeviz - it seems like a useful tool).

This being said, we shouldn't complain too much about pandas - it is in the end a very important and useful tool.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#105

Post by [deleted] »

leto_ii wrote: post #54

> Having many different ways to express the same logic makes it hard for developers to understand programs of heterogeneous styles. Besides having varying ways to express the same simple logic, the sheer number of APIs (> 200) that are not only overloaded but also have default parameters that may change version to version, making it hard to remember the APIs. It's a bit tangential to the main point, but I do agree wi…

[deleted]

Re: Is a Dataframe Just a Table? (2019) [pdf]

#106

Post by noelsusman »

somurzakov wrote: post #74
kortex wrote: post #55

Earlier quoted context omitted.

Can you drop nans, parse strings to floats, apply arbitrary lambdas over a rolling window, take the cosine of those values, and plot a graph against time, in a single line of sane SQL? Easy in Pandas. Dataframes are not tables; tables are not dataframes. It's nearly as apples-to-bananas as comparing Python lists to C arrays.

everything you mentioned can be done easily through database schema. window functions work well in SQL. plots are easily done in any BI solution that hooks up to any database. pandas is just poor man's SQL+BI. pandas stores everything in memory and has many limitations. in SQL Server I can easily churn through terabyte sized database and get the data I need, because the schema is well designed with partitioned tables…

Of course you shouldn't use Pandas to analyze terabytes of data, but most people aren't analyzing terabytes of data.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#107

Post by tanilama »

kortex wrote: post #55
tanilama wrote: post #21

It is worse than a table. Dataframe don't really have a clear boundary as what it can do or cant do. I see no problem why you can't shoehorn some control flow logic into it which makes it pretty much just an executor of arbitrary computation graph specified in DSL, which relies on however the developer decides to implement it. I'd rather take SQL because I have a better understanding what it is doing.

Can you drop nans, parse strings to floats, apply arbitrary lambdas over a rolling window, take the cosine of those values, and plot a graph against time, in a single line of sane SQL? Easy in Pandas. Dataframes are not tables; tables are not dataframes. It's nearly as apples-to-bananas as comparing Python lists to C arrays.

> Can you drop nans, parse strings to floats, apply arbitrary lambdas over a rolling window, take the cosine of those values, and plot a graph against time, in a single line of sane SQL?

Yes you can, Postgres support Python UDF btw. Though I don't think that is necessary a daily common feature to use.

SQL is pretty powerful if you look deep

Re: Is a Dataframe Just a Table? (2019) [pdf]

#108

Post by idclip »

em500 wrote: post #23
nnq wrote: post #20

Earlier quoted context omitted.

How do you aggregate without field types? You'd end up writing custom code for untyped map-reduce... How can you code know what data to expect without typed field? Your code size can grow up even 10x if you need to assume that any record can have any shape... You could have more complex and user defined types in an ideal super-SQL, like "int or map:string->bool" etc., but you WANT types. They reduce complexity at all…

None of your remarks come close to answering my question (is having a type definition in the field/schema fundamental to a table, or just a way that most SQL databases decided to implement them)? Regarding your questions, in case of sqlite3, a _value_ is typed, but a _field_ is not. sqlite3 only has a very small number of types (numbers, text and nulls) and functions (both aggregation and non-aggregation) that expect…

I think u both complete each other as a full argument with a pro and contra arrangement. Lovely af to read you both honestly.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#109

Post by perl4ever »

roenxi wrote: post #14

Tables and data frames are both leaves in the far more fundamental flow that beginners don't pick up because it is too simple - the relational model of data. The real problem is the the basic normal forms are so obvious and simple it is difficult to tell if people designed around them on purpose or stumbled onto the right path. I suspect the distinction between tables and data frames (and arguing about query language…

I always felt like the relational model, SQL, and the extensions and add-ons like PL/SQL come from people with totally different worldviews and make quite a mess together. I couldn't design something better than PL/SQL, but it just seems so, so wrong to me.

On the other hand, the purists* who rant about nulls, I think have missed something as well.

But on the whole, I would like to see something related to SQL, that tries to improve it without grafting things on, that has an overall vision and an approach of simplifying and making it more coherent. Not being committed to the syntax, and wanting to close the loop more between manipulating data and manipulating the language itself.

I don't know, if you say SQL is a functional programming language, ok, and you take some common functional language that has nice syntax and make it consistent with the relational model, what do you get?

*Of which my impression was formed by (IIRC): "Relational Database Writings, 1989-1991 by C. J. Date" (which Amazon seems to want over $1,000 for right now, in paperback)

Re: Is a Dataframe Just a Table? (2019) [pdf]

#110

Post by perl4ever »

AmericanChopper wrote: post #22
ak39 wrote: post #18

Earlier quoted context omitted.

But that was exactly how programming was taught 30 years ago. Relational data modeling was your starting point. Then object orientation came along and we suffered the dreaded object relational impedance mismatch when we couldn’t translate invoice header to invoice line-items in an object oriented way. Yet somehow the majority of development continued down this design philosophy, abstracting the data model further and…

I think NoSQL is only as popular as it is because a lot engineers don’t have a proper understanding of the relational model (which isn’t hard to understand at all, it just seems to have fallen out of popularity). I certainly think there are completely valid use cases for denormalized datastores, but I don’t think those use cases are what’s driving their adoption. You can see this in how so many mongo/dynamo... apps e…

The dreaded generic object table* is something I think people have put in SQL databases forever and will continue to do so, so it seems superficially logical to me to say "why not just use an object store of some sort?"

*I have experience with a system, not designed by me, that had one, and we were always going to redo/split it but never did.

Post reply on HN