Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is a Dataframe Just a Table? (2019) [pdf]

plateau-workshop.org

Post reply on HN
Print view View original

41–50 of 120 posts

Re: Is a Dataframe Just a Table? (2019) [pdf]

#41

Post by shawnz »

em500 wrote: post #23
nnq wrote: post #20

Earlier quoted context omitted.

How do you aggregate without field types? You'd end up writing custom code for untyped map-reduce... How can you code know what data to expect without typed field? Your code size can grow up even 10x if you need to assume that any record can have any shape... You could have more complex and user defined types in an ideal super-SQL, like "int or map:string->bool" etc., but you WANT types. They reduce complexity at all…

None of your remarks come close to answering my question (is having a type definition in the field/schema fundamental to a table, or just a way that most SQL databases decided to implement them)? Regarding your questions, in case of sqlite3, a _value_ is typed, but a _field_ is not. sqlite3 only has a very small number of types (numbers, text and nulls) and functions (both aggregation and non-aggregation) that expect…

Who's to say who gets to write the formal definition? And just because the formal definition claims or doesn't claim something, why does that mean every single implementation has to behave exactly that way?

Re: Is a Dataframe Just a Table? (2019) [pdf]

#42

Post by gpderetta »

js8 wrote: post #6

> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…

As a big fan of the relational model (but not so much of SQL) I just wish that pandas didn't tread indices specially but just as a normal columns. Also, multiindices should just be indices on multiple columns. And I should be able to have as many indices as I want.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#43

Post by kthielen »

js8 wrote: post #6

> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…

> Anyway, I think the way out of this mess is to have a functional, Haskell-like language for processing tables

I agree, and in a Haskell-like setting we can have any structured types (rather than just primitives).

That’s why I built hobbes: https://github.com/Morgan-Stanley/hobbes

Re: Is a Dataframe Just a Table? (2019) [pdf]

#44

Post by TeMPOraL »

synthc wrote: post #27
AmericanChopper wrote: post #22

Earlier quoted context omitted.

I think NoSQL is only as popular as it is because a lot engineers don’t have a proper understanding of the relational model (which isn’t hard to understand at all, it just seems to have fallen out of popularity). I certainly think there are completely valid use cases for denormalized datastores, but I don’t think those use cases are what’s driving their adoption. You can see this in how so many mongo/dynamo... apps e…

I think a factor in this is that NoSql databases have nice API's that programmers can use to setup tables, do simple queries etc, which makes it much easier to get started. For RDBMS's you have to muck around with connections and SQL, which is more powerful but requires much more ceremony. (Connection pooling, prepared statements etc) The lack of understanding of the relational model is not the limiting factor in my…

> but requires much more ceremony. (Connection pooling, prepared statements etc)

NoSQL databases need you to go through this ceremony too. It may be less obvious if you're just passing JSONs over HTTP all the way, but something still needs to keep connections and sanitize untrusted input.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#45

Post by gpderetta »

kumarvvr wrote: post #38
lincpa wrote: post #33

Earlier quoted context omitted.

Everything is RMDB I advocate: Building a relational data model on top of hash-map to achieve a combination of NoSQL and RMDB advantages. This is actually a reverse implementation of PostgreSQL. [Clojure is a functional programming language based on relational database theory]( https://github.com/linpengcheng/PurefunctionPipelineDataflow... ) [Everything is RMDB]( https://github.com/linpengcheng/PurefunctionPipelineD…

Can you elaborate on the topic of a "relational data model on top of a hash map"? Are there any books that cover the concepts?

not the parent (but I like their thinking); I think the key concept is to treat the hash map as just a (potentially primary) index over your data. Of course keeping secondary indices up to date is now the job of the application (or the abstraction layer)

Re: Is a Dataframe Just a Table? (2019) [pdf]

#46

Post by eru »

js8 wrote: post #6

> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…

Have you had a look at Datalog? It's a carefully selected subset of prolog that corresponds to primitive recursive functions.

So: plenty of computational expressivity, but solidly removed from Turing completeness.

Adding types for relational algebra to Haskell is a bit of a slog. Not because it's impossible, but just because the natural way to express those data types is not what comes natural in Haskell.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#47

Post by mjburgess »

imtringued wrote: post #39
ak39 wrote: post #18

Earlier quoted context omitted.

But that was exactly how programming was taught 30 years ago. Relational data modeling was your starting point. Then object orientation came along and we suffered the dreaded object relational impedance mismatch when we couldn’t translate invoice header to invoice line-items in an object oriented way. Yet somehow the majority of development continued down this design philosophy, abstracting the data model further and…

I've never understood the myth of the "object relational impedance mismatch" (ORIM). Objects are just a collection of attributes. Table rows are just a collection of attributes. There are some things that objects can do that the relational model can't but if you are designing a database schema why would you care about those things? There is no rule that you have to use every feature of a programming language everywhe…

This misunderstands the relational model -- it doesn't mean "my app's data model". Ie., it's not that there's fields and rows.

It's that the data is represented with sets, and there's an algebra over those that provides (strong) guarantees and principled way of composing operations.

Yes, you use objects as mere key-value pairs and provide a bizarre semantics for the relational algebra over sets of these objects -- but! -- this isn't object orientation.

Eg., in OO objects compose -- in the relational algebra rows dont "compose", eg., even having person.address.street breaks the semantics of 'SELECT'

The interpretation of p.a.s has to be as a subset on a product of relations (ie., tables P and A filtered on a join of p.id to a.id, etc...)

This is one of the key impedance mismatches in OORelations -- composition =/= join. Hence awkward and and ugly workarounds in all ORMs.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#48

Post by gowld »

The article is a nice comparison of pros and cons of Pandas and SQL, but the title and ensuing comments are misleading and off the point, akin to favorite arguments like "is Haskell types just C++ objects"? They have similar and differences, in core semantics and in ergonomics.

Pandas has a perlish API designed for abbreviating common tasks, combined with a Pythonic (per common practice, despite being against Putin's stated principles) disregard for informative use of types, instead a simple coherent, principled, layered API. That makes it look like "SQL done wrong" because the semantic differences aren't laid out explicitly.

Re: Is a Dataframe Just a Table? (2019) [pdf]

#50

Post by silveraxe93 »

idclip wrote: post #34
js8 wrote: post #8

Earlier quoted context omitted.

> Anyone with examples of things that arnt essentially tables ? Well, my definition of table: Fixed number of columns of possibly heterogenous types. Variable number of rows that have all the same type. So, for starters, a matrix is not a table. Nor is a list, a tree, or a hash table (KV store).

Oh interesting, because i wanted to say “everything is a tensor” before i write the long post So generics and mixed types is where it hits a wall with tables? Though those mixed structured can just be broken down to collections of tables and enumerated, no? Then call the table of tables a table again? Also why is a matrix not a table? Because its a super type ?

A matrix is not a table because all the columns need to be the same type. While you can convert a matrix to a table, the converse is not true.

(I assume this is what he meant, as it is the case in R.)

Post reply on HN