11–20 of 120 posts
> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…
Summary: no, because row order matters in dataframes. That's why matrix operations are a better computational framework for dataframes than relational algebra.
> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…
I suspect the distinction between tables and data frames (and arguing about query languages) is from people who have lost sight of the important things when dealing with data - the relational model and a system that supports relational algebra. Beyond that, features and optimisations need to be justified in terms of today's (and tomorrow's) needs which are specific to the data. The only major issue with SQL is a lack of support for something like R's tidyverse gather() and spread() operations which could be considered as missing operations in relational algebra.
> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…
In the re-inventing SQL department, I'd take a look at EdgeQL/EdgeDB. It's not perfect, but much closer to a functional language and composes well. I would not consider GraphQL a poor reinvention of SQL, since its niche of decoupling and simplifying untrusted high latency clients is too different for flexible queries created by a trusted server. It competes with REST and RPC, not SQL. GraphQL's native operations are…
But I do consider GraphQL somewhat unnecessary, because if those REST APIs composed just like tables do in the database, then you wouldn't need GraphQL, and you could run a normal query. (There is also a problem of externalities, putting the processing costs on the client is cheaper.)
And thanks for pointing out EdgeDB.
Tables and data frames are both leaves in the far more fundamental flow that beginners don't pick up because it is too simple - the relational model of data. The real problem is the the basic normal forms are so obvious and simple it is difficult to tell if people designed around them on purpose or stumbled onto the right path. I suspect the distinction between tables and data frames (and arguing about query language…
But I also agree with you. I think we are teaching programming wrong, we start with imperative programming, but perhaps we should start with data modelling. How the data look like? What are the constraints? Can we test the constraints? How many bits I need to store it?
"Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious." -- Fred Brooks
“Now GraphQL has almost 15K GitHub stars and a large and active developer community. What should we make of it?“ don’t know why, made me smile. Didnt know the DB world is so edgy - its cute. I say bygons about graphql, usability always trumps fad in the end and “nature will decide”, nature the masses and hordes of “US over Time”. Does GraphQL do warehousing well ? Well im not militant and think its cool. One could sa…
> Anyone with examples of things that arnt essentially tables ? Well, my definition of table: Fixed number of columns of possibly heterogenous types. Variable number of rows that have all the same type. So, for starters, a matrix is not a table. Nor is a list, a tree, or a hash table (KV store).
Why do they need to have the same type? In sqlite a field can have a different type in every record/row. (https://www.sqlite.org/datatype3.html). Is having a fixed typed fields fundamental to the concept of a table, or just a property of most SQL implementations?
Tables and data frames are both leaves in the far more fundamental flow that beginners don't pick up because it is too simple - the relational model of data. The real problem is the the basic normal forms are so obvious and simple it is difficult to tell if people designed around them on purpose or stumbled onto the right path. I suspect the distinction between tables and data frames (and arguing about query language…
I think one limitation of what you're saying is that you need to process the data before they become relational model. You need to integrate and normalize, and therefore, at some point, you need to support datasets that are not in normal form. Or perhaps you don't even need to normalize. But I also agree with you. I think we are teaching programming wrong, we start with imperative programming, but perhaps we should s…
The central design premise for relational data modeling was “if your model could potentially allow inconsistencies, assume the inconsistencies”. Today that premise is easily brushed aside with “you’ll never get anyone manipulating this table without going through this layer” or “we then group by these columns to show unique records to the user”.
Tables and data frames are both leaves in the far more fundamental flow that beginners don't pick up because it is too simple - the relational model of data. The real problem is the the basic normal forms are so obvious and simple it is difficult to tell if people designed around them on purpose or stumbled onto the right path. I suspect the distinction between tables and data frames (and arguing about query language…
Earlier quoted context omitted.
> Anyone with examples of things that arnt essentially tables ? Well, my definition of table: Fixed number of columns of possibly heterogenous types. Variable number of rows that have all the same type. So, for starters, a matrix is not a table. Nor is a list, a tree, or a hash table (KV store).
> Variable number of rows that have all the same type. Why do they need to have the same type? In sqlite a field can have a different type in every record/row. ( https://www.sqlite.org/datatype3.html ). Is having a fixed typed fields fundamental to the concept of a table, or just a property of most SQL implementations?
How can you code know what data to expect without typed field? Your code size can grow up even 10x if you need to assume that any record can have any shape...
You could have more complex and user defined types in an ideal super-SQL, like "int or map:string->bool" etc., but you WANT types. They reduce complexity at all levels! You want them in you code, you want them in your data!
Sure, go ahead, just use MongoDB with no constraints on collections structure... You'll end up with 5x more code in you application and 5x more bugs.
SQL and the relational model help you think more, code less (and slower - but this is an advantage) and have less bugs with less effort! If you're lazy and prefer thinking deeper to "working harder", you'll always prefer properly typed SQL :P (Unless you're working with stuff that's naturally shaped as graphs or tensors.)