I was unable to learn R because I couldn't understand what a dataframe is. It was irritating that it wasn't defined clearly and there seemed to be no connection to terminology that was familiar to me (relational databases, SQL tables etc.).
Is a Dataframe Just a Table? (2019) [pdf]
31–40 of 120 posts
Re: Is a Dataframe Just a Table? (2019) [pdf]
#32Earlier quoted context omitted.
I think NoSQL is only as popular as it is because a lot engineers don’t have a proper understanding of the relational model (which isn’t hard to understand at all, it just seems to have fallen out of popularity). I certainly think there are completely valid use cases for denormalized datastores, but I don’t think those use cases are what’s driving their adoption. You can see this in how so many mongo/dynamo... apps e…
I think a factor in this is that NoSql databases have nice API's that programmers can use to setup tables, do simple queries etc, which makes it much easier to get started. For RDBMS's you have to muck around with connections and SQL, which is more powerful but requires much more ceremony. (Connection pooling, prepared statements etc) The lack of understanding of the relational model is not the limiting factor in my…
What you say is True, but can be solved by an ORM, though that adds an extra layer of complexity
Re: Is a Dataframe Just a Table? (2019) [pdf]
#33> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…
Everything is RMDB
I advocate: Building a relational data model on top of hash-map to achieve a combination of NoSQL and RMDB advantages. This is actually a reverse implementation of PostgreSQL.
[Clojure is a functional programming language based on relational database theory](https://github.com/linpengcheng/PurefunctionPipelineDataflow...)
[Everything is RMDB](https://github.com/linpengcheng/PurefunctionPipelineDataflow...)
[Implement relational data model and programming based on hash-map (NoSQL)](https://github.com/linpengcheng/PurefunctionPipelineDataflow...)
Re: Is a Dataframe Just a Table? (2019) [pdf]
#34“Now GraphQL has almost 15K GitHub stars and a large and active developer community. What should we make of it?“ don’t know why, made me smile. Didnt know the DB world is so edgy - its cute. I say bygons about graphql, usability always trumps fad in the end and “nature will decide”, nature the masses and hordes of “US over Time”. Does GraphQL do warehousing well ? Well im not militant and think its cool. One could sa…
> Anyone with examples of things that arnt essentially tables ? Well, my definition of table: Fixed number of columns of possibly heterogenous types. Variable number of rows that have all the same type. So, for starters, a matrix is not a table. Nor is a list, a tree, or a hash table (KV store).
Oh interesting, because i wanted to say “everything is a tensor” before i write the long post
So generics and mixed types is where it hits a wall with tables? Though those mixed structured can just be broken down to collections of tables and enumerated, no? Then call the table of tables a table again?
Also why is a matrix not a table? Because its a super type ?
Re: Is a Dataframe Just a Table? (2019) [pdf]
#35I was unable to learn R because I couldn't understand what a dataframe is. It was irritating that it wasn't defined clearly and there seemed to be no connection to terminology that was familiar to me (relational databases, SQL tables etc.).
In R a dataframe is just a named list of vectors, where each vector has equal length.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#36Earlier quoted context omitted.
In the re-inventing SQL department, I'd take a look at EdgeQL/EdgeDB. It's not perfect, but much closer to a functional language and composes well. I would not consider GraphQL a poor reinvention of SQL, since its niche of decoupling and simplifying untrusted high latency clients is too different for flexible queries created by a trusted server. It competes with REST and RPC, not SQL. GraphQL's native operations are…
The way I see it, SQL and GraphQL are solving somewhat complementary problems. In SQL, I have a structure (all these tables that possibly have to be joined) in the database and I want to pick something out as a simple result table. In GraphQL, I create the more complex structure on the output. But I do consider GraphQL somewhat unnecessary, because if those REST APIs composed just like tables do in the database, then…
>you wouldn't need GraphQL, and you could run a normal query.
You're making the assumption that there is something to run a normal query on. As soon as you write even a single line of server side code this assumption is broken. What if the GraphQL query doesn't actually use an SQL database and just reads something from a file or another service? What if the server is responsible for granting row level access under very complicated rules that cannot be implemented with just SQL and would be completely insecure if it was done on the client? What if you actually need to do things like implement business logic?
What you're talking about is downright nonsensical within that context.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#37Summary: no, because row order matters in dataframes. That's why matrix operations are a better computational framework for dataframes than relational algebra.
It matters in many non-trivial queries as well.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#38> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…
Everything is RMDB I advocate: Building a relational data model on top of hash-map to achieve a combination of NoSQL and RMDB advantages. This is actually a reverse implementation of PostgreSQL. [Clojure is a functional programming language based on relational database theory]( https://github.com/linpengcheng/PurefunctionPipelineDataflow... ) [Everything is RMDB]( https://github.com/linpengcheng/PurefunctionPipelineD…
Can you elaborate on the topic of a "relational data model on top of a hash map"?
Are there any books that cover the concepts?
Re: Is a Dataframe Just a Table? (2019) [pdf]
#39Earlier quoted context omitted.
I think one limitation of what you're saying is that you need to process the data before they become relational model. You need to integrate and normalize, and therefore, at some point, you need to support datasets that are not in normal form. Or perhaps you don't even need to normalize. But I also agree with you. I think we are teaching programming wrong, we start with imperative programming, but perhaps we should s…
But that was exactly how programming was taught 30 years ago. Relational data modeling was your starting point. Then object orientation came along and we suffered the dreaded object relational impedance mismatch when we couldn’t translate invoice header to invoice line-items in an object oriented way. Yet somehow the majority of development continued down this design philosophy, abstracting the data model further and…
I've never understood the myth of the "object relational impedance mismatch" (ORIM).
Objects are just a collection of attributes. Table rows are just a collection of attributes. There are some things that objects can do that the relational model can't but if you are designing a database schema why would you care about those things? There is no rule that you have to use every feature of a programming language everywhere. So just use the least common denominator of features and you're good to go.
Complaining about ORIM is like complaining about the universal serial impedance mismatch. You can't plug in USB 3 (OOP) in USB 1.1 (relational) but you can plug in USB 1.1 into USB 3.0 and since we know the old model is still good enough for a lot of use cases (think keyboard and mouse) we still use it even though we have a completely different standard installed into our computers.
What this means in practice is that your Domain/Entity classes look exactly like your database schema, not the other way. You still have to write queries but your ORM makes it ten times easier by offering very convenient query builders that let you build dynamic queries without string concatenation.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#40> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…
I don't think it needs so much a full blown language - its just an api that allows you to feed data in and pull it out for any arbitrary variety of data (including types/classes and links between data), but then has special commands to tell it how to optimise data storage (for speeding up a particular kind of access need, for minimising data storage, etc.)