A dataframe may be a table, but it can also be a matrix by supporting some matrix operations.
yes, the problem is, it is not a very good table.
Is a Dataframe Just a Table? (2019) [pdf]
51–60 of 120 posts
Re: Is a Dataframe Just a Table? (2019) [pdf]
#52> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing jus…
Everything is RMDB I advocate: Building a relational data model on top of hash-map to achieve a combination of NoSQL and RMDB advantages. This is actually a reverse implementation of PostgreSQL. [Clojure is a functional programming language based on relational database theory]( https://github.com/linpengcheng/PurefunctionPipelineDataflow... ) [Everything is RMDB]( https://github.com/linpengcheng/PurefunctionPipelineD…
I agree, but prefer a tree map in combination with natural keys where possible to get ranges & sorting without complicating the design. Then I embed additional tree maps in records to model has-many-relations. And I know SQL very well, but this setup often provides the capabilities I need and is much more convenient to deal with.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#53This doesn't seem to be from 2016 (some identifiers suggest that it is, but it cites papers up to 2018 and it says that a tweet from 2016 is two years old).
Indeed. "Conference on Very Important Topics 2016" is not a real conference, but placeholder from a template. Maybe it was left behind by accident? The paper is from the PLATEAU Workshop 2019.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#54> Having many different ways to express the same logic makes it hard for developers to understand programs of heterogeneous styles. Besides having varying ways to express the same simple logic, the sheer number of APIs (> 200) that are not only overloaded but also have default parameters that may change version to version, making it hard to remember the APIs.
It's a bit tangential to the main point, but I do agree with this remark. I have always found Pandas uncomfortable to work with. I'm never sure if I'm doing things in the most efficient/idiomatic way and I've found it hard to be consistent over time, especially since I've picked up different bits of code from different places.
I've gotten a lot more efficiency out of R, especially the data.table package.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#55It is worse than a table. Dataframe don't really have a clear boundary as what it can do or cant do. I see no problem why you can't shoehorn some control flow logic into it which makes it pretty much just an executor of arbitrary computation graph specified in DSL, which relies on however the developer decides to implement it. I'd rather take SQL because I have a better understanding what it is doing.
Can you drop nans, parse strings to floats, apply arbitrary lambdas over a rolling window, take the cosine of those values, and plot a graph against time, in a single line of sane SQL?
Easy in Pandas.
Dataframes are not tables; tables are not dataframes. It's nearly as apples-to-bananas as comparing Python lists to C arrays.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#56Earlier quoted context omitted.
Can you elaborate on the topic of a "relational data model on top of a hash map"? Are there any books that cover the concepts?
not the parent (but I like their thinking); I think the key concept is to treat the hash map as just a (potentially primary) index over your data. Of course keeping secondary indices up to date is now the job of the application (or the abstraction layer)
But isnt that just a sql db?
Re: Is a Dataframe Just a Table? (2019) [pdf]
#57Earlier quoted context omitted.
not the parent (but I like their thinking); I think the key concept is to treat the hash map as just a (potentially primary) index over your data. Of course keeping secondary indices up to date is now the job of the application (or the abstraction layer)
But isnt that just a sql db?
the idea is, if for whatever reason you do not have access to a proper DB, can you build your own relational model on top of the bits you have?
Re: Is a Dataframe Just a Table? (2019) [pdf]
#58Earlier quoted context omitted.
Everything is RMDB I advocate: Building a relational data model on top of hash-map to achieve a combination of NoSQL and RMDB advantages. This is actually a reverse implementation of PostgreSQL. [Clojure is a functional programming language based on relational database theory]( https://github.com/linpengcheng/PurefunctionPipelineDataflow... ) [Everything is RMDB]( https://github.com/linpengcheng/PurefunctionPipelineD…
Can you elaborate on the topic of a "relational data model on top of a hash map"? Are there any books that cover the concepts?
"a relational data model on top top of hash-map" is my original idea, and then combine [The Pure Function Pipeline Data Flow v3.0 with Warehouse/Workshop Model](https://github.com/linpengcheng/PurefunctionPipelineDataflow), can perfectly realize the simplicity and unity combination of system architecture.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#59Earlier quoted context omitted.
The way I see it, SQL and GraphQL are solving somewhat complementary problems. In SQL, I have a structure (all these tables that possibly have to be joined) in the database and I want to pick something out as a simple result table. In GraphQL, I create the more complex structure on the output. But I do consider GraphQL somewhat unnecessary, because if those REST APIs composed just like tables do in the database, then…
>you wouldn't need GraphQL, and you could run a normal query. You're making the assumption that there is something to run a normal query on. As soon as you write even a single line of server side code this assumption is broken. What if the GraphQL query doesn't actually use an SQL database and just reads something from a file or another service? What if the server is responsible for granting row level access under ve…
These counterpoints are probably valid for most database systems, but with Postgres it's actually far more efficient to use it as the substrate in which everything else is embedded.
* Postgres has a robust, battle tested role based security model with inheritance.
* Postgres has foreign data wrappers that let you encapsulate external resources as tables that behave the same way as local tables for most use cases.
* Postgres has plugins for most of the popular programming languages.
If you really like GraphQL, the Postgres approach can still give you that too, using Hasura or PostGraphile.
Re: Is a Dataframe Just a Table? (2019) [pdf]
#60Earlier quoted context omitted.
not the parent (but I like their thinking); I think the key concept is to treat the hash map as just a (potentially primary) index over your data. Of course keeping secondary indices up to date is now the job of the application (or the abstraction layer)
But isnt that just a sql db?
But it is also Nosql (hash-map). It is flexible according to your needs, as sqldb or Nosql or data structure.