21–30 of 131 posts
I'm looking through this guys website for 'today I learned' and at first I'm impressed by how many of them there are. But then I start thinking: when you're trying to solve a problem you search for a lot of data. None of his posts are attributed. He's getting all his information from somewhere and then he goes and posts these articles just ripping off other sources. I can understand when its based on your original wo…
I'm looking through this guys website for 'today I learned' and at first I'm impressed by how many of them there are. But then I start thinking: when you're trying to solve a problem you search for a lot of data. None of his posts are attributed. He's getting all his information from somewhere and then he goes and posts these articles just ripping off other sources. I can understand when its based on your original wo…
Honestly I don't see a problem with the few posts I looked at. It's like recipes. You can't copyright recipes. At least it's not AI-generated blogspam, but a modicum of at least curating went in here.
Btw, am I alone in thinking that DataFrame abstractions in OOP languages (like Pandas in Python) are oftentimes simply inferior to relational algebra? I'm not sure that many Data Scientists are aware of the expressive power of SQL.
For example let's try changing/fixing sampling rate of a dataset (.resample() in Pandas).
Or something like .cumsum() -- easy with SQL windowing functions, but man they are cumbersome.
Or quickly store the result in .parquet.
But all the above doesn't matter, because I feel like 99% of Pandas work involves quickly drawing charts on the data look at it or show to teammates.
Btw, am I alone in thinking that DataFrame abstractions in OOP languages (like Pandas in Python) are oftentimes simply inferior to relational algebra? I'm not sure that many Data Scientists are aware of the expressive power of SQL.
SQL implements a kind of set theory with relational elements and a bunch of practical features like pivots, window functions etc.
Pandas does the same. Most data frame libraries like dplyr etc. implement a common set of useful constructs. There’s not much difference in expressiveness. LINQ Is another language around manipulating sets that was designed with the help of category theory, and it arrives at the same constructs.
However SQL is declarative, which provides a path for query optimizers to parse and create optimized plans. Whereas with chained methods, unless one implements lazy evaluation one misses out on look aheads and opportunities to do rewrites.
Btw, am I alone in thinking that DataFrame abstractions in OOP languages (like Pandas in Python) are oftentimes simply inferior to relational algebra? I'm not sure that many Data Scientists are aware of the expressive power of SQL.
SQL is super expressive, but I think pandas gets a bad rap. At it's core the data model and language can be more expressive than relational databases (see [1]).
I co-authored a paper that explained these differences with a theoretical foundation[1].
I'm looking through this guys website for 'today I learned' and at first I'm impressed by how many of them there are. But then I start thinking: when you're trying to solve a problem you search for a lot of data. None of his posts are attributed. He's getting all his information from somewhere and then he goes and posts these articles just ripping off other sources. I can understand when its based on your original wo…
Here's a query showing the 23 posts that link to StackOverflow, for example: https://til.simonwillison.net/tils?sql=select+*+from+til+whe...
And 41 where I credit someone on Twitter: https://til.simonwillison.net/tils?sql=select+*+from+til+whe...
More commonly I'll include a link from the TIL back to a GitHub Issue thread where I figured something out - those issue threads often link back to other sources.
For that IPv6 one: https://til.simonwillison.net/networking/http-ipv6
I had tried and failed to figure this out using Google searches in the past. I wrote that up after someone told me the answer in a private Slack conversation - saying who told me didn't feel appropriate there.
My goal with that page was to ensure that future people (including myself) who tried to find this with Google would get a better result!
(I'm a bit upset about this comment to be honest, because attributing people is something of a core value for me - the bookmarks on my blog have a "via" mechanism for exactly that reason: https://simonwillison.net/search/?type=blogmark )
sqlite3 :memory: -cmd '.mode csv' ... It should be a war crime for programs in 2022 to use non-UNIX/non-GNU style command line options. Add it to the Rome Statute's Article 7 list of crimes against humanity. Full blown tribunal at The Hague presided over by the international criminal court. Punishable by having to use Visual Basic 3.0 for all programming for the rest of their life.
Btw, am I alone in thinking that DataFrame abstractions in OOP languages (like Pandas in Python) are oftentimes simply inferior to relational algebra? I'm not sure that many Data Scientists are aware of the expressive power of SQL.
There are loads of things that are not possible or are very cumbersome to write in SQL, but that pandas and many other dataframe systems allow. Examples are dropping null values based on some threshold, one-hot encoding, covariance, and certain data cleaning operations. These are possible in SQL but very cumbersome to write. There are also things that are outright impossible in a relational database related to metada…
Btw, am I alone in thinking that DataFrame abstractions in OOP languages (like Pandas in Python) are oftentimes simply inferior to relational algebra? I'm not sure that many Data Scientists are aware of the expressive power of SQL.
SQL does not exactly implement relational algebra in its pure form. SQL implements a kind of set theory with relational elements and a bunch of practical features like pivots, window functions etc. Pandas does the same. Most data frame libraries like dplyr etc. implement a common set of useful constructs. There’s not much difference in expressiveness. LINQ Is another language around manipulating sets that was designe…
> However SQL is declarative
Pick one :) the way I see it, if declarativeness is not a factor in assessing expressiveness, then expressiveness reduces to the uninteresting notion of Turing-equivalence.