71–80 of 117 posts
Earlier quoted context omitted.
If you only have 1tb of data then you can have it in ram on a modern server.
In the last few years, sure, but certainly not in 2012.
Not cheap, but these were at companies with 100s of SWEs / billions in revenue / would eventually have multi-million dollar cloud bills for what little they migrated there.
It's not the point of the blog post, but I love the fact that the author's 2012 MacBook Pro is still useable. I can't imagine there are too many Dell laptops from that era still alive and kicking.
I would definitely not trade that for a pre-computed analytics approach. The freedom to explore in real time is enlightening and freeing.
I think you have restricted yourself to recomputed fix analytics but real time interactive analytics is also an interesting area.
> History is full of “what if”s, what if something like DuckDB had existed in 2012? The main ingredients were there, vectorized query processing had already been invented in 2005. Would the now somewhat-silly-looking move to distributed systems for data analysis have ever happened? I like the gist of the article, but the conclusion sounds like 20/20 hindsight. All the elements were there, and the author nails it, but…
Is only after being burned many many times that arise the need for simplicity.
Is the same of NoSql. Only after suffer it you appreciate going back.
ie: Tools like this circle back only after the pain of a bubble. It can't be done inside it
I mostly see clickhouse,postgress etc
Earlier quoted context omitted.
I'm not sure I really follow, you can create new tables for any step if you want to do it entirely within the db, but you can also just run duckdb against your dataframes in memory.
In R, data sources, intermediate results, and final results are all dataframes (slight simplification). With DuckDB, to have the same consistency you need every layer and step to be a database table, not a data frame, which is awkward for the standard R user and use case.
Data.Table is competitive with DuckDb in many cases, though as a DuckDB enthusiast I hate to admit this. :)
This makes a completely valid point when you constrain the meaning of Big Data to “the largest dataset one can fit on a single computer”.
At companies I've worked at "Big Data" was often used to mean "too big to open in Excel" or in the extreme case "too big to fit in RAM on my laptop"
Around 0.5 to 50 GB is such an annoying area, because Excel starts falling over on the lower end and even nicer computers will start seriously struggling on the larger end if you're not being extremely efficient.
Earlier quoted context omitted.
> Maybe it was all VC funded solutions looking for problems? Remember, from 2020-2023 we had an entire movement to push a thing called "Modern data stack (MDS)" with big actors like a16z lecturing the market about it [1]. I am originally from Data. Never worked with anything out of the Data: DS, MLE, DE, MLOps and so on. One thing that I envy from other developer careers is to have bosses/leaders that had battle-test…
Right.. shove your data in our data platform. OK now you need PortCo1's company analytics platform, PortCo2's orchestration platform, PortCo3's SRE platform, PortCo4's Auth platform, PortCo5's IaC platform, PortCo6's Secrets Mgmt Platform, PortoCo7's infosec platform, etc. I am sure I forgot another 10 things. Even if some of these things were open source or "open source", there was the upsell to the managed/supporte…