Listen to “Reason”
Big data is dead
201–210 of 444 posts
Re: Big data is dead
#202I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…
On the flip side, there's some great action coming from data insights. Look at Strella Biotech - they're putting sensors in sealed warehouses to detect spoilage for certain vegetables and fruits. That's something that can have great returns with just a few IoT devices and a novel sensor.
Re: Big data is dead
#203"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…
I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…
Reminds me of the book "Bullshit Jobs"
Re: Big data is dead
#204We need to re-think how to make data _useful_. The fact that the value hasn't materialized after decades of attempts, billions of dollars, and lots of tools and technology points to the fact that our core assumptions and patterns are wrong.
This post doesn't go far enough. It challenges the assumption that everyone's data is "big data" or that every company's data will eventually grow to be big data. I agree that "big data" was the wrong model. We also need to challenge that all data should be stored in one place (warehouse, lake, lakehouse). We need to challenge that one tool can be used for every data need. We need to challenge how we build systems both from a technology and people standpoint. We need to embrace that the problems and needs of companies _are always changing_.
We are living with conceptual inertia. Many of our patterns are an evolution from the 70's and 80's and the first relational databases. It's time to rethink how we "do data" from first principles.
Re: Big data is dead
#205Why would I use DuckDB instead of Clickhouse or similar? Is it just because I want to have the database embedded in my app and not connect to a server?
Re: Big data is dead
#206To the extent "Big Data" originally and is still often claimed to mean "data beyond what fits on a single [process/RAM/disk/etc]", it's always been strange to me how much it's identified with analytics pipelines doing largely trivial transformations producing ultra-expensive "BI" pablum.
Yes, thank goodness that part is dead. But meanwhile - we've still got more actual data than ever to store, and ever-tighter deadlines on finding and delivering it. If we can get back to that and let the PySpark bootcampers fade away, maybe things can get a little better for once.
In other words:
Even when querying giant tables, you rarely end up needing to process very much data. Modern analytical databases can do column projection to read only a subset of fields, and partition pruning to read only a narrow date range. They can often go even further with segment elimination to exploit locality in the data via clustering or automatic micro partitioning. Other tricks like computing over compressed data, projection, and predicate pushdown are ways that you can do less IO at query time. And less IO turns into less computation that needs to be done, which turns into lower costs and latency.
Big data is "dead" because data engineers (the programming ones, not the analysts-in-all-but-title) spent a ton of effort building DBs with new techniques that scale better than before, with other storage patterns than before. Someone still has to write and maintain those! And it would be even better if those tools and techniques could escape the half dozen major data cloud companies and be more directly accessible to the average small team.
Re: Big data is dead
#207From about 2008/2009/2010 or so on there was perhaps an over-emphasis on specialized tools for the mass acquisition of streams of data. Maybe in large part due to the explosion of $$ in ad-tech. Some people had legitimately insane click/impression streams -- I worked at a couple companies like that. Development of DBs based on LSM trees or other write-specialized storage structures became important. Existing relational databases weren't particularly well built for this stuff. This was part of, but not the whole story with the whole NoSQL thing. People were willing to go completely denormalized in order to gain some advantage or ability here. It helped that much of the data looked at was of perhaps little structural complexity.
In the meantime SSD storage took off, so the IOPS from a stock drive have skyrocketed, business domains for large data sets have broadened beyond click/impression streams, and the challenge now is not "can I store all this data" it's "WTH do I do with it?"
Regardless of quantity of data, structuring and analysis and querying of said data remains paramount. The challenge for anybody working with data is to represent and extract knowledge. I remain convinced that logic -- first order logic and its offshoot in the relational model -- remains the best tool for reasoning about knowledge. Codd's prognostications on data from the 1970s are still profound.
I think we're in a space now where we can turn our attention to knowledge management, not just accumulating streams of unstructured data. The challenge in a business is to discover and capture the rules and relationship in data. SQL is an existing but poor tool for this, based on some of the concepts in the relational model but tossing them together in a relatively uncomposable and awkward way (though it remains better than the dogs breakfast of "NoSQL" alternatives that were tossed together for a while there.)
My employer is working in this space, I think they have a really good product: https://relational.ai/
Re: Big data is dead
#208Earlier quoted context omitted.
There are functionally less than 1000 organizations that currently require distributed compute for data analysis. You can get off the shelf AWS units with 1000 cores, terabytes of ram and storage, etc. The cost of compute has decreased faster than the amount of data we have to store and process. What we used to do with spark jobs we can do with python on a single box.
This is not true. Any column store database (bigquery, Redshift, snowflake) implements distributed compute behind the scenes. When an analyst/business intelligence people have a query return in 3 seconds instead of 15 seconds, it's actually huge. Not just in aggregate amount of time saved, but in creating a quick feedback loop in testing hypothesizes. This is especially true considering that most analyst type people…
Yup.
I was just playing with some data from our manufacturing system, about 30 GB. I pulled the data to my laptop (very expensive Apple one) and while it fits on my disk just fine, it took about 15 minutes to download.
I imported it to ClickHouse which took a while due to figuring out whatever compression and LowCardinality() and so on. I ran a query and it took ClickHouse about 15 seconds. DuckDB pointed to the parquet files on my SSD took 19 seconds to do the same. Our big data tool took 2 seconds, while working with data directly in cloud storage.
Now of course this is entirely unfair - the big data thingie has over twenty times more CPUs than my laptop, and cloud storage is also quite fast when accessed from many machines at once. If I ran ClickHouse or DuckDB on 100 CPU machine with terabyte of RAM it might have still turned out faster.
But this experiment (I was thinking of using some of the new fancy tech to serve interactive applications with less latency) made me realize that big data is still a thing. This was a sample - one building from one site, which we have quite a few of.
Re: Big data is dead
#209The database was the key technology in the 2001-2011 decade: it allowed companies to store massive amount of data in an organized way, so that they could provide basic functionality (search, monitoring) to users.
Statistical learning is being the key "technology" of 2011-today: it allowed companies, which had stored massive amount of data, to feedback predictions to users.
I think AR/Computer Graphics will be the key technology of the next decade: it will allow users to interact directly and seamlessly with the insights produced by ML systems, and possibly feed-back information.
Re: Big data is dead
#210Earlier quoted context omitted.
I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…
> if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be). > Data Scientist's job is to launder management's intuition using quantitative methods :) It’s no different than the days when grey bearded wisemen would read the stars and weave a tale about the great glory that awaits the king if h…
And the alternative is to use the data as bones, throw it up in the air and let it tell you what to do?