Earlier quoted context omitted.
Assessment Assurance, for double the bang.
Cherry Picker
Good role for all Perky Cheekers, surely!
Big data is dead
421–430 of 444 posts
Re: Big data is dead
#422"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…
I've found this to be true in the (albeit small) # of samples I've run up against.
There's an agenda by some level of management, and they use "data" to forward their agenda, or disregard it due to "unexplainability" (a legitimate concern for BD/ML) if it disagrees with the agenda.
Re: Big data is dead
#423This entire post reads like "you probably don't actually have big data". What do these blockchains do that have to keep data around forever, with high throughput, and need to expose it quickly do? Are you saying they should delete parts of data in the chain? Seriously, I've spent my career working on big data systems, and while the answer is sometimes "yes you need to delete your data", I don't think that's going to…
And what about these blockchains? The full history of Bitcoin blockchain is less than 500gb, so for any analysis just getting a machine with a terabyte of RAM is both simpler and cheaper (once you include dev+ops time) than doing any horizontal scaling across multiple machines with "Big Data" approaches. "You probably don't actually have big data" is a very valid point, not that many organizations do - most businesse…
Bitcoin is notoriously slow. I don't think it's a good example of a high-throughput system. There are chains out there with 100x the number of transactions per second than that of Bitcoin. https://realtps.net/
Re: Big data is dead
#424Earlier quoted context omitted.
> Data Scientist's job is to launder management's intuition using quantitative methods Ouch. This is savage, but sadly correct in many cases. HOWEVER, to play devil's advocate here, I've also seen corporate data scientists overstate the conclusions / generalizability of their analysis. I've also seen data scientists fall prey to believing that their analysis proves would should be done, rather than what is likely to…
> The role of an executive or decision maker is to apply a normative lens to problems. The role of the data scientist / economist / whatever is to reduce the uncertainty that an action will have the desired effect. Where do business analysts fit into this dichotomy? Their whole job is to poke around in Tableau in order to surface high-ROI strategies for the business to pursue. (Where, in choosing which proposals to s…
People who poke around in Tableau might not get a lot of respect in the hierarchy of DataFolk, but descriptive statistics and thoughtfully chosen visualizations can be immensely useful. Exploratory data analysis sometimes reveals patterns that are so obvious that to apply statistical inference is just vanity.
If understanding the data generating processes is the goal, I'd rather see some useful plots than wade through a technical description of some model whose assumptions were flagrantly violated.
Re: Big data is dead
#425"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…
I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…
"Data Scientists exist not to uncover insights ..."
That was a plot point in Dirk Gentley's Holistic Detective Agency (1989), though the observation much pre-dates this.
Re: Big data is dead
#426Earlier quoted context omitted.
Developing apps as local-mostly with remote being "just storage" might've been interesting approach but oh so many stuff moved to webshit from native apps and browsers still don't even have decent data management.
Well said! I wonder if Web3 could solve such a problem (or a zero trust solution). Where you provide your service that can run in a special container
I don't see incentive to host a bunch of stranger's stuff on your machine. The moment you make it easy and "bulletproof", the bad kind of content nobody wants coming from their IP will come with it.
Re: Big data is dead
#427> Customer data sizes followed a power-law distribution. The largest customer had double the storage of the next largest customer, the next largest customer had half of that, etc I’m no statistician, but I’m like 99% sure that’s an exponential, not a power law There’s a world of difference. The point of an exponential is that you can ignore big things. The point of a power law is that you can’t.
>> Customer data sizes followed a power-law distribution. The largest customer had double the storage of the next largest customer, the next largest customer had half of that, etc
> I’m no statistician, but I’m like 99% sure that’s an exponential, not a power law
I'm no expert either, but it seems correct. The power law distribution has each X value in an X/Y series decreasing by a specific factor, like this: https://en.wikipedia.org/wiki/Power_law#/media/File:Long_tai...
The exponential has each X value increasing by a specific factor, like this: https://en.wikipedia.org/wiki/Exponential_function#/media/Fi...
Re: Big data is dead
#428This is what most of the companies are doing.
Re: Big data is dead
#429Confirmation bias exists almost everywhere. Confirmation bias especially among senior management is highly dangerous as decisions are based on not on data and facts, rather they are based on anecdotes, hunch/feelings, with high probability of going wrong. This is precisely where data scientists play a significant role, by providing recommendations and presenting facts based on hard data and mathematical models, in order to ensure that senior management decisions are based on facts/data, and not on anecdotes and hunches. Furthermore, a data driven organisation must have a supporting culture, where data driven decisions are given precedence, and data scientists (data messengers) must be empowered to present facts as is, no matter whether these facts are aligned or not with the basic assumptions and biases held by the senior management team. Creating such a supporting organization culture is extremely important but definite not easy. Culture is one of the factors that makes a difference between success or failure in a data driven organisation.
Re: Big data is dead
#430The less than a terabyte datasets being common had me awestruck. I, singular post-doctoral scientist noobermin[0], have processed terabytes of data at a time on HPC systems. Sure, a lot of it was garbage and I had to wade through it, but no one paid me millions to do it, I just did it to publish the papers. Sure, I needed the system which cost someone a lot of money, I suppose. But, I considered myself a small fry co…
That's a different category of big data. I worked for a big pharma and they were building their big data department with Spark and friends. I was quite surprised that their biggest dataset had something like 200 GB. At the same time, though, there was a lot of DNA sequencing data, we were designing CRISPR probes etc. But Spark and Hadoop aren't really that helpful in this area, so the Big Data team wasn't involved in…
I think it does depend on the problem. Genetic stuff always seemed not easily parallelizable like my field (physics simulation) is. That said, the culture here is that MPI works and thus cray still builds computers that work better with it, so we use MPI so it works...etc etc.