I think that big data has made math sexy, and selling applied statistics and operations research to small and medium-sized businesses under the guise of "big data" with the intention of providing applied mathematical tools is what is happening in the market.
Most data isn’t “big,” and businesses are wasting money pretending it is
11–20 of 160 posts
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#12> The “bigger” your data, the more false positives will turn up in it, when you’re looking for correlations
I think they are talking about the Sharpshooter Fallacy
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#13I think that big data has made math sexy, and selling applied statistics and operations research to small and medium-sized businesses under the guise of "big data" with the intention of providing applied mathematical tools is what is happening in the market.
Not a bad assessment of what seems to be going on.
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#14The reason the "big data" pimps can get away with this is that most of the people that should know, (that aren't DB programmers, DBAs, true scientists or engineers in the domain), don't know shit about data and generally too fscking lazy to learn. So they buy into the latest wave of buzz words and hype.
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#15Clarifying the scope of a project or data collection & analysis effort is paramount. You never want to attempt to boil the ocean. The key is to figure out the data the matters most per your company or organization's strategy.
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#16Most companies today are already using scaled up servers to host their medium size warehouses (think Teradata or Exadata). That approach is very expensive (> millions of dollars), only works well with well-defined data, and does not scale well beyond a few TBs.
Hadoop is not just about running large jobs on very large data. Hadoop also makes sense when trying to scale on commodity hardware or running ad hoc queries (which can target a small amount of data) on medium to large data sets.
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#17Sure if you're dealing with 1GB of data it probably isn't worth spinning up a Hadoop cluster to run your analysis. However, if you already have Hadoop up an running for something that genuinely requires it, that 1GB job might make sense there. The data may already be in HDFS, and you already have the infrastructure there to manage and monitor jobs. The references to Facebook & Yahoo running small jobs on huge cluster…
Like that huge firetruck used to put out small fires. Cities only need them for big fires, but, if you gotta have one and keep it ready, it makes sense to deploy it every time.
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#18I think that big data has made math sexy, and selling applied statistics and operations research to small and medium-sized businesses under the guise of "big data" with the intention of providing applied mathematical tools is what is happening in the market.
It's still amazing what businesses are able to accomplish with summing, counting, percentage of total, % change period over period, average, median, min, max.
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#19If I ever want to get rich, I'll set up shop convincing small businesses they need to do things the way Google does, if only they want to remain competitive.
Oracle has used exactly this business model to great success, and obscene profit, for over 30 years.
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#20I've maintained for awhile now that the distinction isn't between "big" and "small" data, but between coarse and fine data. Now that everything is done through the web, previously common data sources (surveys, sales summaries, etc) are being supplanted by microdata (web logs, click logs, etc). It does take a different skill set to analyze noisy, machine-generated data than to analyze clean, survey-like data; it's a skill set that is more biased towards computational knowledge than classical experimental design, hence the shift in emphasis.