Extrapolating relatively few truly random data points from massive datasets, for analysis and modeling, is what "Big Data" is all about. This article would have you think that working with clusters or snippets of impossibly ginormous datasets is somehow less "Big", but that's sorta the point. Perhaps somehow should inform the author that the more data available doesn't translate into working with more data.
Most data isn’t “big,” and businesses are wasting money pretending it is
21–30 of 160 posts
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#22[deleted]
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#23Be wary about drawing conclusions from "most of the jobs were small." Most of my jobs are small -- because I'm running experiments so I won't have to redo the big one.
That said, I'm a huge proponent of running stuff simply at first. Few businesses will ever grow to the point that they need more than a single large database server and one or two backups. Don't waste your time prepping for something you'll probably never need, especially when fixing the problem when the time comes is only marginally more painful than doing it right in the first place.
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#24If I ever want to get rich, I'll set up shop convincing small businesses they need to do things the way Google does, if only they want to remain competitive. Oracle has used exactly this business model to great success, and obscene profit, for over 30 years.
I know that's not what you mean, but I find it quite amusing that you describe Oracle (a 30 year old company)'s business as convincing people they need to do things the same way Google (a 15 year old company) does it.
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#25"We don't have big data. Our data is small, and could be easily stored in a MySQL or even a flat file" said no dev team ever. Everyone is "just like Google" so they need NoSQL, scaling, clouds and so on.
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#26I've seen a fair number of startups that throw around how they are going to make big money by utilizing the data they gather (called "big data" regardless of size) - it's all a bit of magical underpants thinking: we'll gather a bunch of people/users, we can't figure out how to make money off of advertising or charging them, so then we'll talk about how the "big data" they produce will be worth a fortune and people will pay to have access to it. Know some folks in the HR SaaS space that think this is how they'll hit $100m. It's just comedy.
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#27For most data, it is in fact a waste of money.
Personally, I am loading the data I play with on a postgreSQL database on my laptop (if you have a mac and want to do that quickly, you may want to check out the link I just submitted http://en.blog.guylhem.net/post/50310070182/running-postgres... )
You can do crazy things with the current hardware specs. Like loading all the data the world bank offers you to download, index it and use it for regressions (I do). In 2013 you only need a laptop for that.
Most data is not big. Big data is "big" like in a gold rush, where the ones selling the tools are making the biggest profits.
EDIT: Thanks for the postgresapp.com link! It is a little bit diffent- here I wanted to use the very same sources as Apple, without adding too much cruft (like a UI to start/stop the daemon as I had seen in other packages). I also wanted to see by myself how hard it was to 'make it work' with OSX (quite easy besides the missing AEP.make and the logfile error). It was basically an experiment in recompiling from the sources given by apple opensource website, while staying as close to the OSX spirit as possible (ex: keeping the same user group, using dscl, using launchdaemon to start the daemon automatically during the boot sequence like for Apache)
That being said, you're right, for most people postgresapp.com will be a simpler and faster way to run a postgresql server :-)
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#28If I ever want to get rich, I'll set up shop convincing small businesses they need to do things the way Google does, if only they want to remain competitive. Oracle has used exactly this business model to great success, and obscene profit, for over 30 years.
I know that's not what you mean, but I find it quite amusing that you describe Oracle (a 30 year old company)'s business as convincing people they need to do things the same way Google (a 15 year old company) does it.
He said they have the same business model. Business models are abstract.
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#29Sure if you're dealing with 1GB of data it probably isn't worth spinning up a Hadoop cluster to run your analysis. However, if you already have Hadoop up an running for something that genuinely requires it, that 1GB job might make sense there. The data may already be in HDFS, and you already have the infrastructure there to manage and monitor jobs. The references to Facebook & Yahoo running small jobs on huge cluster…
"Buzzard" isn't an eggcorn I've ever heard before! Did you mean "buzz word"?
Re: Most data isn’t “big,” and businesses are wasting money pretending it is
#30I think that big data has made math sexy, and selling applied statistics and operations research to small and medium-sized businesses under the guise of "big data" with the intention of providing applied mathematical tools is what is happening in the market.
Statistics involves checking modeling assumptions. A lot of what I've seen with the big data people is the repetition of algorithms to the exclusion of understanding and checking modeling assumptions.
While it's nice that the big data craze is making statistics more popular in the mainstream press, it is important that statistics does not become just an application of numerical methods without consideration of underlying assumptions. I stress this because this has been largely underappreciated in my experience.