41–50 of 160 posts
I think that big data has made math sexy, and selling applied statistics and operations research to small and medium-sized businesses under the guise of "big data" with the intention of providing applied mathematical tools is what is happening in the market.
It's still amazing what businesses are able to accomplish with summing, counting, percentage of total, % change period over period, average, median, min, max.
For most data, it is in fact a waste of money. Personally, I am loading the data I play with on a postgreSQL database on my laptop (if you have a mac and want to do that quickly, you may want to check out the link I just submitted http://en.blog.guylhem.net/post/50310070182/running-postgres... ) You can do crazy things with the current hardware specs. Like loading all the data the world bank offers you to download, i…
I think this is probably even quicker than the steps you provided (though you have to remember to start it manually, rely on them updating the build, etc) http://postgresapp.com/
I think that big data has made math sexy, and selling applied statistics and operations research to small and medium-sized businesses under the guise of "big data" with the intention of providing applied mathematical tools is what is happening in the market.
Statistics involves checking modeling assumptions. A lot of what I've seen with the big data people is the repetition of algorithms to the exclusion of understanding and checking modeling assumptions. While it's nice that the big data craze is making statistics more popular in the mainstream press, it is important that statistics does not become just an application of numerical methods without consideration of underl…
For most data, it is in fact a waste of money. Personally, I am loading the data I play with on a postgreSQL database on my laptop (if you have a mac and want to do that quickly, you may want to check out the link I just submitted http://en.blog.guylhem.net/post/50310070182/running-postgres... ) You can do crazy things with the current hardware specs. Like loading all the data the world bank offers you to download, i…
You're not going to know what correlations are important and which are not until you study the data. Telling people to just collect the "important data" is like telling someone who has lost his keys just to go back to where he left them.
It's also more than a little insulting to FB and Yahoo to insist they are not web scale. The problem of small jobs on MR clusters is real, but even with small jobs, Hadoop turns out to be a lot more cost-effective than various other proprietary solutions which are your only real enterprise alternative. The problem of small MR jobs is being solved by things like Cloudera Impala, which can run on top of raw HDFS to perform interactive queries.
This article is the equivalent of "horse drawn carriages are perfectly adequate for most journeys, and much more pleasant and commodious to boot." Good luck with that, buddy. You're not going to know what correlations are important and which are not until you study the data. Telling people to just collect the "important data" is like telling someone who has lost his keys just to go back to where he left them. It's al…
The most important thing is knowing what data you have, how best to collect it, and what it can (and can't) tell you. Just because you find correlations doesn't mean that they are real. It takes people with real expertise to help here, and just running your data on a cluster isn't going to help you. In fact, it could even hurt.
I didn't see anything wrong with the article at all.
If I ever want to get rich, I'll set up shop convincing small businesses they need to do things the way Google does, if only they want to remain competitive. Oracle has used exactly this business model to great success, and obscene profit, for over 30 years.
I know that's not what you mean, but I find it quite amusing that you describe Oracle (a 30 year old company)'s business as convincing people they need to do things the same way Google (a 15 year old company) does it.
Oracle writes shitty 'enterprise apps' (god I hate that phrase) that they sell to big companies, because their salesmen/women wear great attire and are good at mirroring dumb ceos/cios, like the ones that run several companies I have worked for. Will someone please end this nonsense? At what point does usability/stability/utility become factors?
The truth is that very often, X is broken inside an organization not because of executive management, most of whom don't care what software packages get used or who they buy from or anything else like that, but rather big software companies get brought in because the technology/backoffice organization inside the company is a disaster.
Accounting system doesn't properly allocate widget expenses to different cost centers? Takes a week to update the homepage? No one knows where exactly sensitive data is being stored?
That's all the technology organization's failure in one way or another. And when things get bad enough, senior management says, "Okay, our homegrown accounting system is just not doing the job for us anymore", and here comes Oracle, happy to sell them their accounting system, which has all of the features they could possibly want, and sure, it's expensive, but it works, as opposed to the busted system they've got currently.
Of course, the next failure then, is that the people who will be running and overseeing and architecting this solution are either the same people who cocked up the accounting system in the first place, or consultants who have absolutely zero incentive to do anything other than maximize billable hours.
This means that instead of the organization saying, "We will adapt to off the shelf software and change our processes to better align with the way the software is designed to be used", they say, "Make your software work the way we do things".
Now we're off to the races, as various fiefdoms inside of the big company make their pitch about what needs to be customized. Everything from the layout of the screens to the workflow processes to the data model, everything has to be matched to exactly the way the customer wants to do things.
Back at Oracle HQ, the RFEs have been flying in from not just that customer, but the other 200 new customers being onboarded , and every one is basically a demand for a way to modify this or that option - no one is saying, "We wish there were fewer fields on this page"
So the customers demand more features, Oracle delivers them, and then the customers promptly use those features to further complicate their platforms, because they don't have the technical discipline to say, "No, we really don't need to support different SKU revenue allocations based on currency, we'll just do it by hand at the end of every quarter".
Looking at it a different way - how is making the software simpler going to help Oracle win business? If anything, the more features the product has, the more points they get on the RFP from the next big customer.
So everyone is to blame - Oracle makes money selling and implementing very complex technology solutions because they're answering the demands of their customers who depend on overly complex technical requirements because their technology organizations are poorly run because they don't have any discipline because senior management isn't technical enough to recognize where the failure is.
tl;dr - enterprise software is not broken because of the sales people or upper management, it's broken because the technology organizations are bad at their jobs
Most, heck something like 99.99% of all so-called big data I've dealt is something I wouldn't even classify as small data. I've seen data feeds in KB's sent over to be handled in as big data. It happens all the time. A simple data problem sufficient enough to be easily solved on something like a small db solution like sqlite is generally taken to 'grid' these days. It reminds me of the XML days when everything had to be XML. I mean every damn thing, these days its NoSQL and Big data.
People wrongly do their schema design just so that it can get into a NoSQL, then use something like Pig to generate data for it. The net result is they end up badly reinventing parts of SQL all over the place. If only they understand a little SQL and why its exists they can save themselves all that pointless complexity they get into. Besides avoiding to use SQL where its appropriate creates all sorts of data problems in your system. You will go endlessly reinventing ways doing things similar to what SQL offers while bloating your code. You will go reading a big part of the code, only to figure out the person actually intended to something like a nested select query albeit here very badly.
Besides I find much of this big data thing a total sham. Back in the yester years we would write Perl scripts to do all sorts complex data processing(With SQL of course). Heck I've run some very big analytic systems, and automation set ups in Perl to do far difficult things people do using 'Big data tools' today.
In larger corporation this has become fashion now. If you want to be known as a great 'architect' all you need to do is bring in these pointless complexities. Ensure the set up becomes so complicated it can't explained without the help of a hundred jargons totally incomprehensible to anybody beyond your cubicle. That is how you get promoted to become a architect these days.