Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most data isn’t “big,” and businesses are wasting money pretending it is

qz.com

Post reply on HN
Print view View original

101–110 of 160 posts

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#101

Post by flarg »

rmrfrmrf wrote: post #77
mattzito wrote: post #49

Earlier quoted context omitted.

This is a viewpoint that I hear a lot, mostly from people who are not in the room when these grand enterprise implementation decisions are made. While it's true that a good salesperson can make a difference in winning a deal vs. another vendor, salespeople almost never convince a company that they need a big enterprise software platform. 95% of the time, the company has already decided that the current way they do X…

> This means that instead of the organization saying, "We will adapt to off the shelf software and change our processes to better align with the way the software is designed to be used", they say, "Make your software work the way we do things". This must be a damned if you do, damned if you don't kind of situation, because I work for a company that attempted to use a OOTB Oracle software package and ended up getting…

(My experience has been that) most large companies lack defined processes and business models in critical areas because their models are too complex to fully define in software -- outside organisations know crap all about these complexities, and internally the skill sets are too low to implement a complex system --- and to be honest, who wants to spend three years and millions of dollars coding up an application for a business model that has already changed in that time? Sometimes, simple CRUD Apps are all that is needed.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#102

Post by StavrosK »

dasil003 wrote: post #89
maggit wrote: post #86

Earlier quoted context omitted.

I don't have a problem with relational databases. Or rather, I don't have a problem with relational data modelling. I do have a problem with Oracle. Even the Oracle experts at my former job could barely get Oracle to do something sensible, and running it on my own computer was basically a death sentence for getting anything done. I have a problem with MySQL, from a sysadmin perspective. When I had it installed, MySQL…

This is all pretty sane, except the schema-less part. I just don't understand why people get all hung up over schemas. Sure, migrations are a minor inconvenience, but if you just add fields in an ad-hoc fashion over time the data becomes messy and it's hard to determine any invariants about a large dataset. Sure this is avoided through careful code organization, but then aren't you just re-inventing schemas out-of-ba…

Yeah, I have experimented with schemaless stores a bit (mostly with JSON fields in postgres, for now), to avoid adding columns for various things to the database, and I am not impressed. Not only is initialization and migration hard, but I've come to realize that there's no downside in adding these to the schema itself.

Sure, the schema gets a bit bloated and dirty, but having undocumented fields in a dict whose existence is signified only by a line of code assigning something is not better.

Where schemaless stores are great is prototyping. Especially for more algorithmic code, where I don't really know what storage I'll be needing and the algorithm will only live there for a few days, schemas are just a burden. That's why I wrote Goatfish: https://github.com/stochastic-technologies/goatfish

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#103

Post by giulianob »

kamaal wrote: post #50

As some one who is currently dealing with these sort of things I can tell this article hits the nail on its head. Most, heck something like 99.99% of all so-called big data I've dealt is something I wouldn't even classify as small data. I've seen data feeds in KB's sent over to be handled in as big data. It happens all the time. A simple data problem sufficient enough to be easily solved on something like a small db…

I've also seen cases where people who run on EC2 end up having to do a lot of extra work because of the bad IO/CPU. They end up with these solutions that are way overkill for such small websites.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#104

Post by mathnode »

lifeisstillgood wrote: post #99
Djehngo wrote: post #97

Earlier quoted context omitted.

Ha, I knew that looked familiar, Hugh Darwin is a lecturer where I was at university so half of the coursework for a databases module was in Tutorial D.

Sorry I missed the reference - mathnode == Hugh Darwin ?

mathnode != Hugh Darwen

They are some big boots to fill. I'm just some DBA in London.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#105

Post by Choronzon »

cmccabe wrote: post #46

This article is the equivalent of "horse drawn carriages are perfectly adequate for most journeys, and much more pleasant and commodious to boot." Good luck with that, buddy. You're not going to know what correlations are important and which are not until you study the data. Telling people to just collect the "important data" is like telling someone who has lost his keys just to go back to where he left them. It's al…

The problem is your that your ability to explore the data and the data volume are inversely correlated.You are far more likely to find interesting things exploring an in-memory dataset using something like ipython and pandas than throwing pig jobs at a few dozen TB of gunk. Big data is great if you know exactly what you are looking for. If you get into a stage where you are trying to explore a huge DB looking for relationships your need to be very good at machine learning and statistical analysis (spurious correlations ahoy!) to come out significantly ahead.Its also an enormous time sink. In summation the bigger the data the simpler the analysis you can throw at it efficiently.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#106

Post by vasyainv »

jackmaney wrote: post #88
Twisol wrote: post #29

Earlier quoted context omitted.

"Buzzard" isn't an eggcorn I've ever heard before! Did you mean "buzz word"?

You have to take some of these colloquialisms with a grain assault.

I don't like to be kept dark and dry on this one

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#107

Post by vasyainv »

raverbashing wrote: post #94
dasil003 wrote: post #89

Earlier quoted context omitted.

This is all pretty sane, except the schema-less part. I just don't understand why people get all hung up over schemas. Sure, migrations are a minor inconvenience, but if you just add fields in an ad-hoc fashion over time the data becomes messy and it's hard to determine any invariants about a large dataset. Sure this is avoided through careful code organization, but then aren't you just re-inventing schemas out-of-ba…

So that you can store/retrieve data without knowing the schema. Oh but you have to know the schema right? Yes, some other part of the application knows the schema, but this part doesn't have authority over the DB. Also, the schema may be data as well. NoSQL reduces the work needed for that.

Perhaps, but can't you just organize your objects by having a table for each type and adding a column as needed? it doesn't sound like such a big deal

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#108

Post by ctrager »

Even if the data isn't big, there can be a benefit from the Hadoop infrastructure. Say you have just 86,400 rows of data but each row takes 1 second. That adds up to 24 hours of elapsed time, and waiting for that run can be painful, especially if you are trying to experiment, iterate. With HDFS/MapReduce you can distribute that work across N machines and divide the elapsed time by N, speeding up the pace of iteration. I've worked on a project that had exactly this challenge, before Hadoop was available, and so we had to invent our own crappy ways of distributing the data to the N machines, monitoring them, collecting the results. Hadoop HDFS and Map/Reduce, with Job Tracker, etc, would have been much better than what we came up with.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#109

Post by Choronzon »

guylhem wrote: post #27

For most data, it is in fact a waste of money. Personally, I am loading the data I play with on a postgreSQL database on my laptop (if you have a mac and want to do that quickly, you may want to check out the link I just submitted http://en.blog.guylhem.net/post/50310070182/running-postgres... ) You can do crazy things with the current hardware specs. Like loading all the data the world bank offers you to download, i…

SQLite is also an excellent option for a datastore on OSX.Its not nearly as full featured as postgres but no application is required and as you have a OS independent file per db which is extremely portable.SQLite Professional is a relatively decent free gui you can use also.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#110

Post by xradionut »

Choronzon wrote: post #109
guylhem wrote: post #27

For most data, it is in fact a waste of money. Personally, I am loading the data I play with on a postgreSQL database on my laptop (if you have a mac and want to do that quickly, you may want to check out the link I just submitted http://en.blog.guylhem.net/post/50310070182/running-postgres... ) You can do crazy things with the current hardware specs. Like loading all the data the world bank offers you to download, i…

SQLite is also an excellent option for a datastore on OSX.Its not nearly as full featured as postgres but no application is required and as you have a OS independent file per db which is extremely portable.SQLite Professional is a relatively decent free gui you can use also.

SQLite has limitations on the data types it supports. Most of this can be worked around by application code, but it can be a pain when you have data that needs to be accessable by more than one application.
Post reply on HN