Most data isn’t “big,” and businesses are wasting money pretending it is • Hacker News Board

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#111

Post by sytelus » Mon, May 13, 2013, 12:40 PM UTC

As someone who runs jobs on giant clusters day in day out, I just looked at my last job. It indeed had input data of ~100GB. However size of input data is misleading. Job does a lot of processing and generates ~5TB of intermediate data and it took 800+ machine hours to complete. If I'd ran that on my desktop I would be waiting for a month to finish. On cluster it took ~4 hours.

I'd to smile at the statement "Is more data always better? Hardly". There is old saying the world of data scientists: There is no data like more data. Yes, the value of it may be diminishing but when your competitor is trying to squeeze out gain in second decimal, you are probably better off accepting more data.

So the moral of the story is, all these really depends. People do get fired for buying clusters. Modern cluster management software track several utilization metrics and someone some day would going to look at it and point out how bad decision it was.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#112

Post by zapov » Mon, May 13, 2013, 12:43 PM UTC

matwood wrote: post #82

rmrfrmrf wrote: post #79

Earlier quoted context omitted.

I really don't understand why anyone has a problem with relational databases. Once you take the time to understand how they work (by taking a class or reading a book), it's really straightforward and makes a lot of sense. Not to mention it's really fast and quite reliable. I get that a NoSQL-ish alternative makes sense for companies that have tons of shards spanning the globe, but for the vast majority of people, a r…

I grew up on rdbmses and think they are great. I've created numerous pieces of software backed by them. I have long preached the mantra that data is more important than the application. The data will almost always outlive the original application and become shared to others. With that said, the above comes at a time/flexibility cost (not as bad as it used to be but still there) when building a product that isn't quit…

Time/flexibility problem can be solved (but it's hard). Take a look at DSL Platform (shameless plug) if you are interested in building on top of database, while having very flexible model.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#113

Post by raverbashing » Mon, May 13, 2013, 12:50 PM UTC

vasyainv wrote: post #107

raverbashing wrote: post #94

Earlier quoted context omitted.

So that you can store/retrieve data without knowing the schema. Oh but you have to know the schema right? Yes, some other part of the application knows the schema, but this part doesn't have authority over the DB. Also, the schema may be data as well. NoSQL reduces the work needed for that.

Perhaps, but can't you just organize your objects by having a table for each type and adding a column as needed? it doesn't sound like such a big deal

Looks like fun

But why would I waste developer time doing that if I can only do db.table.insert(obj) - in MongoDB for example (obj is a JS object)

Also, finding all objects with a field named 'field1' and value '3' is slower if you do that in a relational DB (and that's the simplest case)

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#114

Post by mattzito » Mon, May 13, 2013, 12:55 PM UTC

rmrfrmrf wrote: post #77

mattzito wrote: post #49

Earlier quoted context omitted.

This is a viewpoint that I hear a lot, mostly from people who are not in the room when these grand enterprise implementation decisions are made. While it's true that a good salesperson can make a difference in winning a deal vs. another vendor, salespeople almost never convince a company that they need a big enterprise software platform. 95% of the time, the company has already decided that the current way they do X…

> This means that instead of the organization saying, "We will adapt to off the shelf software and change our processes to better align with the way the software is designed to be used", they say, "Make your software work the way we do things". This must be a damned if you do, damned if you don't kind of situation, because I work for a company that attempted to use a OOTB Oracle software package and ended up getting…

Yeah, it very much is a tough row to hoe on either side - and btw, even just adapting to the OOTB package will still cost you a ton of money, and often in ways you didn't expect:

I was loosely associated a few years back with a manufacturing company migrating from their 20 year-old mainframe-based ERP solution to Oracle's ERP. They really had the worst of both worlds, because not only did they have 20 year-old business processes that no one wanted to change, but the whole interface for Oracle was so radically different from the "green screen" 3270 interface of the current system that you couldn't even make Oracle look anything like that. It was doomed to be a complete mess.

But to the point I'd originally planned to make, they tested the system in limited release, and then went live with it for one particular function, which was generating and printing order cards or something like that. What no one had thought of, and didn't occur in testing because it wasn't a real workload was that the old system sent raw text to the printers at the various factory sites, while Oracle (iirc) was generating postscript, complete with logos and formatting, and sending that to the printers at the factories....which it turns out were connected over 128kb/sec links that were promptly swamped by the size of the files.

So the whole project had to be put on hold until all of the links between HQ and the factories could be upgraded, which took months, and the feedback from the userbase was, "What a piece of shit Oracle is, our 20 year old system can print to the factories, why is it so hard for them to do that?!?!"

EDIT: looked back in my notes, 128kb/sec lines, not 512

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#115

Post by yummyfajitas » Mon, May 13, 2013, 1:12 PM UTC

gizzlon wrote: post #90

dasil003 wrote: post #89

Earlier quoted context omitted.

This is all pretty sane, except the schema-less part. I just don't understand why people get all hung up over schemas. Sure, migrations are a minor inconvenience, but if you just add fields in an ad-hoc fashion over time the data becomes messy and it's hard to determine any invariants about a large dataset. Sure this is avoided through careful code organization, but then aren't you just re-inventing schemas out-of-ba…

I think this is due to how you work and what you build. If you plan it out, focus on the data structures you need and then build it, schemas are fine because you know up-front what you want. In more ad-hoc, constantly changing, "ops this didn't work because of unknown factor X" type of projects, schemas are a pain. It's sounds really nice to have a data store that adapts when it's impossible to know up-front what you…

In more ad-hoc, constantly changing, "ops this didn't work because of unknown factor X" type of projects, schemas are a pain.

Such a pain:

    ALTER TABLE foo DROP COLUMN bar;
    ALTER TABLE foo ADD COLUMN baz varchar(64);

From what I've seen, a significant chunk of the desire to use schemaless NoSQL hipster DBs is simply a desire to avoid learning SQL as if it were a real programming language.

The only real use case I've ever seen for schemaless DBs is fields with lots of ad-hoc columns added by multiple people (typically logging/metrics data).

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#116

Post by sergiosgc » Mon, May 13, 2013, 1:31 PM UTC

derefr wrote: post #98

dasil003 wrote: post #89

Earlier quoted context omitted.

This is all pretty sane, except the schema-less part. I just don't understand why people get all hung up over schemas. Sure, migrations are a minor inconvenience, but if you just add fields in an ad-hoc fashion over time the data becomes messy and it's hard to determine any invariants about a large dataset. Sure this is avoided through careful code organization, but then aren't you just re-inventing schemas out-of-ba…

> but if you just add fields in an ad-hoc fashion over time the data becomes messy and it's hard to determine any invariants about a large dataset In systems large enough to be running multiple versions of an app at the same time, talking to the same database, you have to do exactly this--but you have to shoehorn it into relations (doing things like having a field where 99.999% of entries in a column are null, and gr…

No. NoSQL lets you say nothing about the schema, and so it becomes a problem for the application layer, above the DB, to handle. In fact, this is much what happens with most "solutions" NoSQL presents to SQL database problems: Let's not implement it, then it's not a problem.

What happens when you push problems up the stack? Do they get solved automatically? No? Will they get solved? Perhaps, if really needed. And, for the cherry on top: Will the solutions be similar to the ones SQL databases use? They will.

You see, when you are implementing atomic transactions, for example, you may get ahead if you have some information about the problem domain. However, for most cases, you are solving the same problem SQL databases solved decades ago. And you'll find the same solution. Just not as well implemented nor as well tested.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#117

Post by ims » Mon, May 13, 2013, 1:34 PM UTC

Choronzon wrote: post #105

cmccabe wrote: post #46

This article is the equivalent of "horse drawn carriages are perfectly adequate for most journeys, and much more pleasant and commodious to boot." Good luck with that, buddy. You're not going to know what correlations are important and which are not until you study the data. Telling people to just collect the "important data" is like telling someone who has lost his keys just to go back to where he left them. It's al…

The problem is your that your ability to explore the data and the data volume are inversely correlated.You are far more likely to find interesting things exploring an in-memory dataset using something like ipython and pandas than throwing pig jobs at a few dozen TB of gunk. Big data is great if you know exactly what you are looking for. If you get into a stage where you are trying to explore a huge DB looking for rel…

Very true. Wouldn't the typical approach to this involve probabilistic methods like taking large-ish (but not "Big") samples from your multi TB data and doing your EDA with those?

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#118

Post by ims » Mon, May 13, 2013, 1:47 PM UTC

philip1209 wrote: post #11

I think that big data has made math sexy, and selling applied statistics and operations research to small and medium-sized businesses under the guise of "big data" with the intention of providing applied mathematical tools is what is happening in the market.

This is exactly right. I'm a member of INFORMS (the operations research professional society), and I can report that a staggering amount of ink has been spilled over the last few years about how to capitalize on the recent "Analytics" and "Big Data" trends.

On the one hand, people are starting to realize that quantitative analysis can help their businesses (mind blowing, right?) -- on the other hand, so much of what you see about "analytics" and "big data" is nonsensical jargon. You have two camps within the OR world: people who want to ride this bandwagon all the way to the bank, and people who want to refocus on getting the message out about what OR really is.

The bandwagon-riders have succeeded to some extent. INFORMS created a monthly "Analytics" magazine[1], created an Analytics Certification[2] (their first professional certification), and so on.

The other camp has a legitimate concern that OR already has an "identity crisis" (operations research vs. management science vs. systems engineering vs. industrial engineering vs. applied math vs. applied statistics etc etc). INFORMS has spent millions trying to get business people to just be aware that it exists. The fear is that hitching our wagon to these trends will just be another blow to our profile when these fad words are replaced by the next big thing.

[1] http://analytics-magazine.org/ (you can get a good feel for the type of content in this publication just by reading the article titles...)

[2] https://www.informs.org/Build-Your-Career/Analytics-Certific...

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#119

Post by dasil003 » Mon, May 13, 2013, 2:02 PM UTC

gizzlon wrote: post #90

dasil003 wrote: post #89

Earlier quoted context omitted.

This is all pretty sane, except the schema-less part. I just don't understand why people get all hung up over schemas. Sure, migrations are a minor inconvenience, but if you just add fields in an ad-hoc fashion over time the data becomes messy and it's hard to determine any invariants about a large dataset. Sure this is avoided through careful code organization, but then aren't you just re-inventing schemas out-of-ba…

I think this is due to how you work and what you build. If you plan it out, focus on the data structures you need and then build it, schemas are fine because you know up-front what you want. In more ad-hoc, constantly changing, "ops this didn't work because of unknown factor X" type of projects, schemas are a pain. It's sounds really nice to have a data store that adapts when it's impossible to know up-front what you…

> It's sounds really nice to have a data store that adapts when it's impossible to know up-front what you need from your data structures.

This seems backwards to me. Relational databases are much better at ad-hoc querying of data, whereas NoSQL scales for narrower access patterns. The fact that you can dump arbitrary columns without a migration is a nice convenience, but in general it will be less queryable than it would be if you added the column in a SQL database.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#120

Post by VLM » Mon, May 13, 2013, 2:24 PM UTC

yummyfajitas wrote: post #115

gizzlon wrote: post #90

Earlier quoted context omitted.

I think this is due to how you work and what you build. If you plan it out, focus on the data structures you need and then build it, schemas are fine because you know up-front what you want. In more ad-hoc, constantly changing, "ops this didn't work because of unknown factor X" type of projects, schemas are a pain. It's sounds really nice to have a data store that adapts when it's impossible to know up-front what you…

In more ad-hoc, constantly changing, "ops this didn't work because of unknown factor X" type of projects, schemas are a pain. Such a pain: ALTER TABLE foo DROP COLUMN bar; ALTER TABLE foo ADD COLUMN baz varchar(64); From what I've seen, a significant chunk of the desire to use schemaless NoSQL hipster DBs is simply a desire to avoid learning SQL as if it were a real programming language. The only real use case I've e…

"lots of ad-hoc columns added by multiple people"

Inevitably completely un-normalized and junk data. Even worse with no documentation or procedure anything ever added becomes permanent legacy that can never be removed. Been there, lived it, hated it, won't allow it to happen again.