Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most data isn’t “big,” and businesses are wasting money pretending it is

qz.com

Post reply on HN
Print view View original

121–130 of 160 posts

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#121

Post by IsaacL »

Djehngo wrote: post #97
mathnode wrote: post #95

Earlier quoted context omitted.

Hanging off this great comment, please anyone wanting to learn about relational databases I strongly suggest reading Database in Depth by C J Date. It's around 200 pages, you will learn a lot about the relational model, database/query/index design in a short amount of time, with a little bit of logic thrown in there too. Just need a good SQL reference? 'The Art of SQL' or 'SQL and Relational Theory' Are you not enter…

Ha, I knew that looked familiar, Hugh Darwin is a lecturer where I was at university so half of the coursework for a databases module was in Tutorial D.

Funny to see another ex-Warwickian here. Yes, I remember that coursework - I also found in interesting that Hugh Darwin thought SQL was too lax and flexible. I wish I'd taken the chance to ask his opinions on nosql.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#122

Post by VLM »

sergiosgc wrote: post #116
derefr wrote: post #98

Earlier quoted context omitted.

> but if you just add fields in an ad-hoc fashion over time the data becomes messy and it's hard to determine any invariants about a large dataset In systems large enough to be running multiple versions of an app at the same time, talking to the same database, you have to do exactly this--but you have to shoehorn it into relations (doing things like having a field where 99.999% of entries in a column are null, and gr…

No. NoSQL lets you say nothing about the schema, and so it becomes a problem for the application layer, above the DB, to handle. In fact, this is much what happens with most "solutions" NoSQL presents to SQL database problems: Let's not implement it, then it's not a problem. What happens when you push problems up the stack? Do they get solved automatically? No? Will they get solved? Perhaps, if really needed. And, fo…

"Let's not implement it, then it's not a problem."

Its interesting that culturally more than a decade ago, when mysql tried this strategy with transactions, namely, not having them until roughly the turn of the century, it was reviled mostly by people who don't know what transactions are nor did they need them, but were nonetheless very unhappy about mysql not having that checkbox get checked.

Now its culturally seen as a huge win to simply not implement something difficult.

I don't know if its a decline in feature list length as a fetish or just simple copy catting of others behavior (perhaps in both situations) or some kind of pull yourself up by your bootstraps romantic outlook on reimplementation or the inevitable result of homer simpson meets the database, but whatever it is, its an interesting major cultural change.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#123

Post by VLM »

dspillett wrote: post #100
mathnode wrote: post #95

Earlier quoted context omitted.

Hanging off this great comment, please anyone wanting to learn about relational databases I strongly suggest reading Database in Depth by C J Date. It's around 200 pages, you will learn a lot about the relational model, database/query/index design in a short amount of time, with a little bit of logic thrown in there too. Just need a good SQL reference? 'The Art of SQL' or 'SQL and Relational Theory' Are you not enter…

And once you know the basics, I strongly recommend "SQL Anti-Patterns" for a good guide for what not to do and why not (and what mitigating circumstances might make an otherwise bad choice OK (or simply the only available option)). I read it while considering myself experienced and found it to be a useful refresher. The style/tone is light and well organised by task/objective, so I suspect everyone down to a beginner…

"I read it while considering myself experienced and found it to be a useful refresher."

I'll be honest and say I read it and found it terribly embarrassing yet comforting. Remember that dumb thing I did back in '96? (insert red face) Yeah I guess I'm not the only guy to learn that the hard way. That lack of deep experience is a significant danger of nosql designs. The folks doing that now, don't even see the icebergs that relational folks successfully dodged decades ago. Much better off being nostalgic about the olden days of steam engine trains than not even seeing the diesel-electric headlight at the end of the tunnel rushing toward you.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#124

Post by yummyfajitas »

VLM wrote: post #120
yummyfajitas wrote: post #115

Earlier quoted context omitted.

In more ad-hoc, constantly changing, "ops this didn't work because of unknown factor X" type of projects, schemas are a pain. Such a pain: ALTER TABLE foo DROP COLUMN bar; ALTER TABLE foo ADD COLUMN baz varchar(64); From what I've seen, a significant chunk of the desire to use schemaless NoSQL hipster DBs is simply a desire to avoid learning SQL as if it were a real programming language. The only real use case I've e…

"lots of ad-hoc columns added by multiple people" Inevitably completely un-normalized and junk data. Even worse with no documentation or procedure anything ever added becomes permanent legacy that can never be removed. Been there, lived it, hated it, won't allow it to happen again.

So you won't allow logfiles to happen again?

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#125

Post by VLM »

dizzystar wrote: post #41
lmg643 wrote: post #18

Earlier quoted context omitted.

It's still amazing what businesses are able to accomplish with summing, counting, percentage of total, % change period over period, average, median, min, max.

It's even more amazing how few businesses are able to compute those operations.

Add in bonuses based on those numbers and its amazing any consistency exists in their calculation. Basically in practice you're only allowed a consistent and analytically defensible system if no ones bonus depends on the process being obfuscated. This is why a lot of "big data" is oriented around generating new ideas and new numbers, rather than fixing existing systems and data...

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#126

Post by VLM »

yummyfajitas wrote: post #124
VLM wrote: post #120

Earlier quoted context omitted.

"lots of ad-hoc columns added by multiple people" Inevitably completely un-normalized and junk data. Even worse with no documentation or procedure anything ever added becomes permanent legacy that can never be removed. Been there, lived it, hated it, won't allow it to happen again.

So you won't allow logfiles to happen again?

LOL, no, not as a primary store of data, no never again.

Plain text logs are a great place to funnel all "unable to connect to database" type of errors for dbas / sysadmins to ponder, however.

I've implemented quite a few systems where data changes are logged into a log table, all fully rationalized, so various reports can be generated about discrepancies and access rates and stuff like that. This is also cool for endusers to see who made the last change etc.

Trying to reverse engineer how some data got into a messed up condition using logs that can be JOINed to actual data tables as necessary is pretty easy, compared to trying to write a webserver log file parser to read multiple files to figure out who did what, when, to the data resulting in it being screwed up. You parse log files for data just once before you decide to do that stuff relationally. Debug time drops by a factor of 100x.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#127

Post by scorpion032 »

As also some one that has been in the thick of some of the "big data" projects in the industry recently, I have to agree with the article.

One of the terms I learnt in the PyData Silicon Valley in March is "Medium Data". Unless you are dealing with terabytes of RAM and Exa bytes of storage, google style, the overhead of having to maintain a cluster is something most (intelligent) people try to avoid.

When you cant avoid hundreds of machines, the cluster is a necessity and you design that way. But given where the Moore's law curve stands today, most organisations really dont need that.

You can buy servers on Amazon with 250 gigs of RAM for a few dollars an hour. They specifically call it the big data cluster. It is possible to analyse the data using tools like Pandas/Matplotlib and others in the Scientific Python eco system fairly easily.

These tools are being used by scientists and industry for a really long time, except they aren't really advertised that way.

For instance, here is some analysis I was doing recently of the children names in the US, from 1880, with 3 million records: http://nbviewer.ipython.org/53ec0c5a2fabcfebb358. My Mac could handle it without even breaking a sweat.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#128

Post by DenisM »

VLM wrote: post #126
yummyfajitas wrote: post #124

Earlier quoted context omitted.

So you won't allow logfiles to happen again?

LOL, no, not as a primary store of data, no never again. Plain text logs are a great place to funnel all "unable to connect to database" type of errors for dbas / sysadmins to ponder, however. I've implemented quite a few systems where data changes are logged into a log table, all fully rationalized, so various reports can be generated about discrepancies and access rates and stuff like that. This is also cool for en…

I'd like to pick your brain as that's the problem I'm facing right now - I have a web site that is accessed by users, and I would like to get a comprehensive picture of what they do. I already have a log table for logging all changes (as you said - I can show it to the users themselves so they know who hanged what in a collaborative environment), but I struggle defining meaningful way to log read access - should I record every hit? Would that be too much data to store and process later? Should I record some aggregates instead?

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#129

Post by physicslover »

maggit wrote: post #86
rmrfrmrf wrote: post #79

Earlier quoted context omitted.

I really don't understand why anyone has a problem with relational databases. Once you take the time to understand how they work (by taking a class or reading a book), it's really straightforward and makes a lot of sense. Not to mention it's really fast and quite reliable. I get that a NoSQL-ish alternative makes sense for companies that have tons of shards spanning the globe, but for the vast majority of people, a r…

I don't have a problem with relational databases. Or rather, I don't have a problem with relational data modelling. I do have a problem with Oracle. Even the Oracle experts at my former job could barely get Oracle to do something sensible, and running it on my own computer was basically a death sentence for getting anything done. I have a problem with MySQL, from a sysadmin perspective. When I had it installed, MySQL…

Have to agree with the above. The problem with relational databases isn't the relational model, per se, but the complexity and cost of maintaining a relational database.

Typically, they require specialized database administrators whose primary job is to tune the database and keep it running.

Many businesses, even of moderate size, reach point where they will need to purchase expensive hardware (million dollar RamSans and expensive servers) to optimize the performance of their database because partitioning databases is challenging.

So the overhead of running an Oracle or Sql Server database is quite high.

There is huge room for improvement with these traditional database products. If someone made a good cloud database that supported the same feature set but with lower administration and maintenance costs then that might be a better option.

Re: Most data isn’t “big,” and businesses are wasting money pretending it is

#130

Post by gizzlon »

VLM wrote: post #122
sergiosgc wrote: post #116

Earlier quoted context omitted.

No. NoSQL lets you say nothing about the schema, and so it becomes a problem for the application layer, above the DB, to handle. In fact, this is much what happens with most "solutions" NoSQL presents to SQL database problems: Let's not implement it, then it's not a problem. What happens when you push problems up the stack? Do they get solved automatically? No? Will they get solved? Perhaps, if really needed. And, fo…

"Let's not implement it, then it's not a problem." Its interesting that culturally more than a decade ago, when mysql tried this strategy with transactions, namely, not having them until roughly the turn of the century, it was reviled mostly by people who don't know what transactions are nor did they need them, but were nonetheless very unhappy about mysql not having that checkbox get checked. Now its culturally seen…

I see it more as a move from "one size fits all" to more specialized tools. The term "nosql" is pretty useless as it's way too general. Both your comments about "not implementing the hard stuff", and GP's about schema, only applies to some of the "nosql" projects.

Instead of looking at it as "aaawwm! NoSQL is attacking our bellowed RDBMS", try looking at the different projects and what they bring to the table. Maybe some of them can be a useful addition to your systems.

Post reply on HN