Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

Post reply on HN
Print view View original

91–100 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#91

Post by misterman0 »

StreamBright wrote: post #85
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

Wait, what problem does Hadoop solve?? Hadoop is an extremely poor reimplementation of some Google services. The time when billions went into Hadoop startups even Google stopped using MapReduce because it turned out to be inefficient and very limited to many of the distributed computational problems. MR was extremely well suited for a single thing: aggregating web logs and computing very simple summary statistics. Wh…

>> Google stopped using MapReduce

How are they then querying over big data these days?

We don't know, do we? Or did they open-source their search engine?

By using Hadoop people are trying to not reinvent the big data wheel, partly because it's a motherfucker of a problem to have to solve and party because they want to solve the business problem, not the technical one. I don't see how that is in any way worthy of being frowned upon.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#92

Post by thrower123 »

danpalmer wrote: post #60
geezerjay wrote: post #38

Earlier quoted context omitted.

> Hadoop solved a hard problem that nobody had. This baseless assertion is patently wrong on so many levels. Building computing clusters on COTS hardware is a very mundane problem. Running processing jobs on data shards is a very mundane problem. Scaling COTS clusters transparently is a very mundane problem.

There are broadly 3 levels of the amount of data that people have: - Fits on one computer (most of the market) - Fits on several computers (most of the rest) - Requires a significant cluster of machines (50+ to store it) Hadoop only really solves the last one. It has huge overheads in terms of speed and in terms of resources and headcount to run it properly, so it only makes sense at a particular scale. It's like a m…

> - Fits on one computer (most of the market)

There's one category above that, which is "Fits in memory" and that is a huge chunk of the market. I've seen first hand people getting way too cute and complicated planning for scale, and then it works out that they don't even have more than a couple of GB of data.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#93

Post by Bootvis »

otabdeveloper1 wrote: post #62
eeZah7Ux wrote: post #52

Earlier quoted context omitted.

> A slightly smarter .tar.gz would have solved the problem just as well. It's called "OS package" ;) and can provide more strict sandboxing using a systemd unit file: unit files provide seccomp, cgroups and more.

Indeed you're right, but the problem is that your devs' machines and your production systems are running different OS's/distributions. Nix tries to solve this, but it isn't there just yet.

I know there is cost to this solution but it's a good one:

Use the same OS and similar hardware for development and production.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#94

Post by StreamBright »

misterman0 wrote: post #91
StreamBright wrote: post #85

Earlier quoted context omitted.

Wait, what problem does Hadoop solve?? Hadoop is an extremely poor reimplementation of some Google services. The time when billions went into Hadoop startups even Google stopped using MapReduce because it turned out to be inefficient and very limited to many of the distributed computational problems. MR was extremely well suited for a single thing: aggregating web logs and computing very simple summary statistics. Wh…

>> Google stopped using MapReduce How are they then querying over big data these days? We don't know, do we? Or did they open-source their search engine? By using Hadoop people are trying to not reinvent the big data wheel, partly because it's a motherfucker of a problem to have to solve and party because they want to solve the business problem, not the technical one. I don't see how that is in any way worthy of bein…

Maybe you and the downvoter crew do not know it.

https://www.datacenterknowledge.com/archives/2014/06/25/goog...

https://www.quora.com/Why-did-Google-stop-using-MapReduce-an...

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#95

Post by bdavis__ »

eeZah7Ux wrote: post #52
otabdeveloper1 wrote: post #50

Earlier quoted context omitted.

90% of the time Docker is used to solve the problem of "how do I upload this bucket of Python crud to a production server?" (Replace 'Python' with any other language to taste.) A slightly smarter .tar.gz would have solved the problem just as well.

> A slightly smarter .tar.gz would have solved the problem just as well. It's called "OS package" ;) and can provide more strict sandboxing using a systemd unit file: unit files provide seccomp, cgroups and more.

docker solves 2 problems. first is you have no control over your devs and allow them to install any software from anywhere. and second is you want to sell cpu time from the cloud in an efficient way (for the seller).

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#96

Post by StreamBright »

otabdeveloper1 wrote: post #50
pytester wrote: post #25

Earlier quoted context omitted.

>Docker solves a simple problem that everyone has. Docker provides an (IMHO pretty buggy) isolation layer that lies between "keeping things that need to be kept separate in separate folders" and "keeping things that need to be kept separate in separate virtual machines". I actually don't have the need for the level of isolation below VM and above folder very often. IMHO this level only really makes sense when contain…

90% of the time Docker is used to solve the problem of "how do I upload this bucket of Python crud to a production server?" (Replace 'Python' with any other language to taste.) A slightly smarter .tar.gz would have solved the problem just as well.

Exactly. The said reality that solving this problem helps many developers who would otherwise not capable of deploying to servers.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#97

Post by StreamBright »

bdavis__ wrote: post #95
eeZah7Ux wrote: post #52

Earlier quoted context omitted.

> A slightly smarter .tar.gz would have solved the problem just as well. It's called "OS package" ;) and can provide more strict sandboxing using a systemd unit file: unit files provide seccomp, cgroups and more.

docker solves 2 problems. first is you have no control over your devs and allow them to install any software from anywhere. and second is you want to sell cpu time from the cloud in an efficient way (for the seller).

Agree with the first one but disagree with the second one. EC2 was selling CPU time long time ago before Docker existed.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#98

Post by kevin_thibedeau »

pytester wrote: post #39
sz4kerto wrote: post #29

Earlier quoted context omitted.

The difference is standardization. Postgres can be installed in subdirectories nicely, as you say -- if you know how. Same is true for JBoss, CI build agent, whatever. Now if you have dozens of these apps then e.g. onboarding people in a remote team suddenly becomes non-trivial (a nightmare, to be precise). With Docker, they can get a complex system running in an hour. The two primary use cases for Docker is, as far…

>The difference is standardization. Postgres can be installed in subdirectories nicely, as you say -- if you know how. Or if you have, y'know, a really simple script. ...the kind which also runs inside most semi-complex Docker containers anyway.

Installation is the easy part. Seamless removal and replacement isn't.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#99

Post by walterstucco »

pytester wrote: post #39
sz4kerto wrote: post #29

Earlier quoted context omitted.

The difference is standardization. Postgres can be installed in subdirectories nicely, as you say -- if you know how. Same is true for JBoss, CI build agent, whatever. Now if you have dozens of these apps then e.g. onboarding people in a remote team suddenly becomes non-trivial (a nightmare, to be precise). With Docker, they can get a complex system running in an hour. The two primary use cases for Docker is, as far…

>The difference is standardization. Postgres can be installed in subdirectories nicely, as you say -- if you know how. Or if you have, y'know, a really simple script. ...the kind which also runs inside most semi-complex Docker containers anyway.

True

But now you have to write it for at least Linux and Windows...

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#100

Post by frenchtoastto »

pytester wrote: post #25
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

>Docker solves a simple problem that everyone has. Docker provides an (IMHO pretty buggy) isolation layer that lies between "keeping things that need to be kept separate in separate folders" and "keeping things that need to be kept separate in separate virtual machines". I actually don't have the need for the level of isolation below VM and above folder very often. IMHO this level only really makes sense when contain…

That's not all that docker or any containerization gives you. For most people I've seen it's about infrastructure as code and quickly deploying Dev/staging environments. Yes you can do the same with ansible or terraform but what if you only want to test locally? Do you really want to wait for a VM to be carved out? Or just run a command and have things come up?
Post reply on HN