Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

Post reply on HN
Print view View original

41–50 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#41

Post by misterman0 »

imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

>> Hadoop solved a hard problem that nobody had.

I'm genuinely curious, if you want to search over big data, which should be a pretty common procedure these days, what alternatives are there to a distributed file system? A dfs seems very complex to me. And it is not clear too me what alternative system designs a dfs will outperform. Is a dfs the only solution to big data?

Relational DBs do break down at a certain scale. What system do you turn to next? Nosql? Will that scale infinitely? Will any system scale infinitely?

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#42

Post by AmericanChopper »

7373737373 wrote: post #27
saagarjha wrote: post #4

Earlier quoted context omitted.

Sandboxed, consistent environments to run code in?

All this money should have been spent on developing new, improving existing or switching to better (operating) systems which solve the resource and communication security problems, instead of creating another inner-platform effect. I hope WebAssembly goes in this direction, instead of trying to adapt to current programming language paradigms.

>All this money should have been spent on developing new, improving existing or switching to better (operating) systems which solve the resource and communication security problems

But this isn’t the problem Docker is trying to solve. It’s just a problem that Docker needed to solve in order for their product to be useful, this is completely transparent to Docker users. Docker abstracts away a whole bunch of work you’d otherwise have to do to implement repeatable builds, it makes those builds widely distributable, and (depending on how you choose to use containers) can also simplify some capacity planning problems.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#43

Post by debiandev »

chime wrote: post #5

> The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, I bought a Raspberry Pi and using a few commands, installed pre-configured Docker ARM images for 8-9 different media applications that would've taken me days to setup and manage individually. I didn't have to worry about dependencies or compilations. It just worked.

> installed pre-configured Docker ARM images for 8-9 different media applications ... It just worked.

Often the applications are packaged by random people on the Internet and do not receive security updates.

There's plenty of evidence showing how bad the problem is and there's no way around it.

You need the security team of a distribution to backport security fixes into a stable distribution and a large user community test them.

Only with this you can run apt-get upgrade without breaking things.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#44

Post by threeseed »

huffmsa wrote: post #37
threeseed wrote: post #28

Earlier quoted context omitted.

Maybe that developer knew something unique about that use case i.e. data needs were expected to grow or they had plans to use it for Data Science (very common in telcos). And you just weren't aware of it. It's always easy to pass judgement at technology choices but in my experience they are often made with the best intentions based on requirements that not everyone is aware of.

Doesn't matter what he "might have known or intended to do", he didn't build a solution which met the specifications the business needed.

You really didn't get my point did you.

We don't know about the specifications because we don't work there and since the OP said he wasn't there he might not know either. And the fact is that requirements regularly change over time.

The point is that it's really easy to judge when you aren't there and are privy to all the facts.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#45

Post by raverbashing »

misterman0 wrote: post #41
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

>> Hadoop solved a hard problem that nobody had. I'm genuinely curious, if you want to search over big data, which should be a pretty common procedure these days, what alternatives are there to a distributed file system? A dfs seems very complex to me. And it is not clear too me what alternative system designs a dfs will outperform. Is a dfs the only solution to big data? Relational DBs do break down at a certain sca…

The issue is that 80% of "big data" isn't

https://adamdrake.com/command-line-tools-can-be-235x-faster-...

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#46

Post by chungleong »

kumarvvr wrote: post #3

I have come across quite a few articles that mention that for 99% of 'big' data problems, Hadoop and the like are an overkill. Simple tools, with a beefy machine is just as sufficient for the task. Is that the reality of today? Personally I too feel that distributed computing is an overkill for most 'big' data problems.

99% of businesses don't experience exponential growth. But hope springs eternal in the human breast.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#47

Post by ForHackernews »

pytester wrote: post #25
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

>Docker solves a simple problem that everyone has. Docker provides an (IMHO pretty buggy) isolation layer that lies between "keeping things that need to be kept separate in separate folders" and "keeping things that need to be kept separate in separate virtual machines". I actually don't have the need for the level of isolation below VM and above folder very often. IMHO this level only really makes sense when contain…

I sort of agree with this, but the advantage I see of Docker is in providing a "standard" (ymmv) interface that's less heavyweight than running an entire simulated virtual machine, but less dependent on the idiosyncrasies of a specific environment.

I can give my coworker a docker image and it mostly "just work" without failing because she happens to be running a slightly different version of Ubuntu with different system libraries present.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#48

Post by debiandev »

beagle3 wrote: post #16
chime wrote: post #5

> The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, I bought a Raspberry Pi and using a few commands, installed pre-configured Docker ARM images for 8-9 different media applications that would've taken me days to setup and manage individually. I didn't have to worry about dependencies or compilations. It just worked.

Properly packaged Debian/raspbian apps are still an “apt install” away. Your use case, which is common, tells me that packaging / distributing may need some love, not that there’s a fundamental difference. And the convenience does not come free - a random docker is almost as bad as a random executable.

> a random docker is almost as bad as a random executable

You can sandbox a random executable with seccomp, you cannot effectively sandbox a whole container without breaking many things.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#49

Post by AmericanChopper »

threeseed wrote: post #35
bayareanative wrote: post #20

Yup. Tech faabionabilism is akin to Big4/MBB FUD flavor of the quarter... populism/marketing doesn't a necessity make. Docker, Kubernetes, gulp, Hadoop, mosh, Nix, SmartOS, serverless, cloud, virtualization, [insert tech fashion hype > utility here]. Speaking of Hadoop: my vehicle is parked outside one of the HQ's of another top 10 Hadoop startup. It's one of the most expensive, nearly empty buildings in the highest…

I love how the "cloud" is a tech fad in your eyes as well as technologies like Kubernetes which are core to Google and virtualisation which is core to every VPS/cloud provider in existence. Be a pretty awful world if everyone took your advice.

Isn’t it obvious? Everybody who doesn’t operate their own datacenters are mindless technology hipsters.

Whenever a new big piece of tech comes out, most of the detractors seem to either a) find a use case that isn’t fit for purpose or b) try to use it without bothering to learn how, and the proceed to say ‘see, it’s not all it’s cracked up to be’.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#50

Post by otabdeveloper1 »

pytester wrote: post #25
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

>Docker solves a simple problem that everyone has. Docker provides an (IMHO pretty buggy) isolation layer that lies between "keeping things that need to be kept separate in separate folders" and "keeping things that need to be kept separate in separate virtual machines". I actually don't have the need for the level of isolation below VM and above folder very often. IMHO this level only really makes sense when contain…

90% of the time Docker is used to solve the problem of "how do I upload this bucket of Python crud to a production server?" (Replace 'Python' with any other language to taste.)

A slightly smarter .tar.gz would have solved the problem just as well.

Post reply on HN