Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

Post reply on HN
Print view View original

31–40 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#31

Post by vkazanov »

imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

I'll correct you: Hadoop solves a hard problem that lots of big enterprises have. A distributed file system is a big deal, and everything that on top also is.

Now making (big)money with its ecosystem is another question.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#32

Post by aduitsis »

sasavilic wrote: post #21

You can't really compare VM with Docker. Managing containers with Docker + Kubernetes is far easier then managing VMs. Docker might be replaced with something else in future (i.e. rkt), but basic concept IMHO is here to stay.

What about setup and maintenance of the infrastructure that hosts either VMs or containers?

Given a mediocre number of physical machines (say, 40), which is easier, install, setup and maintain a VMWare cluster or a Kubernetes cluster?

If someone has any insight, preferably backed by actual experience, it'll be most appreciated.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#33

Post by nell »

vkazanov wrote: post #31
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

I'll correct you: Hadoop solves a hard problem that lots of big enterprises have. A distributed file system is a big deal, and everything that on top also is. Now making (big)money with its ecosystem is another question.

Solutions to hard problems of big enterprises should be the most juicy way to make money no?

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#34

Post by huffmsa »

kumarvvr wrote: post #3

I have come across quite a few articles that mention that for 99% of 'big' data problems, Hadoop and the like are an overkill. Simple tools, with a beefy machine is just as sufficient for the task. Is that the reality of today? Personally I too feel that distributed computing is an overkill for most 'big' data problems.

Yes. Very very very few people have data as big as "the internet" and the need for speed (which was the reason Google developed a look of their distributed tooling) that you get with these frameworks.

And it really only makes sense to have permanent infrastructure for distributed computing if you're constantly using it. Like if you're constantly rebuilding an index of the internet. Which most people aren't.

For occasional reindexing jobs, I've personally had success with Kubernetes. Our customer facing services were deployed with it, so it was trivial to bump up the number of nodes, and then schedule a bunch of workers containers, then roll everything down when the job is complete. No need to learn the ins-and-outs of Hadoop.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#35

Post by threeseed »

bayareanative wrote: post #20

Yup. Tech faabionabilism is akin to Big4/MBB FUD flavor of the quarter... populism/marketing doesn't a necessity make. Docker, Kubernetes, gulp, Hadoop, mosh, Nix, SmartOS, serverless, cloud, virtualization, [insert tech fashion hype > utility here]. Speaking of Hadoop: my vehicle is parked outside one of the HQ's of another top 10 Hadoop startup. It's one of the most expensive, nearly empty buildings in the highest…

I love how the "cloud" is a tech fad in your eyes as well as technologies like Kubernetes which are core to Google and virtualisation which is core to every VPS/cloud provider in existence.

Be a pretty awful world if everyone took your advice.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#36

Post by eeZah7Ux »

pytester wrote: post #25
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

>Docker solves a simple problem that everyone has. Docker provides an (IMHO pretty buggy) isolation layer that lies between "keeping things that need to be kept separate in separate folders" and "keeping things that need to be kept separate in separate virtual machines". I actually don't have the need for the level of isolation below VM and above folder very often. IMHO this level only really makes sense when contain…

The isolation is done by many other tools, and with less bugs and vulnerabilities.

> applications that have weirdly specific, non-standard system level dependencies

Spot on. 99% of software the world needs can be written against libraries provided by OSes. And then packaged properly.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#37

Post by huffmsa »

threeseed wrote: post #28
more-coffee wrote: post #17

Earlier quoted context omitted.

About 5 years ago I worked at a small, business oriented telco. The biggest ETL was processing external CDRs, applying call tariffs, and creating bills and reports. A previous developer had been adamant about using a Hadoop cluster to process this, storing all of it in Cassandra NoSQL. The idea was interesting, but it didn't quite work out. At some point he left the company and we had to do something about the pipeli…

Maybe that developer knew something unique about that use case i.e. data needs were expected to grow or they had plans to use it for Data Science (very common in telcos). And you just weren't aware of it. It's always easy to pass judgement at technology choices but in my experience they are often made with the best intentions based on requirements that not everyone is aware of.

Doesn't matter what he "might have known or intended to do", he didn't build a solution which met the specifications the business needed.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#38

Post by geezerjay »

imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

> Hadoop solved a hard problem that nobody had.

This baseless assertion is patently wrong on so many levels. Building computing clusters on COTS hardware is a very mundane problem. Running processing jobs on data shards is a very mundane problem. Scaling COTS clusters transparently is a very mundane problem.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#39

Post by pytester »

sz4kerto wrote: post #29
pytester wrote: post #25

Earlier quoted context omitted.

>Docker solves a simple problem that everyone has. Docker provides an (IMHO pretty buggy) isolation layer that lies between "keeping things that need to be kept separate in separate folders" and "keeping things that need to be kept separate in separate virtual machines". I actually don't have the need for the level of isolation below VM and above folder very often. IMHO this level only really makes sense when contain…

The difference is standardization. Postgres can be installed in subdirectories nicely, as you say -- if you know how. Same is true for JBoss, CI build agent, whatever. Now if you have dozens of these apps then e.g. onboarding people in a remote team suddenly becomes non-trivial (a nightmare, to be precise). With Docker, they can get a complex system running in an hour. The two primary use cases for Docker is, as far…

>The difference is standardization. Postgres can be installed in subdirectories nicely, as you say -- if you know how.

Or if you have, y'know, a really simple script.

...the kind which also runs inside most semi-complex Docker containers anyway.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#40

Post by threeseed »

huffmsa wrote: post #34
kumarvvr wrote: post #3

I have come across quite a few articles that mention that for 99% of 'big' data problems, Hadoop and the like are an overkill. Simple tools, with a beefy machine is just as sufficient for the task. Is that the reality of today? Personally I too feel that distributed computing is an overkill for most 'big' data problems.

Yes. Very very very few people have data as big as "the internet" and the need for speed (which was the reason Google developed a look of their distributed tooling) that you get with these frameworks. And it really only makes sense to have permanent infrastructure for distributed computing if you're constantly using it. Like if you're constantly rebuilding an index of the internet. Which most people aren't. For occas…

It is very, very, very common for companies to have enough data to need a distributed approach to ETL and especially common if they are doing any machine learning which is most of them.

In telcos you have network telemetry data. In supermarkets and retail you have purchase data and often credit card gateway data. In banking and finance you obviously have transaction data.

And with Kubernetes you still need a compute framework. Like I don't know the standard in the industry the Hadoop Spark framework.

Post reply on HN