Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

Post reply on HN
Print view View original

71–80 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#71

Post by jabl »

stingraycharles wrote: post #57
geezerjay wrote: post #38

Earlier quoted context omitted.

> Hadoop solved a hard problem that nobody had. This baseless assertion is patently wrong on so many levels. Building computing clusters on COTS hardware is a very mundane problem. Running processing jobs on data shards is a very mundane problem. Scaling COTS clusters transparently is a very mundane problem.

Insert "almost" into parent's sentence and it becomes correct. Many people use/used Hadoop for problems that did not warrant the overhead and complexity that comes with Hadoop. I've seen it countless times with my own eyes that people pre-emptively use tools like Hadoop and Spark because of a chance that they will hit a massive scale in the future. This happens in both startups and enterprises alike: people like to t…

> This happens in both startups and enterprises alike: people like to think they have big problems too often.

A.k.a. resume-driven development. Having Hadoop on your CV looks sexier than awk.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#72

Post by wongarsu »

misterman0 wrote: post #41
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

>> Hadoop solved a hard problem that nobody had. I'm genuinely curious, if you want to search over big data, which should be a pretty common procedure these days, what alternatives are there to a distributed file system? A dfs seems very complex to me. And it is not clear too me what alternative system designs a dfs will outperform. Is a dfs the only solution to big data? Relational DBs do break down at a certain sca…

Relational databases work just fine for terabytes of data. Aparently there are postgres deployments with petabytes of data. There aren't all that many companies with datasets that require anything else. Google and Twitter have the problems Hadoop solves, but it's fair to say that approximately nobody else has those problems.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#73

Post by aduitsis »

kklimonda wrote: post #54
aduitsis wrote: post #32

Earlier quoted context omitted.

What about setup and maintenance of the infrastructure that hosts either VMs or containers? Given a mediocre number of physical machines (say, 40), which is easier, install, setup and maintain a VMWare cluster or a Kubernetes cluster? If someone has any insight, preferably backed by actual experience, it'll be most appreciated.

Assuming unlimited budget for hardware and licenses, then building a team that can deploy and manage VMWare on 40 nodes will be much easier - however raw VM is not really comparable to what k8s is giving you, so you'll be also solving reproducible deployments, load balancing traffic to your cluster etc. Still, with enough money those can be solved by purchasing more hardware&software, and you'll have easier time find…

Many thanks!

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#74

Post by misterman0 »

arnon wrote: post #59
misterman0 wrote: post #41

Earlier quoted context omitted.

>> Hadoop solved a hard problem that nobody had. I'm genuinely curious, if you want to search over big data, which should be a pretty common procedure these days, what alternatives are there to a distributed file system? A dfs seems very complex to me. And it is not clear too me what alternative system designs a dfs will outperform. Is a dfs the only solution to big data? Relational DBs do break down at a certain sca…

Not all relational DBs break at scale. Relational DBs break at scale if you rely on very specific optimizations like specific types of indexes. Even Postgres got this right recently with the introduction of the BRIN index, which is a lot more lightweight. Look at Netezza, Oracle Exadata, and (disclaimer: I work on this) SQream DB, which can absolutely handle hundreds of terabytes without too much fuss.

Which of those indexes/databases should I be using if I want to query over big data using natural language?

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#75

Post by jarfil »

eeZah7Ux wrote: post #52
otabdeveloper1 wrote: post #50

Earlier quoted context omitted.

90% of the time Docker is used to solve the problem of "how do I upload this bucket of Python crud to a production server?" (Replace 'Python' with any other language to taste.) A slightly smarter .tar.gz would have solved the problem just as well.

> A slightly smarter .tar.gz would have solved the problem just as well. It's called "OS package" ;) and can provide more strict sandboxing using a systemd unit file: unit files provide seccomp, cgroups and more.

Docker containers provide seccomp, cgroups and more.

Yes, systemd unit files are containers, just like Docker.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#76

Post by hardwaresofton »

raesene9 wrote: post #61
sasavilic wrote: post #21

You can't really compare VM with Docker. Managing containers with Docker + Kubernetes is far easier then managing VMs. Docker might be replaced with something else in future (i.e. rkt), but basic concept IMHO is here to stay.

small aside, but it looks like rkt is not going to be that replacement. Recent disclosure of vulnerabilities in rkt that Redhat are apparently not going to bother patching indicates they don't currently see it as an active project.

rkt is basically dead and so is rktnetes.

However, docker internals have basically been replaced with containerd[0] in this point -- the two front runners in the battle to actually run your containers (and power higher level abstraction tools like docker) are containerd[0] and cri-o[1].

I personally prefer containerd, but there are a lot of people who are obsessed with cri-o (big company backers, from what I can remember), despite the fact that it's chronically behind on features (for example alternate runtimes, runtimeClass support), but they're both excellent.

Note that there are also other projects like podman[2] that also aim to serve as docker replacements.

Discussion on containerd shim in docker can be found on google group[3] way back in 2016.

[0]: https://github.com/containerd

[1]: https://github.com/cri-o/cri-o

[2]: https://github.com/containers/libpod

[3]: https://groups.google.com/forum/#!topic/docker-dev/zaZFlvIx1...

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#78

Post by coredog64 »

stingraycharles wrote: post #57
geezerjay wrote: post #38

Earlier quoted context omitted.

> Hadoop solved a hard problem that nobody had. This baseless assertion is patently wrong on so many levels. Building computing clusters on COTS hardware is a very mundane problem. Running processing jobs on data shards is a very mundane problem. Scaling COTS clusters transparently is a very mundane problem.

Insert "almost" into parent's sentence and it becomes correct. Many people use/used Hadoop for problems that did not warrant the overhead and complexity that comes with Hadoop. I've seen it countless times with my own eyes that people pre-emptively use tools like Hadoop and Spark because of a chance that they will hit a massive scale in the future. This happens in both startups and enterprises alike: people like to t…

This. A prior employer spent millions on a Cloudera install that had 5 worker nodes and was being used to service a bunch of generic jobs.

Worse still, it didn’t even use HDFS and we eventually got sick of the crappy embedded Zookeeper/Kafka setup.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#79

Post by bezoisevil »

pytester wrote: post #25
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

>Docker solves a simple problem that everyone has. Docker provides an (IMHO pretty buggy) isolation layer that lies between "keeping things that need to be kept separate in separate folders" and "keeping things that need to be kept separate in separate virtual machines". I actually don't have the need for the level of isolation below VM and above folder very often. IMHO this level only really makes sense when contain…

Wrong. Think scaling, Docker will allow me to take a 3rd party app, containerize it and place it in an on prem cloud. I can use the containers for DR or migrating infrastructure. It solves a huge problem for Fortunes

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#80

Post by raesene9 »

debiandev wrote: post #48
beagle3 wrote: post #16

Earlier quoted context omitted.

Properly packaged Debian/raspbian apps are still an “apt install” away. Your use case, which is common, tells me that packaging / distributing may need some love, not that there’s a fundamental difference. And the convenience does not come free - a random docker is almost as bad as a random executable.

> a random docker is almost as bad as a random executable You can sandbox a random executable with seccomp, you cannot effectively sandbox a whole container without breaking many things.

you know that Docker (by default) includes a seccomp filter as one of it's layers of isolation, right?
Post reply on HN