Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

81–90 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#81
I am very much a fan of hot-takes, but this one is trash --

> The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today.

Linux containerization (using the word "docker" for everything isn't right either) is an isolation + sandboxing mechanism, NOT a virtual machine. Even if you talk about things like LXC (orchestrated by LXD), that's basically just the addition of the user namespacing feature. A docker container is not a VM, it is a regular process, isolated with the use of cgroups and namespaces, possibly protected (like any other process) with selinux/apparmor/etc.

Containerization is almost objectively a better way of running applications -- there's only one question, do you want your process to be isolated, or not. All the other stuff (using Dockerfiles, pulling images, the ease of running languages that require their own interpreters since you package the filesystem) is on top of this basic value propostion.

An easy way to tell that someone doesn't know what they're talking about when speaking about containerization is if they call it a VM (and don't qualify/note that they're being fast and loose with terminology).

All this said -- I do think Docker will die, and it should die because Docker is no longer the only game in town for reasonably managing (see: podman,crictl) and running containers (see: containerd/cri-o, libcontainer which turned into runc) .

[EDIT] - I want to point out that I do not mean the Docker the company or Docker the project will "die" -- they have done amazing things for the community and development as a whole that will literally go down in history as a paradigm shift. What I should have written was that "docker " where x is "image", "container", "registry", etc should be replaced by "container ".

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#82

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

They're comparable in the sense that neither technology had enough market demand to sustain the companies attempting to sell products. Great tech that we all benefit from, but the monetizable bits were not worth the investment.

Then again it's called venture capital for a reason so this isn't exactly unexpected. The question should really be more about the scale and hype that was involved.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#83
post #58

Earlier quoted context omitted.

Maybe that developer knew something unique about that use case i.e. data needs were expected to grow or they had plans to use it for Data Science (very common in telcos). And you just weren't aware of it. It's always easy to pass judgement at technology choices but in my experience they are often made with the best intentions based on requirements that not everyone is aware of.

More likely he knew that the pay for a hadoop / bigdata specialist was a lot better than for a mysql specialist, especially 5 years ago.

And the guy's resume probably says

> Implemented big data / real-time Hadoop streaming ETL service processing billions of requests.

I've become very skeptical of anyone who puts a combination of buzzwords and pseudo-numbers in their resume.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#84
post #59

Earlier quoted context omitted.

>> Hadoop solved a hard problem that nobody had. I'm genuinely curious, if you want to search over big data, which should be a pretty common procedure these days, what alternatives are there to a distributed file system? A dfs seems very complex to me. And it is not clear too me what alternative system designs a dfs will outperform. Is a dfs the only solution to big data? Relational DBs do break down at a certain sca…

Not all relational DBs break at scale. Relational DBs break at scale if you rely on very specific optimizations like specific types of indexes. Even Postgres got this right recently with the introduction of the BRIN index, which is a lot more lightweight. Look at Netezza, Oracle Exadata, and (disclaimer: I work on this) SQream DB, which can absolutely handle hundreds of terabytes without too much fuss.

There's also CitusDB, which performs much better than Hadoop because it doesn't have the bloat of the JVM or the inefficiency of brain-dead HDFS. You can write your queries in a few lines of SQL rather than reams of convoluted Java code.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#85

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

Wait, what problem does Hadoop solve?? Hadoop is an extremely poor reimplementation of some Google services. The time when billions went into Hadoop startups even Google stopped using MapReduce because it turned out to be inefficient and very limited to many of the distributed computational problems. MR was extremely well suited for a single thing: aggregating web logs and computing very simple summary statistics. What is the hard problem you mentioned?

Btw. there are many very successful startups in the big data space that understood the limitation of Hadoop and addressed almost every if not all aspects of its shortcomings. A good example would be Snowflake computing.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#86

Earlier quoted context omitted.

90% of the time Docker is used to solve the problem of "how do I upload this bucket of Python crud to a production server?" (Replace 'Python' with any other language to taste.) A slightly smarter .tar.gz would have solved the problem just as well.

No it wouldn't have. What's unzipping and running that code? What's monitoring it and restarting it? How do you mount volumes and env variables? How do you open ports and maintain isolation? A container is vastly more powerful for running an application than a tar file.

No offense, but you're aware you make it sound as if we used to use punch cards until the arrival of docker?

You can often run daemons as different users and set appropriate file permissions. You can add ENV variables to your start up scripts or configuration files. Volumes are mounted by the system (and you set appropriate access rights again). Monitoring and restarting services is managed by your init system (and probably some external monitoring, because sometimes physical hosts go nuts). Depending on your environment you can just produce debs, rpms, or some custom format for packaging/distribution.

Yes, sometimes you still want docker or even a real VM, and there are good reasons for that - I totally agree. But often it is not necessary. I'm often under the impression that some people forget that the currently hyped and cool tech is not always and under every circumstance the right solution to a given type of problem. But that's not an issue with docker alone...

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#87
post #59

Earlier quoted context omitted.

Not all relational DBs break at scale. Relational DBs break at scale if you rely on very specific optimizations like specific types of indexes. Even Postgres got this right recently with the introduction of the BRIN index, which is a lot more lightweight. Look at Netezza, Oracle Exadata, and (disclaimer: I work on this) SQream DB, which can absolutely handle hundreds of terabytes without too much fuss.

Which of those indexes/databases should I be using if I want to query over big data using natural language?

This question makes no sense.

First, NLP based search can be executed on top of any engine (APIs are very handy), relational, kv, graph, filesystem .. so that part is totally irrelevant.

Assuming "big data" in this context is still relational data, then any of those systems would suffice, within their own particular tradeoffs and features.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#90
post #5

> The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, I bought a Raspberry Pi and using a few commands, installed pre-configured Docker ARM images for 8-9 different media applications that would've taken me days to setup and manage individually. I didn't have to worry about dependencies or compilations. It just worked.

Great but how does that help recoup the investment the article mentions?

I’m making no comment on the specifics of the Docker or Hadoop ecosystems as I have no skin in either game but history is full of useful tech that didn’t make money.

Post reply on HN