Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

131–140 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#131

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

Yes! What a load of trash this post is, should be on the last page!

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#132

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

> do you want your process to be isolated, or not. No, not always. Why?. At work I have a few coworkers pushing hard to dockerize (isolate?) everything. This makes debugging when things go wrong a lot harder. I see isolation as one of several qualities a process could have, that sometimes is valuable enough to be worth the sacrifice. Isolation is not some absolute quality that is without significant tradeoffs.

> This makes debugging when things go wrong a lot harder.

I've found the opposite to be true. If you have proper observability into your isolated services you can more easily pinpoint the origin of a bug.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#133
post #59

Earlier quoted context omitted.

Not all relational DBs break at scale. Relational DBs break at scale if you rely on very specific optimizations like specific types of indexes. Even Postgres got this right recently with the introduction of the BRIN index, which is a lot more lightweight. Look at Netezza, Oracle Exadata, and (disclaimer: I work on this) SQream DB, which can absolutely handle hundreds of terabytes without too much fuss.

Which of those indexes/databases should I be using if I want to query over big data using natural language?

There is no direct relation between the query language and the underlying infrastructure.

If you're talking about taking some questions and getting graphs from them, ThoughtSpot does a good job.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#136
post #93

Earlier quoted context omitted.

I know there is cost to this solution but it's a good one: Use the same OS and similar hardware for development and production.

The cost includes making development impossible without internet access, given that devs are not going to be carrying a cluster of servers around with them.

^ This.

Also means developers can work in whatever environment they want, but the result will be reproducible (almost) anywhere.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#137

Earlier quoted context omitted.

Maybe you and the downvoter crew do not know it. https://www.datacenterknowledge.com/archives/2014/06/25/goog... https://www.quora.com/Why-did-Google-stop-using-MapReduce-an...

The same concepts are still there, it's just a nicer API now. Think Hadoop -> spark. Map -> map, filter, flatmap, etc Reduce -> reduce, joins, folds, group by, etc Those other concepts were always expressible as map and reduce, of course, just with a bunch of annoying repetitive work

Dataflow looks like it is just their public version of flumejava (both mention pcollections) [0] which indeed is basically a just a bunch of mapreduces pipelined into a fancy directed acyclic graph.

Here is the abstract: "MapReduce and similar systems significantly ease the task of writing data-parallel code. However, many real-world computations require a pipeline of MapReduces, and programming and managing such pipelines can be difficult. We present FlumeJava, a Java library that makes it easy to develop, test, and run efficient dataparallel pipelines. At the core of the FlumeJava library are a couple of classes that represent immutable parallel collections, each supporting a modest number of operations for processing them in parallel. Parallel collections and their operations present a simple, high-level, uniform abstraction over different data representations and execution strategies. To enable parallel operations to run efficiently, FlumeJava defers their evaluation, instead internally constructing an execution plan dataflow graph. When the final results of the parallel operations are eventually needed, FlumeJava first optimizes the execution plan, and then executes the optimized operations on appropriate underlying primitives (e.g., MapReduces). The combination of high-level abstractions for parallel data and computation, deferred evaluation and optimization, and efficient parallel primitives yields an easy-to-use system that approaches the efficiency of hand-optimized pipelines. FlumeJava is in active use by hundreds of pipeline developers within Google" [0].

[0]: https://ai.google/research/pubs/pub35650

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#138

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

So, last time I had a web dev job was back in 2016, so the whole container thing kind of passed me by. So I don't know the details- but, on the other hand, I also don't have any baggage about it being "a VM".

But, starting from that kind of "clean slate" state I have to say that if it takes exasperated internet posts, like your helpful comment, to explain why containers are not like VMs and how they are not like VMs... well then maybe they are not that much not like VMs to make them such a big new thing.

That goes for many things. Like, I don't get the difference between Volley and Beach Volley. One is played on the beach. So it's volley? Played on the beach?

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#139

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

Docker famously doesn't isolate very well, as known in infosec circles for years now. If you're unaware, search 'containers don't contain'.

MicroVMs start faster and provide better isolation.

Meanwhile, none of this is relevant unless you're building your own cloud platform, which is a huge waste of time for most companies.

MicroVMs, containers, VMs, zones and bare metal are places to execute code. Serverless makes all those distinctions irrelevant.

Sorry if you spent 2015 getting really into Docker. You bet on the wrong horse. It's OK, this happens in tech.

Edit: if it's unclear, I don't mean Docker itself is the wrong horse, I mean containerisation tools per se are the wrong horse - and a bad place to invest your time unless you work for a cloud provider

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#140
post #130

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

I'm not going to support the general thesis of this article, but I want to address something you said. You're right that containers are not VMs, but that's only really relevant as pedantry of technical details. I think that what the author was trying to say (without really understanding it) was a comparison of containers to VMs as units of software deployment. I don't think anyone is credibly using containers as a se…

> You're right that containers are not VMs, but that's only really relevant as pedantry of technical details.

That isn't pedantry, it is an extremely critical point and the one most people miss when figuring out Docker. Both from a security context (docker doesn't provide vm level promises about isolation) and from a resource management side (docker is really close to no overhead).

It is not uncommon to deploy containers on VMs in the real world...

Post reply on HN