Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

Post reply on HN
Print view View original

191–200 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#191

Post by xrd »

panpanna wrote: post #171
hardwaresofton wrote: post #154

Earlier quoted context omitted.

> It’s a black box that uses root to do everything and is a pain to debug. This is less true these days (and on macs that had to use the docker machine "hack" it was barely ever true, per say) -- rootless containers are on the way thanks to user namespaces. For example LXC can run fully rootless containers that act more like VMs themselves (as in they will have systemd as pid 1 inside) -- kernel support, user namespa…

Lxc/lxd is a fantastic piece of software that I wish more people would try. Especially if people already on Ubuntu, Debian and Arch. You absolutely don't need to be a cloud person to use containers. For example I use lxc on my laptop to keep different projects separate and test new software.

Have any links to share that you really liked as examples? I'm using docker compose to do what I think you are saying you do with just pure lxc. I would love to know more about how you do it.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#192

Post by throwaway082729 »

anarazel wrote: post #177
throwaway082729 wrote: post #170

Earlier quoted context omitted.

I'm genuinely curious. Can PG handle hundreds of users querying 100s of GB, sometimes the same set of tables, at the same time?

Well. You're going to run out of CPU and memory bandwidth pretty quickly. So you'd need replicas to share processing load. But I honestly don't think hundreds of users each querying 100s of GBs is all that common.

Again, genuinely curious to learn if you've experience with enterprise companies using 'big data' technologies.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#193

Post by hiram112 »

hardwaresofton wrote: post #154
pixelrevision wrote: post #147

Earlier quoted context omitted.

It’s very easy to underestimate how helpful it can be when you first start working with it. It’s a black box that uses root to do everything and is a pain to debug. Because of this it becomes easy to hand wave it away. Once you have a properly setup project going and your entire build process is mostly repeatable the benefits start becoming more obvious. Yes, you can do all the same things to a certain extent in a vm…

> It’s a black box that uses root to do everything and is a pain to debug. This is less true these days (and on macs that had to use the docker machine "hack" it was barely ever true, per say) -- rootless containers are on the way thanks to user namespaces. For example LXC can run fully rootless containers that act more like VMs themselves (as in they will have systemd as pid 1 inside) -- kernel support, user namespa…

> One of the biggest benefits to running containers is actually E2E tests...

100% agree. Most developers don't even bother testing code that requires DBs and other external services because it's a pain to keep test environments up and running so a lot of the most important and buggy code slips through the cracks.

Then you have the developers who do test but just make heavy use of mocking libs and other tricks which aren't helpful in the long run since the mocks don't actually do anything.

But setting up a docker compose with the whole stack - completely reset each time the tests are run - makes integration tests so much more valuable. And usually it's not a huge deal to set up the plumbing since the docker files are often already created for production.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#194

Post by cmsj »

hardwaresofton wrote: post #161
cmsj wrote: post #130

Earlier quoted context omitted.

I'm not going to support the general thesis of this article, but I want to address something you said. You're right that containers are not VMs, but that's only really relevant as pedantry of technical details. I think that what the author was trying to say (without really understanding it) was a comparison of containers to VMs as units of software deployment. I don't think anyone is credibly using containers as a se…

> I think that what the author was trying to say (without really understanding it) was a comparison of containers to VMs as units of software deployment. I agree, I think the next evolution of software deployment is definitely heading to sub-program sizes. I do want to point out that we've seen this before, it was called CGI. It's not exactly the same, and things will be better this time (more isolation, better tooli…

That's a fair point, small-d-docker should not be a thing.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#195

Post by Gravityloss »

pixelrevision wrote: post #147
hardwaresofton wrote: post #81

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

It’s very easy to underestimate how helpful it can be when you first start working with it. It’s a black box that uses root to do everything and is a pain to debug. Because of this it becomes easy to hand wave it away. Once you have a properly setup project going and your entire build process is mostly repeatable the benefits start becoming more obvious. Yes, you can do all the same things to a certain extent in a vm…

Docker containers must run on a vm on Windows and Mac.

So it's an innovation in packaging.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#196

Post by panpanna »

xrd wrote: post #191
panpanna wrote: post #171

Earlier quoted context omitted.

Lxc/lxd is a fantastic piece of software that I wish more people would try. Especially if people already on Ubuntu, Debian and Arch. You absolutely don't need to be a cloud person to use containers. For example I use lxc on my laptop to keep different projects separate and test new software.

Have any links to share that you really liked as examples? I'm using docker compose to do what I think you are saying you do with just pure lxc. I would love to know more about how you do it.

Not really. I personally use each container like a full VM with it's own software. I don't try to combine a lot of small containers into a system the way you would do with docker.

Here is a general tutorial in anyone wants to test lxc:

https://linuxcontainers.org/lxd/getting-started-cli/

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#197

Post by zerogvt »

imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

"Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem"

Nope. K8s is a cluster operating system that happens to fit very well with the microservices architectural model. Packer is a way to create classic VM imgs (this one is similar-ish to Docker but only if you only care about the 10000 feet non-technical-at-all image. Ansible is an infrastructure as a code tool. You deal with mostly classic infra components and compositions of them as code. Putting all them in the same bag is like saying that all programming languages solve the same problem. It's only true if we cut the conversation down to a level where we consider all digital devices as the exact same thing.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#198

Post by SmokeGS »

hardwaresofton wrote: post #81

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

I appreciate your nuance and defense in this thread.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#199

Post by YeGoblynQueenne »

linuxftw wrote: post #179
YeGoblynQueenne wrote: post #138

Earlier quoted context omitted.

So, last time I had a web dev job was back in 2016, so the whole container thing kind of passed me by. So I don't know the details- but, on the other hand, I also don't have any baggage about it being "a VM". But, starting from that kind of "clean slate" state I have to say that if it takes exasperated internet posts, like your helpful comment, to explain why containers are not like VMs and how they are not like VMs.…

Beach volleyball is played typically with 2's or 4's. The rules are slightly different, court size is smaller. There are no positions in 2's, there are no position faults. Feel free to make analogies between two other similar sports, such as Competition Karate and Taekwon-Do.

I'm sorry but this still sounds like the same game to me. Why is it not called Beach 2-by-2-ball, or somethign similar, if the number of players makes it so very different? If I get a couple of my mates together and we bounce a ball over a net counting points when the ball hits the ground- how is what we're playing not (an informal game of) volleybal?

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#200

Post by geezerjay »

danpalmer wrote: post #119
thrower123 wrote: post #92

Earlier quoted context omitted.

> - Fits on one computer (most of the market) There's one category above that, which is "Fits in memory" and that is a huge chunk of the market. I've seen first hand people getting way too cute and complicated planning for scale, and then it works out that they don't even have more than a couple of GB of data.

I've heard of a startup spending many days of contractor time to optimise their database so they didn't have to go from the 4GB Heroku database to the 8GB one. I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets. I've heard of companies…

> I've heard of a startup spending many days of contractor time to optimise their database so they didn't have to go from the 4GB Heroku database to the 8GB one.

From Heroku's site, Heroku's 4GB database plan goes for 50$/(instance.month) while Heroku's 8GB plan goes for 200$/(instance.month).

Therefore, it isn't a question of if it makes finantial sense (it does) but how long the startup plans to operate to recover their investment.

> I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets.

I find it rather strange how someone believes that it's a decent idea to conduct a company's data analysis work on what an employee manages to fit on an external HD, as it creates a whole lot of hard problems both legal and technical. I mean, how do you ensure the data's provenance is tracked and other data analysts can access the data? Who in their right mind would put himself on a situation where a minor lapse or misfortune (losing/getting the HD stolen) could put the company at risk?

> Roughly I'd suggest that "fits on my laptop" is 100TB. I could be too low on these though.

That's a rather naive and missinformed take on Hadoop. Hadoop might be conflated with big data but it's actually a distributed system designed to reliably process data shards without having to incur a penalty to move data around. It makes absolutely no sense to base your assertion on data volumes alone. What matters if it the performance increase justifies setting up a hadoop cluster with the resourses available to a company.

Post reply on HN