Billions wasted on Hadoop startups, the same will eventually be true of Docker • Hacker News Board

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#211

Post by hardwaresofton » Sat, Jun 01, 2019, 2:46 AM UTC

gerbilly wrote: post #188

hardwaresofton wrote: post #180

Earlier quoted context omitted.

> I don't see it as VMs vs containers. It's not? I didn't mean to pit them against each other in competition, I'm saying that if VMs are worth learning about and taking the time time to understand, so are containers. It doesn't have to be zero sum. > We have a good devops process to deploy onto our instances, so we rarely have resource clashes you mention (ports/directories) because none of that is ever configured ma…

> It seems like it was a class of problems that you have fixed with "good devops process". I'd argue that it probably was a problem at once point, and you improved your devops process to make sure it wasn't. It certainly was but we fixed it and it's not a problem anymore. > Sounds like you could use some more of that "good devops process" you had when you set up the deploy machinery. Yeah, there are people within my…

> I'm fully expecting linux userland tools to go away, to be replaced by custom 'distributions' with only a kernel and a docker API soon.

They're already here!

- CoreOS Container Linux (now owned by Redhat)[0]

- RancherOS[1]

- Kubic[2] (more focused on running Kubernetes, but same idea)

There are also tools like Linuxkit[3] which focus on helping you actually build images that run the containers you want and nothing else @ startup, which is pretty cool I think.

[0]: https://coreos.com/os/docs/latest/

[1]: https://rancher.com/rancher-os/

[2]: https://kubic.opensuse.org/

[3]: https://github.com/linuxkit/linuxkit

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#212

Post by danielscrubs » Sat, Jun 01, 2019, 5:06 AM UTC

hardwaresofton wrote: post #165

kazinator wrote: post #142

Earlier quoted context omitted.

> they call it a VM (and don't qualify/note that they're being fast and loose with terminology). It's only confusing to people who are familiar just with the popularized forms of computer science terms. Isolation and sandboxing is virtualization. In a container, the applications seem to have an operating system and machine to themselves. A single Unix process and its address space is also a kind of virtual machine, c…

> It's only confusing to people who are familiar just with the popularized forms of computer science terms. Yeah given how often people mistakenly call containers VMs or assume they have VM-like properties (like the ability to run a different kernel), this is tripping a lot of developers up, not just lay people. > Isolation and sandboxing is virtualization. In a container, the applications seem to have an operating s…

Errr... how do these people think macOS and Windows can run Linux containers? It is a VM on those platforms.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#213

Post by vkazanov » Sat, Jun 01, 2019, 7:11 AM UTC

notacoward wrote: post #190

vkazanov wrote: post #185

Earlier quoted context omitted.

...ok, then what are the alternatives? And why is it so bad? Something that can handle hundreds of terabytes on hundreds of machines and provides useful tools on top of the whole thing (Spark, Hive, etc)?

Well, rather topically, there's MapR. Depending on the desired scale and level of fidelity to real-filesystem semantics, one might also count Gluster, Ceph, and Lustre. Don't know if PVFS is still around. There are more proprietary offerings from IBM (GPFS/SpectrumScale), EMC, NetApp, etc. plus a plethora of startups. There are other object stores that don't make a pretense of being filesystems. Alternatives abound.…

The thing is that this is exactly my company's scale, and hdfs is perfectly fine here. :-)

And I am curious about possible alternatives: open source, about 100-200 machines, with good support for analytics and SQlish systems.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#214

Post by more-coffee » Sat, Jun 01, 2019, 7:36 AM UTC

threeseed wrote: post #28

more-coffee wrote: post #17

Earlier quoted context omitted.

About 5 years ago I worked at a small, business oriented telco. The biggest ETL was processing external CDRs, applying call tariffs, and creating bills and reports. A previous developer had been adamant about using a Hadoop cluster to process this, storing all of it in Cassandra NoSQL. The idea was interesting, but it didn't quite work out. At some point he left the company and we had to do something about the pipeli…

Maybe that developer knew something unique about that use case i.e. data needs were expected to grow or they had plans to use it for Data Science (very common in telcos). And you just weren't aware of it. It's always easy to pass judgement at technology choices but in my experience they are often made with the best intentions based on requirements that not everyone is aware of.

Yes, there are more use cases I didn't knew of at the time. For example, optimizing revenue by calculating the optimal routing strategy for outbound calls. But data analysis wasn't a prority. Nor was scaling storage or computations.

I know this is circumstantial, but on average there were 2 developers. The company needed something to just get the bills out. It also needed to be correct, stable, easily expandable for new datasets (requiring all kinds of conversions), testable from unit to e2e, integrate with other APIs to retrieve metadata, and .. I forgot a few, probably.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#215

Post by anticensor » Sat, Jun 01, 2019, 8:59 PM UTC

vkazanov wrote: post #53

nell wrote: post #33

Earlier quoted context omitted.

Solutions to hard problems of big enterprises should be the most juicy way to make money no?

I am not a certified specialist in Hadoop-related things but there's definitely an industry of people working with Hadoop/HDFS/Spark/etc. It's almost a standard in Business Intelligence where I work these days. So there should be some money there.

HDFS is Hadoop, other parts being MapReduce, YARN and HDDS.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#216

Post by yoz-y » Mon, Jun 03, 2019, 8:49 AM UTC

StreamBright wrote: post #187

yoz-y wrote: post #173

Earlier quoted context omitted.

Much like accessibility, even if you are capable of doing everything by hand, it is usually nicer to have most of the gruntwork handled for you.

Well, not necessarily by hand. I never gave up deploying our Java application with Ansible. We could have used Docker but the team decided to use fat jars and Ansible instead. Nowadays with Java 11 you can make those fat jars even slimmer. There was no value proposition for us to change.

I did not work much with deployments but one thing I liked with Docker over Ansible is that testing the configurations locally is really easy and independent on the host platform.