Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

Post reply on HN
Print view View original

51–60 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#51

Post by huffmsa »

threeseed wrote: post #44
huffmsa wrote: post #37

Earlier quoted context omitted.

Doesn't matter what he "might have known or intended to do", he didn't build a solution which met the specifications the business needed.

You really didn't get my point did you. We don't know about the specifications because we don't work there and since the OP said he wasn't there he might not know either. And the fact is that requirements regularly change over time. The point is that it's really easy to judge when you aren't there and are privy to all the facts.

But we do know the OPs solution met the specifications, and is still in use, meaning that those specifications haven't changed too much.

And you're damn right I'm going to be judgemental of someone not only promised the moon, but abandoned the work when the rocket exploded on the pad and someone else had to clean up.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#52

Post by eeZah7Ux »

otabdeveloper1 wrote: post #50
pytester wrote: post #25

Earlier quoted context omitted.

>Docker solves a simple problem that everyone has. Docker provides an (IMHO pretty buggy) isolation layer that lies between "keeping things that need to be kept separate in separate folders" and "keeping things that need to be kept separate in separate virtual machines". I actually don't have the need for the level of isolation below VM and above folder very often. IMHO this level only really makes sense when contain…

90% of the time Docker is used to solve the problem of "how do I upload this bucket of Python crud to a production server?" (Replace 'Python' with any other language to taste.) A slightly smarter .tar.gz would have solved the problem just as well.

> A slightly smarter .tar.gz would have solved the problem just as well.

It's called "OS package" ;) and can provide more strict sandboxing using a systemd unit file: unit files provide seccomp, cgroups and more.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#53

Post by vkazanov »

nell wrote: post #33
vkazanov wrote: post #31

Earlier quoted context omitted.

I'll correct you: Hadoop solves a hard problem that lots of big enterprises have. A distributed file system is a big deal, and everything that on top also is. Now making (big)money with its ecosystem is another question.

Solutions to hard problems of big enterprises should be the most juicy way to make money no?

I am not a certified specialist in Hadoop-related things but there's definitely an industry of people working with Hadoop/HDFS/Spark/etc. It's almost a standard in Business Intelligence where I work these days.

So there should be some money there.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#54

Post by kklimonda »

aduitsis wrote: post #32
sasavilic wrote: post #21

You can't really compare VM with Docker. Managing containers with Docker + Kubernetes is far easier then managing VMs. Docker might be replaced with something else in future (i.e. rkt), but basic concept IMHO is here to stay.

What about setup and maintenance of the infrastructure that hosts either VMs or containers? Given a mediocre number of physical machines (say, 40), which is easier, install, setup and maintain a VMWare cluster or a Kubernetes cluster? If someone has any insight, preferably backed by actual experience, it'll be most appreciated.

Assuming unlimited budget for hardware and licenses, then building a team that can deploy and manage VMWare on 40 nodes will be much easier - however raw VM is not really comparable to what k8s is giving you, so you'll be also solving reproducible deployments, load balancing traffic to your cluster etc. Still, with enough money those can be solved by purchasing more hardware&software, and you'll have easier time finding people who can maintain that over on-premise kubernetes.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#55

Post by misterman0 »

raverbashing wrote: post #45
misterman0 wrote: post #41

Earlier quoted context omitted.

>> Hadoop solved a hard problem that nobody had. I'm genuinely curious, if you want to search over big data, which should be a pretty common procedure these days, what alternatives are there to a distributed file system? A dfs seems very complex to me. And it is not clear too me what alternative system designs a dfs will outperform. Is a dfs the only solution to big data? Relational DBs do break down at a certain sca…

The issue is that 80% of "big data" isn't https://adamdrake.com/command-line-tools-can-be-235x-faster-...

That's not at all what my issue is though. I have what you might call "big data problems" and I'm so turned-off by Hadoop that I'm actually rolling my own distributed DB. But what principles other than a DFS should one look into, is the question, really. At least to me.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#56

Post by mattmanser »

misterman0 wrote: post #41
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

>> Hadoop solved a hard problem that nobody had. I'm genuinely curious, if you want to search over big data, which should be a pretty common procedure these days, what alternatives are there to a distributed file system? A dfs seems very complex to me. And it is not clear too me what alternative system designs a dfs will outperform. Is a dfs the only solution to big data? Relational DBs do break down at a certain sca…

What's common about big data?

My guess is that almost all programming jobs are in fields that produce no more data than a gb or two a month.

I'm happy to be proven wrong, but I would guess that there are far more companies making project management software, time tracking apps, invoicing software, etc. than there are facebooks, googles or reddits obsessively logging every user mouse twitch.

And that's data that's much better sitting in a nice, normal, relational database.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#57

Post by stingraycharles »

geezerjay wrote: post #38
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

> Hadoop solved a hard problem that nobody had. This baseless assertion is patently wrong on so many levels. Building computing clusters on COTS hardware is a very mundane problem. Running processing jobs on data shards is a very mundane problem. Scaling COTS clusters transparently is a very mundane problem.

Insert "almost" into parent's sentence and it becomes correct.

Many people use/used Hadoop for problems that did not warrant the overhead and complexity that comes with Hadoop. I've seen it countless times with my own eyes that people pre-emptively use tools like Hadoop and Spark because of a chance that they will hit a massive scale in the future.

This happens in both startups and enterprises alike: people like to think they have big problems too often.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#58

Post by em500 »

threeseed wrote: post #28
more-coffee wrote: post #17

Earlier quoted context omitted.

About 5 years ago I worked at a small, business oriented telco. The biggest ETL was processing external CDRs, applying call tariffs, and creating bills and reports. A previous developer had been adamant about using a Hadoop cluster to process this, storing all of it in Cassandra NoSQL. The idea was interesting, but it didn't quite work out. At some point he left the company and we had to do something about the pipeli…

Maybe that developer knew something unique about that use case i.e. data needs were expected to grow or they had plans to use it for Data Science (very common in telcos). And you just weren't aware of it. It's always easy to pass judgement at technology choices but in my experience they are often made with the best intentions based on requirements that not everyone is aware of.

More likely he knew that the pay for a hadoop / bigdata specialist was a lot better than for a mysql specialist, especially 5 years ago.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#59

Post by arnon »

misterman0 wrote: post #41
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

>> Hadoop solved a hard problem that nobody had. I'm genuinely curious, if you want to search over big data, which should be a pretty common procedure these days, what alternatives are there to a distributed file system? A dfs seems very complex to me. And it is not clear too me what alternative system designs a dfs will outperform. Is a dfs the only solution to big data? Relational DBs do break down at a certain sca…

Not all relational DBs break at scale. Relational DBs break at scale if you rely on very specific optimizations like specific types of indexes.

Even Postgres got this right recently with the introduction of the BRIN index, which is a lot more lightweight.

Look at Netezza, Oracle Exadata, and (disclaimer: I work on this) SQream DB, which can absolutely handle hundreds of terabytes without too much fuss.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#60

Post by danpalmer »

geezerjay wrote: post #38
imtringued wrote: post #12

I don't understand how these are comparable. Hadoop solved a hard problem that nobody had. Docker solves a simple problem that everyone has. It would make sense if you're talking about kubernetes and using it to build hundreds of microservices because it's currently in fashion. Whether you're using Docker, Packer, Ansible or whatever doesn't matter. They are all a solution to the same problem and saying one is better…

> Hadoop solved a hard problem that nobody had. This baseless assertion is patently wrong on so many levels. Building computing clusters on COTS hardware is a very mundane problem. Running processing jobs on data shards is a very mundane problem. Scaling COTS clusters transparently is a very mundane problem.

There are broadly 3 levels of the amount of data that people have:

- Fits on one computer (most of the market)

- Fits on several computers (most of the rest)

- Requires a significant cluster of machines (50+ to store it)

Hadoop only really solves the last one. It has huge overheads in terms of speed and in terms of resources and headcount to run it properly, so it only makes sense at a particular scale. It's like a mainframe – most companies shouldn't buy one.

If you add to this the fact that Hadoop was about batch processing, and its "realtime" capabilities were poor, there really aren't that many potential customers, and many of the potential customers would rather run it in-house, or build their own system.

Post reply on HN