Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

Post reply on HN
Print view View original

111–120 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#111

Post by arethuza »

jabl wrote: post #71
stingraycharles wrote: post #57

Earlier quoted context omitted.

Insert "almost" into parent's sentence and it becomes correct. Many people use/used Hadoop for problems that did not warrant the overhead and complexity that comes with Hadoop. I've seen it countless times with my own eyes that people pre-emptively use tools like Hadoop and Spark because of a chance that they will hit a massive scale in the future. This happens in both startups and enterprises alike: people like to t…

> This happens in both startups and enterprises alike: people like to think they have big problems too often. A.k.a. resume-driven development. Having Hadoop on your CV looks sexier than awk.

I was at a conference once where someone was presenting and describing something where they used Hadoop to do pretty simple processing on ~30MB of data and the audience appeared to be lapping it up.

I just sat there thinking I could probably run what they did on my phone.

And just to be clear - it wasn't a PoC or a demo.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#112

Post by weberc2 »

archi42 wrote: post #86
manigandham wrote: post #68

Earlier quoted context omitted.

No it wouldn't have. What's unzipping and running that code? What's monitoring it and restarting it? How do you mount volumes and env variables? How do you open ports and maintain isolation? A container is vastly more powerful for running an application than a tar file.

No offense, but you're aware you make it sound as if we used to use punch cards until the arrival of docker? You can often run daemons as different users and set appropriate file permissions. You can add ENV variables to your start up scripts or configuration files. Volumes are mounted by the system (and you set appropriate access rights again). Monitoring and restarting services is managed by your init system (and p…

The difference is that with VMs, you have to configure the things that you get for free with container runtimes. Specifically, Amazon can take care of a ton of the most mundane security and compliance burden that our org would otherwise have to own. Those differences means that developers can cost effectively be trained to do much of their own ops and I can solve more interesting problems.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#113

Post by _dps »

thrower123 wrote: post #92
danpalmer wrote: post #60

Earlier quoted context omitted.

There are broadly 3 levels of the amount of data that people have: - Fits on one computer (most of the market) - Fits on several computers (most of the rest) - Requires a significant cluster of machines (50+ to store it) Hadoop only really solves the last one. It has huge overheads in terms of speed and in terms of resources and headcount to run it properly, so it only makes sense at a particular scale. It's like a m…

> - Fits on one computer (most of the market) There's one category above that, which is "Fits in memory" and that is a huge chunk of the market. I've seen first hand people getting way too cute and complicated planning for scale, and then it works out that they don't even have more than a couple of GB of data.

I remember distinctly a conversation I had in early 2012 about what I jokingly called "big memory" as killing off the (then ubiquitous) "big data" trend. Most people's sensibilities have not adjusted to a world where getting 64GB of memory in a server is a triviality, and you can 8x that without too much effort or cost.

Unless you're storing media, or you are truly "web-scale", your business data will very likely fit in 512GB.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#114

Post by vonmoltke »

misterman0 wrote: post #55
raverbashing wrote: post #45

Earlier quoted context omitted.

The issue is that 80% of "big data" isn't https://adamdrake.com/command-line-tools-can-be-235x-faster-...

That's not at all what my issue is though. I have what you might call "big data problems" and I'm so turned-off by Hadoop that I'm actually rolling my own distributed DB. But what principles other than a DFS should one look into, is the question, really. At least to me.

It's a reaction to your statement

> if you want to search over big data, which should be a pretty common procedure these days

It's not a common procedure by any stretch because the vast majority of datasets aren't really "big".

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#116

Post by kkapelon »

otabdeveloper1 wrote: post #50
pytester wrote: post #25

Earlier quoted context omitted.

>Docker solves a simple problem that everyone has. Docker provides an (IMHO pretty buggy) isolation layer that lies between "keeping things that need to be kept separate in separate folders" and "keeping things that need to be kept separate in separate virtual machines". I actually don't have the need for the level of isolation below VM and above folder very often. IMHO this level only really makes sense when contain…

90% of the time Docker is used to solve the problem of "how do I upload this bucket of Python crud to a production server?" (Replace 'Python' with any other language to taste.) A slightly smarter .tar.gz would have solved the problem just as well.

You are confusing Docker = the deployment artifact with Docker = self-contained runtime perfect for CI/CD

Longer answer here https://thenewstack.io/docker-based-dynamic-tooling-a-freque...

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#117

Post by robfig »

AmericanChopper wrote: post #42
7373737373 wrote: post #27

Earlier quoted context omitted.

All this money should have been spent on developing new, improving existing or switching to better (operating) systems which solve the resource and communication security problems, instead of creating another inner-platform effect. I hope WebAssembly goes in this direction, instead of trying to adapt to current programming language paradigms.

>All this money should have been spent on developing new, improving existing or switching to better (operating) systems which solve the resource and communication security problems But this isn’t the problem Docker is trying to solve. It’s just a problem that Docker needed to solve in order for their product to be useful, this is completely transparent to Docker users. Docker abstracts away a whole bunch of work you’…

I thought Docker builds are not generally repeatable, since they often `apt-get update && apt-get install`, which depends on the current state of external package management?

They are definitely not reproducible in the sense of building bit-for-bit identical containers, unless you use Bazel.

That being said, I've found Dockerfiles to be a much more reliable build process than most others (recently struggled to get through LogDevice's cmake-based build.. ugh).

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#118

Post by randallsquared »

Bootvis wrote: post #93
otabdeveloper1 wrote: post #62

Earlier quoted context omitted.

Indeed you're right, but the problem is that your devs' machines and your production systems are running different OS's/distributions. Nix tries to solve this, but it isn't there just yet.

I know there is cost to this solution but it's a good one: Use the same OS and similar hardware for development and production.

The cost includes making development impossible without internet access, given that devs are not going to be carrying a cluster of servers around with them.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#119

Post by danpalmer »

thrower123 wrote: post #92
danpalmer wrote: post #60

Earlier quoted context omitted.

There are broadly 3 levels of the amount of data that people have: - Fits on one computer (most of the market) - Fits on several computers (most of the rest) - Requires a significant cluster of machines (50+ to store it) Hadoop only really solves the last one. It has huge overheads in terms of speed and in terms of resources and headcount to run it properly, so it only makes sense at a particular scale. It's like a m…

> - Fits on one computer (most of the market) There's one category above that, which is "Fits in memory" and that is a huge chunk of the market. I've seen first hand people getting way too cute and complicated planning for scale, and then it works out that they don't even have more than a couple of GB of data.

I've heard of a startup spending many days of contractor time to optimise their database so they didn't have to go from the 4GB Heroku database to the 8GB one.

I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets.

I've heard of companies doing extensive work to set up Hadoop to process a few GB.

Roughly I'd suggest that "fits on my laptop" is 100TB. I could be too low on these though.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#120

Post by nimrody »

wongarsu wrote: post #72
misterman0 wrote: post #41

Earlier quoted context omitted.

>> Hadoop solved a hard problem that nobody had. I'm genuinely curious, if you want to search over big data, which should be a pretty common procedure these days, what alternatives are there to a distributed file system? A dfs seems very complex to me. And it is not clear too me what alternative system designs a dfs will outperform. Is a dfs the only solution to big data? Relational DBs do break down at a certain sca…

Relational databases work just fine for terabytes of data. Aparently there are postgres deployments with petabytes of data. There aren't all that many companies with datasets that require anything else. Google and Twitter have the problems Hadoop solves, but it's fair to say that approximately nobody else has those problems.

Whether Postgres (or similar) can handle the data depends not only on the data size but also on what you want to do with the data.

I.e., if you are generating reports, running aggregations over a large amount of data you definitely need some parallelism and Postgres isn't designed to handle these loads (certainly not petabytes). Even aggregating 100's of GB probably requires (or at least is more cost effective using) multiple machines.

Now Hadoop may not be a particularly efficient solution unless you need 100's of machines. But there is a limit to what a non-parallel single machine database can do. There are other solutions in-between.

And you really don't have to be twitter or google to handle significant amount of data these days. People are recording much more data in the hope of generating new insights and do need tools to process that data.

Post reply on HN