On “Open” Distros, Open Source, and Building a Company

Most people don't realize this yet. They are taking other, existing projects which anyone could install themselves for a long time now and framing it like they built these things for their new fork. So much of their "distribution" is just a collection of other open source projects.

In the case of the security functionality, that is backed by a company named Search Guard who have an Enterprise Version in addition to the Community Version included in the AWS fork. They forked another project, renamed it and now call it their own.

Build a database, not a database engine.

> Some open source is still written by people working for a large employer, who pay them to be dedicated to that product.

Take e.g. tensorflow by google. The software itself, is per se worthless, so they can release it as OSS as some kind of calling card or recruiting tool. The crown jewels they are guarding closely and are never letting any outsider access directly is their data.

If I read this correctly, Amazon tried to get a special deal from Elastic basically in order to produce a competing service. Elastic declined to cooperate and now Amazon is exercising their rights under the Apache license. So, far so good.

It's the provocative website that makes it seem like there's some huge conspiracy by Elastic that provoked Shay Banon's response. And not the fact that Amazon are doing what they are doing (i.e. redistribute the OSS parts of Elasticsearch), Elastic seems to have little problems with this because their license allows this and as Shay points out others have done this as well.

Amazon's complaints boil down to two things:

1) Elastic distributes a version of their software that includes non OSS components. You don't have to use these but you can. Actually Elastic offers rpms containing only the apache licensed components on their download page: https://www.elastic.co/guide/en/elasticsearch/reference/curr.... So this is maybe a bit less true than Amazon is portraying here. This would be what you'd install on a server if you are self hosting. Likewise there are OSS docker images from Elastic. They also have oss docker images. Elastic does this because up-selling this stuff is how they make money. Amazon's beef seems to be that it would be awfully nice for them if they could offer an alternate distribution more prominently without all this stuff so that they give that to their own customers.

2) Their repository contains some well documented directories that contain non Apache licensed source code for these components. As far as I know this is pretty well documented and if you are modifying files in a directory, presumably you know what you are doing. This is because they want to develop most of this stuff in the open and be able to accept pull requests against both their OSS and non OSS stuff. This seems pragmatic and preferable to producing binary distributions with some components from private repositories.

For reference, here's the text of the license file (https://github.com/elastic/elasticsearch/blob/master/LICENSE...):

"Source code in this repository is variously licensed under the Apache License Version 2.0, an Apache compatible license, or the Elastic License. Outside of the "x-pack" folder, source code in a given file is licensed under the Apache License Version 2.0, unless otherwise noted at the beginning of the file or a LICENSE file present in the directory subtree declares a separate license. Within the "x-pack" folder, source code in a given file is licensed under the Elastic License, unless otherwise noted at the beginning of the file or a LICENSE file present in the directory subtree declares a separate license.

The build produces two sets of binaries - one set that falls under the Elastic License and another set that falls under Apache License Version 2.0. The binaries that contain `-oss` in the artifact name are licensed under the Apache License Version 2.0."

So, you might not agree with this arrangement but you can get or build pure OSS quite easily and many people do. And it is hard to argue against Apache 2.x here. The license pretty much makes it obvious how stuff is licensed. This is unsurprising. Any confusion here would be a legal bug; this stuff needs to be unambiguous and it seems to be.

BTW. I think it is great that Amazon is providing their own distribution for their customers. It would be awfully nice though if they'd update it to the current version and not ship already fixed bugs. They have a history of doing this with Elasticsearch; probably because they are not on speaking terms with Elastic apparently. Contrary to the popular belief, this is not a fork. It's a binary distribution. Probably really convenient if you use their hosted version.

Otherwise, get the latest and greatest from upstream. Also refer to the Elastic website for documentation because Amazon does not bother with this either. And go to their discussion forums and stackoverflow for free support and options for getting payed support, etc. Once you figure out that they are not doing a half bad job of that, go check out elastic cloud and use that. It's easier, better, faster, and cheaper and you can get it with AWS billing. Or if you prefer on other clouds or your own cloud. And yes, I have used both elastic cloud, aws elasticsearch and have done self hosting as well.

I think this may have been the crux of the friction between the companies. Elastic sure looks to have collaborated with GCP [0][1], if not other enterprises.

> Our answer has always been a constant: send a pull request, like everybody else does. The quality will speak for itself.

I look around and spot at least three substantial bug/fixes from AWS employees (?) [2][3][4], so I don't see any lack of effort from AWS? AWS engs were sending pull requests, just like the others. That said, any sort of heavy handedness from maintainers of elasticsearch might also play a role in forcing enterprises to take control of software they depend upon so much where legally allowed. We have seen this happen with Node.js/IO.js and MariaDB/MySQL before.

> The FUD mostly comes from large(r) companies that fear what such a movement can cause.

The irony of this statement is not lost on me. Elastic is that larger company (one that crucially controls a majority of the ecosystem around elasticsearch: kibana, logstash, beats, xpack to name a few), and AWS is an upstart here with their investment in open-distro, presumably to fight FUD.

> We all sometimes need to self reflect on what and why we did that made us successful, to make sure we stay true to our course. [..] to others out there, that face so many reasons to be distracted, keep your focus [..] And last, to express our shared commitment to continue to build great products [..] It is our true north.

I think one right way for Elastic to show commitment to opensource would be to entrust the projects to a community agreed governance model, like others before them have done [5][6][7][8].

Also, what's interesting is that its not just Amazon here, but Expedia and Netflix as well putting their weight behind open-distro [9].

Like someone pointed out, this response from Elastic does seem to be emotionally charged [10].

---

[0] https://www.elastic.co/about/partners/google-cloud-platform

[1] blog.google "With the Elastic partnership in place, we’re kicking off our joint engineering work and will start rolling out managed Elasticsearch on GCP later this year." https://www.blog.google/products/google-cloud/google-cloud-p...

[2] https://github.com/elastic/elasticsearch/pull/27628/files

[3] https://github.com/elastic/elasticsearch/issues/31479

[4] https://github.com/elastic/elasticsearch/issues/29531

[5] GridGrain and https://ignite.apache.org

[6] Various networking companies and http://opennetworking.org

[7] Cloudera and https://impala.apache.org

[8] LinkedIn/Confluent and https://kafka.apache.org/

[9] https://aws.amazon.com/blogs/opensource/keeping-open-source-...

[10] "It is always Day 0 at Elastic (like the developers we serve, we use zero based numbering)."