OpenSearch: AWS fork of Elasticsearch and Kibana • Hacker News Board

Re: OpenSearch: AWS fork of Elasticsearch and Kibana

#281

Post by systemvoltage » Mon, Apr 12, 2021, 10:26 PM UTC

amzn-throw wrote: post #179

TechBro8615 wrote: post #9

It's hard for me to know whether to feel bad for ES in this case. Did they bring it on themselves? Is Amazon too big and a bully? From my perspective, Amazon has made most of its profit price gouging consumers on bandwidth after vendor locking them into their ecosystem, where they bootstrap new services by wrapping open source software with some provisioning scripts, management dashboards and cookie-cutter API / cons…

> they can invest more resources into developing it than the original team ever could. I know this is a popular narrative, but as someone who works on AWS, I think you would be shocked by how small the individual dev teams are that build and maintain the services that everyone uses. I'm not going to downplay the network effects involved. Of course AWS has a tremendous advantage in being able to standardize the custom…

Could you please shed some light on how many people would be behind a product like AWS Lambda or AWS CloudWatch?

As an outsider, I would guess huge swaths of developers with a massive hierarchy. Buildings full of folks working on AWS services. I have no idea and extremely curious.

Re: OpenSearch: AWS fork of Elasticsearch and Kibana

#282

Post by narism » Mon, Apr 12, 2021, 10:27 PM UTC

bigtones wrote: post #253

dijit wrote: post #210

Earlier quoted context omitted.

You cannot use the cloud systems without paying; how on earth do you figure I’m not a customer from what I wrote?

Because you said you're "self hosted" on 'some cloud'. You don't need to pay Elastic if you're self-hosted and not using the Elastic Cloud - you can just download it and use it for free.

They could be paying Elastic for gold/plat/etc licenses on their self hosted instances or “self hosted, some cloud” could mean they run mostly self hosted with some Elastic cloud usage.

Re: OpenSearch: AWS fork of Elasticsearch and Kibana

#283

Post by marvindanig » Mon, Apr 12, 2021, 10:30 PM UTC

TechBro8615 wrote: post #9

It's hard for me to know whether to feel bad for ES in this case. Did they bring it on themselves? Is Amazon too big and a bully? From my perspective, Amazon has made most of its profit price gouging consumers on bandwidth after vendor locking them into their ecosystem, where they bootstrap new services by wrapping open source software with some provisioning scripts, management dashboards and cookie-cutter API / cons…

Coincidentally, AWS hasn't open-sourced anything that they use internally. Zilch. Nada. And yet they are using 'open source' developed by another firm (smaller is inconsequential here) to market themselves.

IMO, FOSS licensing is completely broken. Its definitions (of what is free/open) are from a boomer's era that is no longer sustainable. At least I wouldn’t want any of my FOSS projects become “corporate strategy” of any particular proto-nation.

Re: OpenSearch: AWS fork of Elasticsearch and Kibana

#284

Post by hospadar » Mon, Apr 12, 2021, 10:33 PM UTC

damon_c wrote: post #106

busterarm wrote: post #59

Earlier quoted context omitted.

Indeed, although I'd say for a majority of situations, Amazon's hosted Elasticsearch service is a complete non-starter.

Why is it a non-starter? I have a few ElasticSearch clusters that have been running for years with zero effort/downtime/hassle.

Personal experience is that AWS elasticsearch has often been missing some really useful stuff, index rebalancing, some of the utility endpoints that avoid me having to spend forever rebuilding indexes, etc. I'd be a little spooked to run something really huge and customer-facing (maybe it's possible and I'm a n00b, but for the money we pay, it should be n00b friendly).

It's always great (really, it was quite easy to get started and usually works) until it's not (a couple times have had indexes break, or had to reindex to a fresh cluster to fix balancing problems).

I'm sure LOTS of people use aws elasticsearch for big, user-facing stuff, but I often feel you'd be better off managing it yourself if it were truly critical.

Re: OpenSearch: AWS fork of Elasticsearch and Kibana

#285

Post by marricks » Mon, Apr 12, 2021, 10:34 PM UTC

dariusj18 wrote: post #211

jimmy2020 wrote: post #178

Earlier quoted context omitted.

Yes exactly, you have actually to google it maybe you'll be able to find a useful link that makes Amazon looks good.

I'm not sure what you mean.

Amazon will of course have a PR statement about how it cares about open source. It's not like Apple is known for it's commitment to open source but it has a similar page: https://developer.apple.com/opensource/

Re: OpenSearch: AWS fork of Elasticsearch and Kibana

#286

Post by cthalupa » Mon, Apr 12, 2021, 10:36 PM UTC

dimitrios1 wrote: post #235

ncann wrote: post #199

Earlier quoted context omitted.

When I think of Google I think of Go, Angular, protobuf, Bazel, Dart, Flutter, Android, Chromium, Kubernetes, Tensorflow, etc. When I think of MS I think of C#, TypeScript, VSCode, .NET When I think of Facebook I think of React, Flux, Jest, PyTorch, GraphQL, Haxl When I think of Apple I think of Swift, WebKit When I think of Amazon...I can't really think of anything

Facebook: Relay, Cassandra, React Native, RocksDB, Presto, Reason, btrfs, osquery. Apple made significant contributions to LLVM and Clang Hell IBM has more open source contributions (many through the Apache foundation and standards committees) than Amazon, which is saying something.

>Cassandra

This is an interesting one in the context of this discussion. It likely does not exist as we know it without Amazon's Dynamo paper.

Re: OpenSearch: AWS fork of Elasticsearch and Kibana

#287

Post by rabbidruster » Mon, Apr 12, 2021, 10:41 PM UTC

busterarm wrote: post #263

bmcahren wrote: post #143

I feel Amazon took the feedback from the DocumentDB/MongoDB fiasco to heart and made positive change in their approach. DocumentDB is a closed source proprietary database created by Amazon to emulate the MongoDB API. Think Google's Dalvik runtime vs Sun/Oracle's JVM. This time around we have an open source fork of ES with big backers all contributing and very permissive licensing. In both cases, Amazon gets to implem…

On the other hand, MongoDB is already sort of obsolete and trending towards death by the time that all ended up happening while ElasticSearch is hot and "new".

Where do you get the impression MongoDB is trending towards death? Seems to be growing by some metrics; the stock price has more than doubled in the last year. Not a fan myself, but still seems a long way from death to me and seem to be doing something right in enterprise market.

Re: OpenSearch: AWS fork of Elasticsearch and Kibana

#288

Post by sdesol » Mon, Apr 12, 2021, 10:53 PM UTC

jillesvangurp wrote: post #269

sdesol wrote: post #78

Will be interesting to see the resources that AWS will throw at this. You can get a sense of the resource that elastic.co is throwing at elasticsearch at https://public-001.gitsense.com/insights/github/repos?r=gith... I'm currently indexing the fork, so in about an hour or two, I'll provide the insights for the fork as well.

I scrolled back through the commits. It looks like they've been removing traces of x-pack, Elastic branding, licensing checks, etc. since the beginning of March. So far it looks like one person is doing the bulk of all that work. If there are new features, I haven't seen any. The real question is do they have a team for new feature work that they are putting together or is this just a fork that is doomed to fall behi…

I really need to add a compare feature to my tool as it would make analysis a lot easier. Having said that, there is no denying there is a huge difference in work being done in both projects over the past 30 days.

Amazon does have 16 open pull requests though, with about 7 having 20 or more file changes, but I didn't dig into them to understand their significance. Maybe it's another feature I'll need to work on.

If you look at the one year window for elasticsearch

https://public-001.gitsense.com/insights/github/repos?q=wind...

its churn and activity has been extremely consistent and I'm not sure if this is an investment Amazon can and/or is willing to make.

However, knowing enterprise, I'm not sure if this will make a huge difference as those making the decisions might not really care and they'll just accept whatever Amazon tells them.

Re: OpenSearch: AWS fork of Elasticsearch and Kibana

#289

Post by api » Mon, Apr 12, 2021, 10:53 PM UTC

TechBro8615 wrote: post #9

It's hard for me to know whether to feel bad for ES in this case. Did they bring it on themselves? Is Amazon too big and a bully? From my perspective, Amazon has made most of its profit price gouging consumers on bandwidth after vendor locking them into their ecosystem, where they bootstrap new services by wrapping open source software with some provisioning scripts, management dashboards and cookie-cutter API / cons…

You described the cloud lock in model very well, especially the bandwidth part. What they charge for outbound is nuts, and the other clouds are not much better. Inbound is free of course to make it easy to send your data in but costly to get it out.

Re: OpenSearch: AWS fork of Elasticsearch and Kibana

#290

Post by reducesuffering » Mon, Apr 12, 2021, 10:56 PM UTC

galaxyLogic wrote: post #242

ncann wrote: post #199

Earlier quoted context omitted.

When I think of Google I think of Go, Angular, protobuf, Bazel, Dart, Flutter, Android, Chromium, Kubernetes, Tensorflow, etc. When I think of MS I think of C#, TypeScript, VSCode, .NET When I think of Facebook I think of React, Flux, Jest, PyTorch, GraphQL, Haxl When I think of Apple I think of Swift, WebKit When I think of Amazon...I can't really think of anything

I would add Kotlin to Google's credit. But I think it's important to note that these companies don't contribute to open source out of any moral obligation, do they? I think they do it to tie more developers and development around their eco-systems and products. Maybe Amazon should get smart and start doing something similar. Or maybe they don't need that. But in any case I don't hold it morally against them that they…

No I would not add Kotlin to Google's credit. All the initial work and exponential adoption started with JetBrains. Google only greenlit it as an official Android dev language eventually (barring whatever OSS work they're doing on it only now).