AWS S3 SDK breaks its compatible services • Hacker News Board

Re: AWS S3 SDK breaks its compatible services

#31

Post by arccy » Thu, Feb 20, 2025, 8:23 PM UTC

pradn wrote: post #27

profmonocle wrote: post #22

Treating a proprietary API as a standard is risky - this is a good example of why. From Amazon's point of view there's no reason to keep the S3 SDK backwards compatible with old versions of the S3 service, because they control the S3 service. Once this feature was rolled out in all regions, it was safe to update the SDK to expect it. Amazon may not be actively hostile to using their SDK with third party services, but…

Well, you do have to worry about customers using old client libraries / SDKs, even if your whole backend has migrated to a new API. Many customers don't like to upgrade unless they need to. It can be significant toil for them. So, you do see some tail traffic in the wild that comes from SDKs released years ago. For a service as big as S3, I bet they get traffic from SDKs ever longer than that.

the server has to be compatible with old clients, but new clients don't have to be compatible with old servers, which is the case here

Re: AWS S3 SDK breaks its compatible services

#32

Post by femto113 » Thu, Feb 20, 2025, 8:26 PM UTC

> the AWS team has implemented it poorly by enforcing it

This is whiny and just wrong. Best behavior by default is always the right choice for an SDK. Libraries/tools/clients/SDKs break backwards compatibility all the time. That's exactly what semver version pinning is for, and that's a fundamental feature of every dependency management system.

AWS handled this exactly right IMO. Change was introduced in Python SDK version 1.36.0 which clearly indicatesbreaking API changes, and their changelog also explicitly mentions this new default

   api-change:``s3``: [``botocore``] This change enhances integrity protections for new SDK requests to S3. S3 SDKs now support the CRC64NVME checksum algorithm, full object checksums for multipart S3 objects, and new default integrity protections for S3 requests.

https://github.com/boto/boto3/blob/2e2eac05ba9c67f0ab285efe5...

Re: AWS S3 SDK breaks its compatible services

#33

Post by tonyhart7 » Thu, Feb 20, 2025, 8:26 PM UTC

js2 wrote: post #6

Okay, but I'd never expect an AWS SDK to remain backwards compatible with third-party services and would be leery about using an AWS SDK with anything but AWS. It's on the third parties to keep their S3-compatible API, well, compatible. On the client side, you just have to pin to an older version of the AWS SDK till whatever compatible service you're using updates, right? Also, this is the first I've heard of OpenDAL…

its unified storage layer well you guessed it using rust

Re: AWS S3 SDK breaks its compatible services

#34

Post by aaronbrethorst » Thu, Feb 20, 2025, 8:28 PM UTC

benatkin wrote: post #29

aaronbrethorst wrote: post #2

Many S3-compatible services are recommending that their users use the S3 SDK directly, and changing the default settings in this way can have a direct impact on their users. This is wholly predictable; AWS isn't in the business of letting other companies OpenSearch them.

That’s an odd way to describe it. Elasticsearch was a wrapper around Lucene. Had they started with a restrictive license rather than wait until it got popular under the nonrestrictive license, Solr might have taken off more. OpenSearch is what the community needed. That Amazon did it is fine.

Here's a good discussion from here in 2021 about the fork that colors my perception: https://news.ycombinator.com/item?id=26780848

Re: AWS S3 SDK breaks its compatible services

#35

Post by benatkin » Thu, Feb 20, 2025, 8:33 PM UTC

aaronbrethorst wrote: post #34

benatkin wrote: post #29

Earlier quoted context omitted.

That’s an odd way to describe it. Elasticsearch was a wrapper around Lucene. Had they started with a restrictive license rather than wait until it got popular under the nonrestrictive license, Solr might have taken off more. OpenSearch is what the community needed. That Amazon did it is fine.

Here's a good discussion from here in 2021 about the fork that colors my perception: https://news.ycombinator.com/item?id=26780848

Yeah Amazon’s motivations aren’t great, but it’s occupying a space that opened up when Elastic changed the license on Elasticsearch. Nobody’s going to create another permissively licensed alternative to it just because they’re annoyed it’s an Amazon project.

Re: AWS S3 SDK breaks its compatible services

#36

Post by freedomben » Thu, Feb 20, 2025, 8:33 PM UTC

Vosporos wrote: post #15

Y'all put blind trust in a proprietary API SDK and all the eggs in the same basket? Just like that??

I'm pretty sure the SDKSs are Apache licensed so not proprietary. Obviously the backend is proprietary, but that doesn't affect the SDK itself. You can pin it, fork it, vendor it, whatever you want, just like any other FOSS project that you don't control.

I think more the issue is that people started thinking of the AWS SDK as a generic open source library rather than what it should be thought of: an open source project run by a particular vendor who not only doesn't care about helping you use competitors, but actively wants to make that difficult. I would guess the truth is somewhere in the middle, but I think the healthy thing to do is treat it like the extreme end.

Re: AWS S3 SDK breaks its compatible services

#37

Post by freedomben » Thu, Feb 20, 2025, 8:34 PM UTC

fijiaarone wrote: post #16

[flagged]

if you're going that direction, rclone is a lot more of an s3-like than scp is. Also a phenomenal piece of software

Re: AWS S3 SDK breaks its compatible services

#38

Post by freedomben » Thu, Feb 20, 2025, 8:36 PM UTC

pradn wrote: post #27

profmonocle wrote: post #22

Treating a proprietary API as a standard is risky - this is a good example of why. From Amazon's point of view there's no reason to keep the S3 SDK backwards compatible with old versions of the S3 service, because they control the S3 service. Once this feature was rolled out in all regions, it was safe to update the SDK to expect it. Amazon may not be actively hostile to using their SDK with third party services, but…

Well, you do have to worry about customers using old client libraries / SDKs, even if your whole backend has migrated to a new API. Many customers don't like to upgrade unless they need to. It can be significant toil for them. So, you do see some tail traffic in the wild that comes from SDKs released years ago. For a service as big as S3, I bet they get traffic from SDKs ever longer than that.

I think you've got the contract backwards. The server can't break old clients, but new clients can break the old server since Amazon controls the old server and can ensure that all of them are fully upgraded before the client updates are published.

Re: AWS S3 SDK breaks its compatible services

#39

Post by dastbe » Thu, Feb 20, 2025, 8:39 PM UTC

with the open sourcing of smithy and the raw s3 specs, open source should be able to build clients in just about any language they care about while keeping relatively high fidelity with implementations s3 releases.

Re: AWS S3 SDK breaks its compatible services

#40

Post by freedomben » Thu, Feb 20, 2025, 8:41 PM UTC

profmonocle wrote: post #22

Treating a proprietary API as a standard is risky - this is a good example of why. From Amazon's point of view there's no reason to keep the S3 SDK backwards compatible with old versions of the S3 service, because they control the S3 service. Once this feature was rolled out in all regions, it was safe to update the SDK to expect it. Amazon may not be actively hostile to using their SDK with third party services, but…

This is the correct take IMHO. I generally dislike Amazon (and when it comes to things like the Kindle, I actively hate them for the harm they are doing), but I think this is the key. S3 is not and never has been advertised as an open standard. It's API was copied/implemented by a lot of other services, but keeping those working is not Amazon's responsibility. It's on the developer of a service using those competitors to ensure they are using a compatible client.

I do think some of the vendors did themselves an active disservice by encouraging use of the aws sdk in their documentation/examples, but again that's on the vendor, not on Amazon who is an unrelated third party in that arrangement.

I would guess that Amazon didn't have hostile intentions here, but truthfully their intentions are irrelevant as Amazon shouldn't be part of the equation. For example, if I use Backblaze, the business relationship here is between me and Backblaze. My choice to use the AWS SDK for that doesn't make Amazon part of it anymore than it would if I found some random chunk of code on github and used that instead.