I wish more services would become de-facto standards and there were many implementations of the same API. I'd love more (cheaper) DynamoDB compatible APIs.
[deleted]
AWS S3 SDK breaks its compatible services
61–70 of 91 posts
Re: AWS S3 SDK breaks its compatible services
#62I was bitten by this using aws’s golang sdk.
The nuance is that Amazon updated their sdk so that the default is that the sdk no longer works for third party services that do not use the new checksum. There is a new configuration option in the sdk that either: (1) requires the checksum (and breaks s3 usage for unsupported services); or (2) only uses the checksum if the s3 service supports it.
The sdk default is (1) when this issue could have been avoided if the sdk default was (2).
Agree with all the comments that Amazon has never said or even implied that updates to their sdk would work on other s3 compatible services. However, the reality is that s3 has become a defacto standard and (unless this is a deliberate Amazon strategy) it would be relatively easy for Amazon to set this default that allows for but does not require changes to s3 compatible services or, if possible, to loudly flag these types of changes well in advance so that they don’t surprise users.
Re: AWS S3 SDK breaks its compatible services
#63Okay, but I'd never expect an AWS SDK to remain backwards compatible with third-party services and would be leery about using an AWS SDK with anything but AWS. It's on the third parties to keep their S3-compatible API, well, compatible. On the client side, you just have to pin to an older version of the AWS SDK till whatever compatible service you're using updates, right? Also, this is the first I've heard of OpenDAL…
Agree too. Just because you can get something to work doesn’t mean it’s supported. Using an Amazon S3 library on a non-Amazon service is “works but isn’t supported.” Stick to only supported things if you want reliability.
a little less charitable is amazon is throwing its weight around to quash competition before they can get started; and shove tech debt onto third parties.
I stopped giving amazon the benefit of the doubt about any aspect of their operations about 8 years ago.
Re: AWS S3 SDK breaks its compatible services
#64As a user of S3, but not any service with an S3 compatible API, this execution of the change is perfect for me as I get the benefits with 0 effort - including needing to learn about the availability of the feature. AWS is beholden first and foremost to their paying customers, and this is the best option for most S3 customers.
amazon is beholden to their shareholders. The customers are an inconvenient necessity.
Re: AWS S3 SDK breaks its compatible services
#65I got bit by this a month ago.
You can disable the new behavior by setting 2 environment variables
export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
export AWS_RESPONSE_CHECKSUM_CALCULATION=when_required
request_checksum_calculation=when_required
response_checksum_validation=when_required
https://github.com/aws/aws-sdk-go-v2/blob/release-2025-01-15...>
https://github.com/boto/boto3/issues/4392>
https://github.com/aws/aws-cli/blob/1.37.0/CHANGELOG.rst#L19>
https://github.com/aws/aws-cli/blob/2.23.0/CHANGELOG.rst#223...>
https://github.com/aws/aws-sdk-java-v2/releases/tag/2.30.0>
https://github.com/aws/aws-sdk-net/releases/tag/3.7.963.0>
https://github.com/aws/aws-sdk-php/releases/tag/3.337.0>
and wait for your S3 Compatible Object store to add a fix to support this.
Re: AWS S3 SDK breaks its compatible services
#66>OpenDAL integrates all services by directly communicating with APIs instead of relying on SDKs, protecting users from potential disruptions like this one.
Implying that the SDKs don't communicate directly with the APIs? This "problem" could have happened in OpenDAL just as it did in AWS SDKs.
Re: AWS S3 SDK breaks its compatible services
#67Earlier quoted context omitted.
Agree too. Just because you can get something to work doesn’t mean it’s supported. Using an Amazon S3 library on a non-Amazon service is “works but isn’t supported.” Stick to only supported things if you want reliability.
a little less charitable is amazon is throwing its weight around to quash competition before they can get started; and shove tech debt onto third parties. I stopped giving amazon the benefit of the doubt about any aspect of their operations about 8 years ago.
Did Amazon recommend that other 3rd party products use their SDK as their own client?
Re: AWS S3 SDK breaks its compatible services
#68Time for the community to support a standards body effort to define S3 compatibility under its own brand and standard. Having a way for vendors to publish some Level of compatibility would be a great help. Eg Tier 1 might mean basic upload/download, Tier 2 might support storage classes and lifecycle policies etc. Right now is just a confusing mess of self-attestation.
I can't imagine there is a motion here that accomplishes what you're looking for unless Amazon S3 adopts the standard. I might be wrong, but I'm betting all these 3rd party clients (including open source projects) choose to be S3 compatible because a majority of their addressable market is currently using the S3 API. "Switching over to our thing doesn't require any code refactoring, just update the url in your existi…
there's openstack (or whatever), they could maintain a backwards compatible fork with security patches if they wanted to. It's been over a decade since i had my hat in this ring, so my who owns who may be wrong. One of the "we're making an open source, self-hosted, AWS compatible platform that can be deployed". another one was XCP.
I get what everyone (all three sides) is saying, i got no love for amazon, but this does not affect me in any way - i don't use AWS APIs for anything except the aws webui to bounce something or edit route53. We mostly self-host everything[1]. mastodon, matrix, nextcloud, subsonic, librephoto, bot services, PBX, VPN.
I'm t1.micro guy and i can't stand managed services.
[1] I have some $5 vps that is a canary and i use amazon lightsail for 1 public website (512mb ram, 0.5vcpu or whatever), glacier, and route53. My goal for 2025 is to become proficient enough with bind or whatever to stop paying that $5 a month to AWS for route53 request handling. A website is one thing, but services tend to chew money on route53 with constant requests. I don't see a need to drop glacier(it's static, ~100GB of family photo backups for my aunt and whatnot) or lightsail just yet.
Re: AWS S3 SDK breaks its compatible services
#69actually the new default is sane. its WAY WAY WAY WAY better than before. especially for multi uploads. it's basically one of the features where gcloud had a insane edge. another thing was If-Match in cloud storage.
Just curious, what’s so much better about a different hash digest? Is CRC32 not fast enough?
The past did not use crc32 , it used content-md5 but not everywhere. It also did not support full object checksums for multipart.
And here is the thing that was problematic: if you uploaded parts you could not check the object after it was uploaded since aws did not calculate a hash and safe it so that you can do a head call and compare your locally generated checksum with the one online. There are cases where generating a checksum up front is infeasible and using content-md5 it was not so easy / fast to chunk the upload and generating the crc while uploading. And here is the biggest benefit:
The crc algorithm do not need to be concatinated in order. So basically you can parallelize the hash generation since you can hash 4kb chunks and concat them.
And in some cases the sdk did not generate a checksum at all.
Edit: I forgot, since full object checksums are now the default, aws can now upload multiple parts in parallel, which was not possible before. (For upload multipart)
Re: AWS S3 SDK breaks its compatible services
#70Earlier quoted context omitted.
Agree too. Just because you can get something to work doesn’t mean it’s supported. Using an Amazon S3 library on a non-Amazon service is “works but isn’t supported.” Stick to only supported things if you want reliability.
a little less charitable is amazon is throwing its weight around to quash competition before they can get started; and shove tech debt onto third parties. I stopped giving amazon the benefit of the doubt about any aspect of their operations about 8 years ago.
Less charitable or More cynical? How is Amazon supposed to track a 3rd party pulling their SDK and then reverse-engineering their own service side to work with the SDK? Assuming we're all okay with that premise to begin with, all sorts of other questions start popping up.
Do these 3rd parties get veto power over a feature they can't support?
Can they delay a launch if they need more time to make their reverse-engineered effort compatible again?
It seems a hard to defend position that this is at all Amazon's problem. The OP even links to the blog post announcing this change months ago. If users pay you for your service to remain S3-compatible that seems like its on you to make sure you live up to that promise, not Amazon.
Clicking through to the actual git issues, it definitely seems like the maintainers of Iceberg have the right mental model here too. This is their problem to fix. After re-reading this post this mostly feels like a click-baity way to advertise OpenDAL, which the author appears to be heavily involved in.