61–70 of 103 posts
Earlier quoted context omitted.
Actually, you can do it either way since it's backed by Redis. You can set all servers to connect to the same Redis instance, or run them all individually.
What if Redis goes down?
Regardless, it's just a caching layer, and requests are passed through to the API in that case.
[deleted]
Why are you comparing S3 costs to "just" HDDs? HDD's aren't web enabled and require other hardware to run and keep running.
[0] http://blog.backblaze.com/2013/02/20/180tb-of-good-vibration...
Also see MimicDB [0]. Runs a transparent cache that responds to most S3 API calls locally using Redis to store metadata. Besides the cost savings, it's extremely fast. Listing an entire bucket with millions of objects is close to instantaneous. [0] http://mimicdb.com/
Also see MimicDB [0]. Runs a transparent cache that responds to most S3 API calls locally using Redis to store metadata. Besides the cost savings, it's extremely fast. Listing an entire bucket with millions of objects is close to instantaneous. [0] http://mimicdb.com/
Is their a SaaS version of mimicdb? I'm ready to signup right now.
Earlier quoted context omitted.
Is their a SaaS version of mimicdb? I'm ready to signup right now.
No, and it wouldn't help much if there was. The biggest benefits come from running the metadata cache locally, or at least on the same local network. The cost/time savings come from preventing external API calls.
Earlier quoted context omitted.
No, and it wouldn't help much if there was. The biggest benefits come from running the metadata cache locally, or at least on the same local network. The cost/time savings come from preventing external API calls.
If I have a bucket with 2 million key, it takes a long time and 2000 requests to get the dir listing. With mimicdb it would only be one fast request.
Send me an email nathan@nathancahill.com and I can show you how to set up a local cache. It's quite simple.
I really hated working with boto for s3. The abstractions they chose are really ambiguous and it feels like it's fighting against the underlying api. If you're building anything on top of s3, it might be easier to write a thin S3 REST client and use it directly instead of going through Boto. So many fewer surprises, and no more digging around in the boto source trying to figure out what so and so function actually do…
(disclaimer: I used to work at Smore, and I'm friends with the author, but I've been burned by Boto myself)
Yikes! That's a horrible thing for a library to do to its users. This definitely should be changed in the library. I noticed that no such Issue exists, so I opened one. https://github.com/boto/boto/issues/2078
James, It looks like a HEAD request can fulfill the same purpose at a much lower cost (this is mentioned by a comment on your Issue: (comment by kislyuk) http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEA... Shall I make the change and pull request it? I didn't want to duplicate effort.
Earlier quoted context omitted.
James, It looks like a HEAD request can fulfill the same purpose at a much lower cost (this is mentioned by a comment on your Issue: (comment by kislyuk) http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEA... Shall I make the change and pull request it? I didn't want to duplicate effort.
Aye, this does look like the right way to handle this functionality. I'm not planning on doing a PR so have at it.