DreamObjects is down • Hacker News Board

Re: DreamObjects is down

#21

Post by zimbatm » Sat, Jun 14, 2014, 4:40 PM UTC

thezach wrote: post #14

hyperliner wrote: post #2

It seems they copied AWS S3 too well , including its "feature" of going down every now and then and it's overall lack of reliability. Nice!

I've only had very minimal issues with S3... one of the things thats holding me back from switching to a cheaper provider from Amazon is if its not broke, don't fix it.

Individual blob access is not very reliable on S3. In a previous company uploading big-ish (1-10000MB) files was part of the service. We would see failing uploads or slow writes all the time. Make sure to queue and wrap your upload script with retries.

Re: DreamObjects is down

#22

Post by McGlockenshire » Sat, Jun 14, 2014, 5:04 PM UTC

tedchs wrote: post #6

Status update says they're running Ceph, an S3 work-alike. I don't want to say "clone" because nobody outside Amazon really knows how S3 works, and Ceph merely has an external HTTP API that supposedly works the same way. Is anybody running Ceph at scale and might be able to comment on what broke down?

> Is anybody running Ceph at scale and might be able to comment on what broke down?

Ceph came from Dreamhost, so it's very likely that they're the best subject matter experts outside of Inktank.

Re: DreamObjects is down

#23

Post by epistasis » Sat, Jun 14, 2014, 5:34 PM UTC

The current status update says that its now resolved:

>Update June 14th, 2014, 10:20 AM PDT: Stability on the DreamObjects cluster has been restored. Requests appear to be resolving properly now that the system has had time to re-balance itself. Our test are reporting properly now. If you do have any questions or concerns, please contact support contact support.

I'm quite interested to know what went on in particular, as I'm far more interested in Ceph than in commercial object stores that I can't extend. Librados is pretty damn awesome too, and I can foresee implementing some highly distributed storage through that directly.

With DreamObjects, it sounds like some API servers went down, and failure happened such that it couldn't serve some requests until the appropriate nodes came back.

It appears that with Ceph it will be easy to keep enough replicas such that data is not lost, but high availability is still being hashed out. Hopefully the lessons from this failure guarantee that this particular failure mode doesn't happen again.

Re: DreamObjects is down

#24

Post by derefr » Sat, Jun 14, 2014, 6:43 PM UTC

tedchs wrote: post #6

Status update says they're running Ceph, an S3 work-alike. I don't want to say "clone" because nobody outside Amazon really knows how S3 works, and Ceph merely has an external HTTP API that supposedly works the same way. Is anybody running Ceph at scale and might be able to comment on what broke down?

> I don't want to say "clone" because nobody outside Amazon really knows how S3 works

You can deduce quite a bit about S3's system architecture from the kind of time and consistency guarantees it provides, the optimal and pessimal cases for bucket key naming, etc. Riak CS hits pretty much all those same notes, so I would hazard that you can get a good high-level sense of S3's architecture by looking at Riak's.

Re: DreamObjects is down

#25

Post by ilaksh » Sat, Jun 14, 2014, 7:30 PM UTC

How is the performance of DreamObjects?

For people who are actually on a budget for their projects, which is more than you might think, if the reliability is OK, this might make a lot of sense over paying much more to use the real S3 for backups.