Bonsai Hosted ElasticSearch Is Down

#1

Post by sstarr » Thu, Jun 19, 2014, 2:06 PM UTC

Bonsai Hosted ElasticSearch Is Down
status.bonsai.io

Re: Bonsai Hosted ElasticSearch Is Down

#2

Post by dorfsmay » Thu, Jun 19, 2014, 2:29 PM UTC

Any word on how those accounts are getting compromised?

Have they been complacent (easy password to guess, keys easy to be compromised (maybe in a public github repo)), or could there be some whole in the AWS secutiy model?

Re: Bonsai Hosted ElasticSearch Is Down

#3

Post by ihsw » Thu, Jun 19, 2014, 2:30 PM UTC

How were they 0wned? Lack of MFA, rogue API key, or something else? Are full-access accounts being handed out willy-nilly instead of IAM accounts?

AWS Multi-Factor Authentication (MFA):

http://aws.amazon.com/iam/details/mfa/

AWS Identity Access and Management (IAM):

http://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPract...

Managing your AWS API Keys:

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSG...

Go a step further with your AWS API keys and use AWS' API access logging (CloudTrail):

http://aws.amazon.com/cloudtrail/

Don't get burned. Check your stuff out.

Re: Bonsai Hosted ElasticSearch Is Down

#4

Post by huntermeyer » Thu, Jun 19, 2014, 2:32 PM UTC

This brought my app down. http://jrdevjobs.com. Our shards were all missing from Bonsai. We looped through each model and saved it.

We're back up.

Re: Bonsai Hosted ElasticSearch Is Down

#5

Post by grandalf » Thu, Jun 19, 2014, 2:44 PM UTC

Not to focus on this when they are experiencing downtime, but Bonsai has been one of the least reliable service providers I've ever used.

Re: Bonsai Hosted ElasticSearch Is Down

#6

Post by count » Thu, Jun 19, 2014, 2:44 PM UTC

dorfsmay wrote: post #2

Any word on how those accounts are getting compromised? Have they been complacent (easy password to guess, keys easy to be compromised (maybe in a public github repo)), or could there be some whole in the AWS secutiy model?

If there were a hole in the AWS security model for this, I think it'd be pretty obvious pretty quickly, given what happens when US-East takes a dive...

This happens constantly, and it's almost always through lack of best practices (as mentioned in higher up comment - IAM, MFA, etc.).

Re: Bonsai Hosted ElasticSearch Is Down

#7

Post by jayzalowitz » Thu, Jun 19, 2014, 3:03 PM UTC

This sucks... I am happy we just put our search cluster on elasticbeanstalk atm, but I wish we had more services like this running.. good news is new security practices will hop up everywhere because of this.

Re: Bonsai Hosted ElasticSearch Is Down

#8

Post by vegardx » Thu, Jun 19, 2014, 3:03 PM UTC

grandalf wrote: post #5

Not to focus on this when they are experiencing downtime, but Bonsai has been one of the least reliable service providers I've ever used.

Good that it's pretty easy to change elastic search provider with little downtime. I'd recommend checking out found.no. We've been pleased with performance and stability. Heck - I can't recall any downtime at all.

Re: Bonsai Hosted ElasticSearch Is Down

#9

Post by bdcravens » Thu, Jun 19, 2014, 3:13 PM UTC

Full-blown AWS console compromise - this sounds similar to what happened to Code Spaces (https://news.ycombinator.com/item?id=7909791) Is there a new vulnerability?

Re: Bonsai Hosted ElasticSearch Is Down

#10

Post by huslage » Thu, Jun 19, 2014, 3:43 PM UTC

AWS needs to improve usability of IAM so that it gets broader adoption. The learning curve is non-trivial.