Matrix.org Security Incident

Did the blog get hacked (again?) in between this being posted and now? It has what looks like password hashes and `uname -a` from every(?) server in their infrastructure.

This is about as bad as IR can get: you realize you got hacked, you re-build your entire infrastructure and publicly say it's fixed, and then you get popped again...

Re: Matrix.org Security Incident

#5

Post by snailmailman » Fri, Apr 12, 2019, 5:43 AM UTC

iancarroll wrote: post #4

Did the blog get hacked (again?) in between this being posted and now? It has what looks like password hashes and `uname -a` from every(?) server in their infrastructure. This is about as bad as IR can get: you realize you got hacked, you re-build your entire infrastructure and publicly say it's fixed, and then you get popped again...

i think so. here is a wayback machine link to the previous blog post https://web.archive.org/web/20190412000400/https://matrix.or...

it stated "Having fully flushed out the attacker [...]" which i guess turned out to be false :-/

also im getting invalid HTTPS certs on the blog now. for some reason im getting a cert that looks like its for github.com ?

edit: now im getting a lets encrypt cert on matrix.org, but a cloudflare SSL error page when i go to www.matrix.org ? the lets encrypt cert looks like it was just issued about an hour ago.

edit2: i guess both with and without www. are lets encrypt, but the with www. cert was issued back in february (and gives a cloudflare SSL error page), while without www. was issued today. (and gives the current hacked message)

Re: Matrix.org Security Incident

#6

Post by deepwell » Fri, Apr 12, 2019, 5:57 AM UTC

snailmailman wrote: post #5

iancarroll wrote: post #4

Did the blog get hacked (again?) in between this being posted and now? It has what looks like password hashes and `uname -a` from every(?) server in their infrastructure. This is about as bad as IR can get: you realize you got hacked, you re-build your entire infrastructure and publicly say it's fixed, and then you get popped again...

i think so. here is a wayback machine link to the previous blog post https://web.archive.org/web/20190412000400/https://matrix.or... it stated "Having fully flushed out the attacker [...]" which i guess turned out to be false :-/ also im getting invalid HTTPS certs on the blog now. for some reason im getting a cert that looks like its for github.com ? edit: now im getting a lets encrypt cert on matrix.org, but a clou…

Here is the message of the hacker, after the matrix.org admins allegedly cleaned everything up and wrote the original blog post: https://web.archive.org/web/20190412055614/https://matrix.or...

Re: Matrix.org Security Incident

#7

Post by ge0rg » Fri, Apr 12, 2019, 6:09 AM UTC

The most favorable reading of the current defacement page is that the attackers still controls the DNS, but no other parts of the infrastructure.

Otherwise, the page probably wouldn't run off github.

Re: Matrix.org Security Incident

#8

Post by localhostdotdev » Fri, Apr 12, 2019, 6:17 AM UTC

content before it gets fixed:

    Time for actual transparency.

    [list of servers, uname -a for each]

    root@[name]:/var/lib/postgresql# df -h
    [list of partitions]

    $ cat users.txt | grep [name] | head -n1
    @[name]:matrix.org|[hash]
    $ wc -l users.txt
    [~6M users]

    See you soon.

(affects whole site, even https://matrix.org, site is on jekyll BTW)

Re: Matrix.org Security Incident

#9

Post by mkj » Fri, Apr 12, 2019, 6:18 AM UTC

> As we had to log out all users from matrix.org, if you do not have backups of your encryption keys you will not be able to read your encrypted conversation history

That seems like a fairly bad usability/security design?

Re: Matrix.org Security Incident

#10

Post by Perceptes » Fri, Apr 12, 2019, 7:31 AM UTC

deepwell wrote: post #3

arunc wrote: post #2

Why was Jenkins running on a production server?

it was not, read again

But it does seem to be the case that the same SSH key pair that was used to access Jenkins also provided access to the production infrastructure. Unless I'm misunderstanding the nature of the attack.