Could this anonymous person be an employee with access to the server?
Ask HN: Anonymous person sent proof of SSH access to our production server
231–240 of 246 posts
Re: Ask HN: Anonymous person sent proof of SSH access to our production server
#232Earlier quoted context omitted.
The guy that couldn't "reverse a binary tree" came from a team that turned out to be incompetent (in operations field; they didn't see anything wrong with distributing all the packages with software using HTTP without SSL/TLS nor any cryptographic signature).
To be fair, some major Linux distros also distribute using http e.g. go to https://www.debian.org/ and check the Download link at the top of the page: it is http.
Yes, but there are some important differences:
- Debian provides SHA-512 checksums of those ISO images
- the checksums are signed cryptographically
- web server is not the only distribution point
- Debian packages are distributed signed, so once you have your OS installer somehow verified, you're much safer than with Homebrew
Granted, SHA-512 checksums and their GPG signatures are not exposed very prominently on Debian's homepage. You need to go to the listing of directory with ISO instead of clicking "download ISO" direct link.
Re: Ask HN: Anonymous person sent proof of SSH access to our production server
#233More and more I get the feeling this is a result of some giant illusion of 'serverless fairy tale, ops skills and competent infrastructure people are no longer needed' a huge part of IT market tries to live in...
Re: Ask HN: Anonymous person sent proof of SSH access to our production server
#234In addition to the wonderful technical advice already here for how to deal with the server, there is the question for how to deal with the anonymous person. If the proof contained the method of exploit I suggest something along the lines of: "Thank you for bringing this problem to our attention! We are taking steps to resolve the problem now, but would like to reward you for your work. If you let us know how you woul…
Unfortunately, it's the later - no details of exploit, just a proof. If this comes to ransom, rather than unethical/unexperienced gray hat thing, are there any good steps to take? Or hiring an expert consultancy is probably the only good option here?
Actually it's reasonable that this person haven't given you the details. If he disclosed specific way he got in you'd probably patch it and carry on. Then he'd probably find another way to get in, disclosed it too, you'd patch it and it could turn into full-time (low/un)paid job for him. Not to mention that all of the holes found by him could earlier be exploited by someone else who could left something on your server.
By sending just the proof he forces you to reconsider your approach to security and start from clean state.
Re: Ask HN: Anonymous person sent proof of SSH access to our production server
#235I had something similar thing happen several years ago when I was a grad student. Me and a couple friends were putting together some Facebook apps (when they were a new thing), and one guy didn't escape user input correctly. Some teenager from an Eastern European country (I can't remember which anymore) ran a script to figure out that we were vulnerable to SQL injections. He was a nice enough guy and didn't want anyt…
So once he had access to your box he lulled you into a false sense of security so you didn't go looking for his proxy? Nice.
Re: Ask HN: Anonymous person sent proof of SSH access to our production server
#236Do not trust random HN or serverfault answers. Cleanly shutting down the server can trigger rootkits that might wipe evidence: talk to a professional. Pulling the plug can still remove the ability to observe the behavior of the attacker: talk to a professional. Touching the disks can expose you to the risk of being accused of tampering evidence : talk to a professional.
The irony here is that your comment is a random HN comment from an account less than 10 days old :D In reality, I think random answers on HN (or any answer/advice anywhere) shouldn't be trusted, but rather taken with a grain of salt and think about whether the answer really helps you. >Touching the disks can expose you to the risk of being accused of tampering evidence I don't understand this. What do you say touchin…
> The irony here is that your comment is a random HN comment from an account less than 10 days old :D
10 days old and a throwaway. No irony here: I'm recommending the reader not to trust random HN comments including my own.
Re: Ask HN: Anonymous person sent proof of SSH access to our production server
#237Do not trust random HN or serverfault answers. Cleanly shutting down the server can trigger rootkits that might wipe evidence: talk to a professional. Pulling the plug can still remove the ability to observe the behavior of the attacker: talk to a professional. Touching the disks can expose you to the risk of being accused of tampering evidence : talk to a professional.
The irony here is that your comment is a random HN comment from an account less than 10 days old :D In reality, I think random answers on HN (or any answer/advice anywhere) shouldn't be trusted, but rather taken with a grain of salt and think about whether the answer really helps you. >Touching the disks can expose you to the risk of being accused of tampering evidence I don't understand this. What do you say touchin…
> What do you say touching the disks is? Like physical touch, or logging in and looking at the logs?
There is a reason why foresics data capture devices are so expensive and certified never to touch a bit.
> I don't think both of those can be attributed to tampering of evidence, like criminal tampering since you use the word 'accused'
There has been various cases of people accused of destruction of evidence for wiping (allegedly) compromised hosts.
Re: Ask HN: Anonymous person sent proof of SSH access to our production server
#238Earlier quoted context omitted.
None of these things you've done will remove access for this person.
Contact me if you need a recommendation. I can point you to good security consultants probably within your budget.
Thanks, will have that in mind, as its' not up to me to allocate funds for consulting, etc.
Any idea how much can such a service cost, assuming web application with a very common stack (such as Ruby on Rails + PostgreSQL)? Is it something like $5k, $10k, or $20k+? Or it really depends? Sorry if it's a very amateur question, I have no experience in dealing with such companies so have no clue how much can it cost.
Re: Ask HN: Anonymous person sent proof of SSH access to our production server
#239Enable two factor authentication on your servers, or else two factor on a bastion server and disable external ssh other than from your bastions on all other servers. I strongly recommend YubiKey, it is convenient and cheap and extremely secure. With ssh it is very easy for someone to create an ssh key that does not have a passphrase. With that, it is possible to log into the server with just the key file and nothing…
Came here to say +1 to this, definitely employ a bastion host and make sure that's the only way to SSH to your servers. This can be a little tricky to do correctly if you don't have someone on your team, but it's a valuable way to reduce your surface area to monitor. Installing fail2ban is also a very basic / smart way to discourage brute force SSH attacks on your boxes. Also you could try piping your SSH logs into s…
fail2ban still doesn't have IPv6 support.
If you use IPv6 (and you should, if possible), it's better to use an alternative that supports it (e.g. SSHGuard).
Re: Ask HN: Anonymous person sent proof of SSH access to our production server
#240In addition to the wonderful technical advice already here for how to deal with the server, there is the question for how to deal with the anonymous person. If the proof contained the method of exploit I suggest something along the lines of: "Thank you for bringing this problem to our attention! We are taking steps to resolve the problem now, but would like to reward you for your work. If you let us know how you woul…
Key here is that you need to figure out how they got in. Then negotiate terms to have them back off. Either way, they're probably being nice about it if they haven't simply 'rm -rf /'d you.
Side topic: Have you ever done this just for fun on an old system or vm? It actually stops pretty early on once it starts into /dev - removing everything actually takes a little more work.