Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On Configuration

sstephenson.us

Post reply on HN
Print view View original

51–55 of 55 posts

Re: On Configuration

#51

Post by jaylevitt »

moe wrote: post #50
jaylevitt wrote: post #47

Earlier quoted context omitted.

In single-server deployments, you're logging in because you're the only employee. In ten-server deployments, you're logging in because every machine is trying to take on a new role or aspect and you're still learning how to automate that. In thousand-server deployments you're logging in because the latest batch of Seagate drives has a statistically significant failure rate and you need to try some experimental firmwa…

In ten-server deployments [...] still learning to automate That's why your original comment tipped me off. It's hard to recover from bad patterns like the one you proposed, it's easier when you try to get as many things right as possible from the start. In thousand-server deployments you're logging in because I take it you have not worked in such an environment before. I can assure you nobody manually logs into hundr…

I take it you have not worked in such an environment before

Does AOL count? I built the mail system. It was kind of a big deal. We pushed 4,000 TPS through servers less powerful than an iPhone. I also spent time at Akamai, though I admittedly worked on a tiny, isolated test lab (150 servers or so, a fraction of a percent of the "real" server base).

I have worked with some amazing people, at all scales, and I have never seen a deployment that managed to completely avoid manual logins. No, you aren't logging into a few hundred hosts manually (well, there was this one shop... but let's not hold that up as exemplary). But you're probably logging into a few to test things out manually before you decide what your automated deployment script will be rolling out.

Even when you do deploy a fix, with a large sysadmin team it's good to know WHO exactly ran that one-off deployment script. When I said "auditing", I wasn't even thinking of PCI, or auditing against malicious actions of any kind - just simple troubleshooting forensics. But we have a great, robust way to see exactly who modified a file and when; it's the file system. Why reinvent it? (I just spent the last day trying to figure out who edited a script. Why, "postgres" did.. of course it did.)

Should you aim to do these manual config changes in the test lab first? Of course. Do you always succeed? No. Some things only get tested in the big lab. Reality intervenes. I try to plan for it.

Mind you, I'm arguing this partly to see if I can be talked out of it. I think I believe it, but I've been wrong about way bigger things. If you've achieved my automation nirvana, and you roll things out to a few thousand servers, never manually, and never wondering who did it, tell me how it works.

Re: On Configuration

#52

Post by moe »

jaylevitt wrote: post #51
moe wrote: post #50

Earlier quoted context omitted.

In ten-server deployments [...] still learning to automate That's why your original comment tipped me off. It's hard to recover from bad patterns like the one you proposed, it's easier when you try to get as many things right as possible from the start. In thousand-server deployments you're logging in because I take it you have not worked in such an environment before. I can assure you nobody manually logs into hundr…

I take it you have not worked in such an environment before Does AOL count? I built the mail system. It was kind of a big deal. We pushed 4,000 TPS through servers less powerful than an iPhone. I also spent time at Akamai, though I admittedly worked on a tiny, isolated test lab (150 servers or so, a fraction of a percent of the "real" server base). I have worked with some amazing people, at all scales, and I have nev…

But we have a great, robust way to see exactly who modified a file and when; it's the file system. Why reinvent it?

That's a strange statement given your background.

Since when does editing a file change the owner or leave an audit-trail behind, on stock linux?

If you're running auditd then that's a different story but I've never seen that used in production for this purpose because it's hard to get right and because there are so many simpler options much better suited to the task (version control, puppet, chef, etc.).

Auditing or not, in my experience you inevitably end up in permission-hell when multiple users mess with files that need to be readable by a common gid (sticky bits, umask, something is always off).

If you really found a way to make this work at scale, a detailed write-up would be interesting (to me).

In my experience that way lies unmitigated madness. I've had to clean up multiple such "sudo-style" deployments in the past.

Re: On Configuration

#53

Post by jaylevitt »

moe wrote: post #52
jaylevitt wrote: post #51

Earlier quoted context omitted.

I take it you have not worked in such an environment before Does AOL count? I built the mail system. It was kind of a big deal. We pushed 4,000 TPS through servers less powerful than an iPhone. I also spent time at Akamai, though I admittedly worked on a tiny, isolated test lab (150 servers or so, a fraction of a percent of the "real" server base). I have worked with some amazing people, at all scales, and I have nev…

But we have a great, robust way to see exactly who modified a file and when; it's the file system. Why reinvent it? That's a strange statement given your background. Since when does editing a file change the owner or leave an audit-trail behind, on stock linux? If you're running auditd then that's a different story but I've never seen that used in production for this purpose because it's hard to get right and because…

GAH! See, I just slipped up and showed my age. I'm thinking of the Stratus VOS filesystem, which did change the modifier when you edited a file. You're right; UNIX/Linux files have an owner, which doesn't change on modification. Which shoots a big blazing hole in my argument. You can back into who modified what if files were changed in a big batch, and they also created a few new files - but you could also back into that via login times.

Since those days, it's less that I've seen the individual-user system work at scale - it's that I've seen the single role account fail, repeatedly. I've managed to avoid permissions hell with a few well-chosen supplemental groups, but maybe I'm too optimistic that what used to work in one environment will work in another. Ask me in six months. :)

Re: On Configuration

#54

Post by bdunbar »

moe wrote: post #49
bdunbar wrote: post #44

Earlier quoted context omitted.

And when you do then 'prod' and 'root' work just fine. They do until you have to show auditors who logged in and when. Scattering user-accounts across machines is a maintenance headache That's why God invented Active Directory.

They do until you have to show auditors who logged in and when. SSH has logging. Also if you have audit-requirements then the login-log is normally the least of your worries. That's why God invented Active Directory. That must be one cruel god you have there...

I agree that SSH logging is sufficient. That was not the finding of our auditor, per best practices, blah blah. Thank SOX for that one.

That must be one cruel god you have there...

I don't have to _deal_ with it directly, just use it to authenticate logins on (most) of my unix servers.

Seen from afar, it's not that bad.

Now, she is a jealous god, as perceived by her acolytes.

Nobody really wants to hear about how, during last year's DR testing at IBM, my stuff was up and running waaay before AD was working.

Re: On Configuration

#55

Post by gosu »

tikhonj wrote: post #23
groby_b wrote: post #17

Earlier quoted context omitted.

Just keep in mind that O(1) = O(1 + c). Depending on the relative size of n and c, you might want to stop and think :)

Happily, this gets amortized as n increases. So I just have to use Emacs more to make it worth it. A delicious proposition. (Hey, I should totally write a mode for browsing HN from Emacs! Then I'll have no need to ever leave :P.)

Try Conkeror. It's exactly what you want.

http://www.conkeror.org/

Post reply on HN