Live data from Hacker News

You've heard of “yes men”. Now learn about “no men” (2011)

rachelbythebay.com

91–100 of 151 posts

Re: You've heard of “yes men”. Now learn about “no men” (2011)

#91
I agree with the author's overall premise, which is that other people who don't own and who are not responsible for your system should not have authority over how you build and operate it. One of the things that I like about the current company where I work is that owners always have the final say about their systems, not any external technical authority. There is no such authority who could say no where I work (with rare and limited exceptions), and I'm glad for that; service owners make the judgment call about trade-offs.

> "so-and-so says no swap in production".

That said, while I understand the point of the article is something different, I broadly agree with this advice: generally no swap in production.

If you need swap, then that's probably a sign that the workload isn't matched well to the resources of the machine (unless the workload was specifically designed to employ swap, which most are not).

The problem is that: maybe most of the time the application's working set will fit within memory, and so it will perform well, but some of the time it will begin to swap and performance will degrade. The result of adding swap to a system is non-deterministic performance.

If you're building high-availability, high-performance production systems, then you don't want the potentiality of arbitrary performance degradation or non-deterministic behavior. Instead, you would prefer to solve the root cause -- such as by increasing the amount of memory available to the process or system, so that swap is no longer required.

An engineer adding swap to a system is like a doctor treating a patient's symptoms instead of curing their disease. Both are useful but one is preferable. If you encounter a situation where you could solve an application problem by adding swap, then an even better solution is to address the root cause -- ensure the application doesn't run out of physical memory. Swap is a band-aid that masks a risk better addressed directly. Hence: no swap in production.

(On the other hand, it might be fine to employ swap in production for services that aren't performance sensitive, and where it's preferable to accept the performance hit from using swap rather than pay more for better hardware. That's a fine trade-off if consciously made. In my experience, variations in performance are usually much more problematic to a business than changes to cost.)

Re: You've heard of “yes men”. Now learn about “no men” (2011)

#92
I used to have a “don’t know/can’t remember” man in my team.

Can you recall what this code you wrote previously do? “I can’t remember.”

Do you know where the files are placed? “I don’t know.”

Where are the documentations for xxx. “I don’t know.”

Do you know the difference between xxx in testing in production server? “I can’t remember.”

It was frustrating.

9 out of 10 questions were “I don’t knows” and “I can’t remembers”.

I’d much rather he said yes or no.

Re: You've heard of “yes men”. Now learn about “no men” (2011)

#93
post #8

I definitely felt this decades ago when I started in the industry. It's 2019 now though. only 8 years later, but a lot of things changed. The industry is growing fast, faster than knowledge can be shared even with all the power of the internet. It's very bottom up, with newer engineers severely outnumbering those that have been around. People are making fat salaries and companies are catering to their whims. They/we'…

Tim Cook agrees - "The hardest decisions we make are all the things not to work on. There is lots of things we'd like to work on but we can't do everything great. Most companies do larger and larger portfolios because its so easy to add. It gets hard to stay focused. Yet we know, we can only do our best work if we stay focused"

Re: You've heard of “yes men”. Now learn about “no men” (2011)

#94
post #83
post #12

Isn't no swap in prod fairly standard? Kubernetes pre-flight checks check it, for example. If enabling swap 'fixes' it, isn't the proper fix to provision more memory?

Wait what? How do machines running K8s exec processes when they are near their physical memory limit? Swap does more than just provide a place for pages to get evicted to, it also increases the virtual address space.

I don't think that's true, at least on Linux.

This program uses 1 TiB of virtual address space, and runs fine on a machine with no swap (and much less than 1 TiB of physical RAM).

    #include 
    #include 
    
    int main()
    {
        size_t const len = 1024 * 1024 * 1024; // 1 GB
        for (int i = 0; i 
The space won't be associated with any physical object (whether RAM or swap) until you actually touch it.

Re: You've heard of “yes men”. Now learn about “no men” (2011)

#95
post #8

I definitely felt this decades ago when I started in the industry. It's 2019 now though. only 8 years later, but a lot of things changed. The industry is growing fast, faster than knowledge can be shared even with all the power of the internet. It's very bottom up, with newer engineers severely outnumbering those that have been around. People are making fat salaries and companies are catering to their whims. They/we'…

[deleted]

Re: You've heard of “yes men”. Now learn about “no men” (2011)

#96
post #93
post #8

I definitely felt this decades ago when I started in the industry. It's 2019 now though. only 8 years later, but a lot of things changed. The industry is growing fast, faster than knowledge can be shared even with all the power of the internet. It's very bottom up, with newer engineers severely outnumbering those that have been around. People are making fat salaries and companies are catering to their whims. They/we'…

Tim Cook agrees - "The hardest decisions we make are all the things not to work on. There is lots of things we'd like to work on but we can't do everything great. Most companies do larger and larger portfolios because its so easy to add. It gets hard to stay focused. Yet we know, we can only do our best work if we stay focused"

There's a big difference between saying "no" to the right things (valuable) and a borderline oppositional personality disorder (not valuable). There's an art to saying no. Ideally you look for ways to say yes, and try and distill the nuggets of value and work them into your future plans while acknowledging that now may not be the right time to pursue them. Or nerd-sniping them with other ideas to leverage their enthusiasm and bright-eyed/bushy-tailed attitudes. Likely they're just looking for impact and don't know how to get it.

Otherwise you're kind of a dick, and nobody wants to work with that. Unless you're in charge and they can't get away, I guess.

Re: You've heard of “yes men”. Now learn about “no men” (2011)

#97
post #91

I agree with the author's overall premise, which is that other people who don't own and who are not responsible for your system should not have authority over how you build and operate it. One of the things that I like about the current company where I work is that owners always have the final say about their systems, not any external technical authority. There is no such authority who could say no where I work (with…

But not having swap will cause the machine to misbehave (https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConseque...). Swap in this situation acts like a backup, surely you would prefer temporal degradation then total system shutdown. The solution here is continuous monitoring of free memory (we use datadog) + swap.

Re: You've heard of “yes men”. Now learn about “no men” (2011)

#98
post #50
post #8

I definitely felt this decades ago when I started in the industry. It's 2019 now though. only 8 years later, but a lot of things changed. The industry is growing fast, faster than knowledge can be shared even with all the power of the internet. It's very bottom up, with newer engineers severely outnumbering those that have been around. People are making fat salaries and companies are catering to their whims. They/we'…

That is why you write that in a standards document, along with the reasons why and have a consensus about it. That way it becomes impersonal. Same thing with auto formatters and linters sidestepping a lot of pointless bickering and bike shedding.

Linters are probably the single biggest productivity boost an organization can engage if they don't yet have one.

Re: You've heard of “yes men”. Now learn about “no men” (2011)

#100
post #91

I agree with the author's overall premise, which is that other people who don't own and who are not responsible for your system should not have authority over how you build and operate it. One of the things that I like about the current company where I work is that owners always have the final say about their systems, not any external technical authority. There is no such authority who could say no where I work (with…

I think what may be happening here is that at some point the Linux kernel's behavior has changed and things run more smoothly with swap than without, even if you aren't swapping appreciably. There might be some code path it hits when it needs to flush cache or something like that that causes issue.
Post reply on HN