Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We used Elixir's Observer to hunt down bottlenecks

blog.sequin.io

Post reply on HN
Print view View original

31–40 of 43 posts

Re: We used Elixir's Observer to hunt down bottlenecks

#31

Post by toast0 »

JohnCurran wrote: post #28
mattbaker wrote: post #21

Earlier quoted context omitted.

For us we have our app deployed to $N containers with a load balancer in front (pretty standard stuff I think?) In Erlang/Elixir you can actually override how instances of the BEAM find each other (instead of the standard EPMD daemon), so we have a module that does some DNS queries, finds the IPs of the other containers and says “hi, here’s your cluster, discovery done.” (Your setup may preclude all that, I know this…

Can you speak more to how you bypass EPMD and send the IPs of the containers to each other? That would be great for a problem we’re seeing where I work

I didn't write this blog, but it looks like it's got the right info: https://www.erlang-solutions.com/blog/erlang-and-elixir-dist...

You can skip to "Let’s use something else" if you've already got a good grasp of the knobs to tune with epmd.

Re: We used Elixir's Observer to hunt down bottlenecks

#32

Post by throwawaymaths »

busterarm wrote: post #9
ananthakumaran wrote: post #5

recon and observer_cli are the tools I reach out first to debug any issues in production. In any other language, I usually think about how to reproduce the issue locally. With Elixir, I just get into a remote shell in the affected machine and live debug the issue, and there are cases where we applied hotfix by using eval right there from the shell. The idea of the remote shell itself is alien to most languages.

And unfortunately the kind of thing that compliance flags as a big no-no once you've got any kind of filing or privacy requirements.

It is possible to certify Erlang and elixir as fips-140 compliant.

Re: We used Elixir's Observer to hunt down bottlenecks

#33

Post by jjtheblunt »

mattbaker wrote: post #21
losvedir wrote: post #2

This is really cool. We use Elixir at work, but we mostly use it in a "traditional web app" (i.e. non-Elixir) way, of Docker containers deployed to independent AWS instances. So I'm always intrigued by some of the more BEAM-specific things that folks do, like using `observer` on a remote (production??) node here, or distributed Elixir where the nodes communicate with each other, or "hot" code updates. How do companie…

For us we have our app deployed to $N containers with a load balancer in front (pretty standard stuff I think?) In Erlang/Elixir you can actually override how instances of the BEAM find each other (instead of the standard EPMD daemon), so we have a module that does some DNS queries, finds the IPs of the other containers and says “hi, here’s your cluster, discovery done.” (Your setup may preclude all that, I know this…

> deployed to $N containers

it's funny how 20 years ago "N containers" would be normal, but now everyone can read both

Re: We used Elixir's Observer to hunt down bottlenecks

#34

Post by mattbaker »

JohnCurran wrote: post #28
mattbaker wrote: post #21

Earlier quoted context omitted.

For us we have our app deployed to $N containers with a load balancer in front (pretty standard stuff I think?) In Erlang/Elixir you can actually override how instances of the BEAM find each other (instead of the standard EPMD daemon), so we have a module that does some DNS queries, finds the IPs of the other containers and says “hi, here’s your cluster, discovery done.” (Your setup may preclude all that, I know this…

Can you speak more to how you bypass EPMD and send the IPs of the containers to each other? That would be great for a problem we’re seeing where I work

Yeah, so we followed the link toast0 shared to get started:

https://www.erlang-solutions.com/blog/erlang-and-elixir-dist...

It took some work to piece together but wasn't too complicated in the end. It's internal code that I can't quickly sanitize or I'd just dump it in a gist :/ Someone on the Elixir Forum might have a template or library handy though.

Re: We used Elixir's Observer to hunt down bottlenecks

#35

Post by mattbaker »

jjtheblunt wrote: post #33
mattbaker wrote: post #21

Earlier quoted context omitted.

For us we have our app deployed to $N containers with a load balancer in front (pretty standard stuff I think?) In Erlang/Elixir you can actually override how instances of the BEAM find each other (instead of the standard EPMD daemon), so we have a module that does some DNS queries, finds the IPs of the other containers and says “hi, here’s your cluster, discovery done.” (Your setup may preclude all that, I know this…

> deployed to $N containers it's funny how 20 years ago "N containers" would be normal, but now everyone can read both

I've been in too many shell scripts lately ;)

Re: We used Elixir's Observer to hunt down bottlenecks

#36

Post by sprite »

throwawaymaths wrote: post #29
JohnCurran wrote: post #28

Earlier quoted context omitted.

Can you speak more to how you bypass EPMD and send the IPs of the containers to each other? That would be great for a problem we’re seeing where I work

Same. I'm not clustered yet, but I plan on it before EOY and that would be amazing. I think route53 has some internal routing capabilities, but some of the setup looks scary, or am I just being silly?

Not sure if it's helpful but I got clustering working on ECS mostly following this guide: https://towardsaws.com/an-elixir-migration-to-microservices-...

Re: We used Elixir's Observer to hunt down bottlenecks

#37

Post by andrewflnr »

busterarm wrote: post #9
ananthakumaran wrote: post #5

recon and observer_cli are the tools I reach out first to debug any issues in production. In any other language, I usually think about how to reproduce the issue locally. With Elixir, I just get into a remote shell in the affected machine and live debug the issue, and there are cases where we applied hotfix by using eval right there from the shell. The idea of the remote shell itself is alien to most languages.

And unfortunately the kind of thing that compliance flags as a big no-no once you've got any kind of filing or privacy requirements.

Is it any different than any other kind of direct prod access? I mean, you have to have controls for accessing prod, so it seems like you could use similar ones for REPL access, but that's logic and I know that only has a tenuous relationship with compliance...

Re: We used Elixir's Observer to hunt down bottlenecks

#38

Post by toast0 »

filmor wrote: post #23
conradfr wrote: post #11

A related anecdote: some months ago I had a memory leak inside a (greatly duplicated) genserver while repeatedly calling a lib[0] function inside it, that would result in the server basically crashing after a while. I never understood what in that lib was causing the leak but I fixed it (or more accurately mitigated it) by wrapping the call in a Task.async/1 Maybe that will help someone else one day. [0] https://hexd…

It was probably leaking refc binaries, see for example https://ferd.github.io/recon/recon.html#bin_leak-1 . Running the function (which probably parses large binaries) in a separate process ensures that it's properly garbage collected in time.

RefC binaries should be taken care of by the virtual binary heap, which seems to be from r13 (although I thought it was newer than that... maybe there's another change I'm thinking of, but can't find).

A related, but different refc binary hazard is a Process that obtains large refc binaries somehow, and makes a subbinary that it sends to another process (or ets!). The large binary is still referenced from the subbinary so there's a significant amount of excess memory. You can also run into this when binary creation is optimized to allow for appending [1], because that makes a binary of much larger than the required size (either double or 256 bytes, whichever is more). Either way, if you have a use case that naturally results in long term storage of binaries or subbinaries that allocate much more space than is really required, binary:copy/1 can be used to make a clean copy that's the exact size and isn't (yet) shared.

I've seen mnesia (ets) nodes where due to the code structure, the memory use ended up at 4x what was needed, and binary:copy added before storing with ets fixed things up with no other code change.

[1] https://www.erlang.org/doc/efficiency_guide/binaryhandling.h...

Re: We used Elixir's Observer to hunt down bottlenecks

#39

Post by depr »

davidw wrote: post #14

> Second, we passed one particularly large data structure from a manager to a pool of dedicated worker processes. This meant we were reincurring the memory cost of this data structure for each worker process. We couldn't eliminate the repetition, but reducing the data to its bare essentials before passing it down to the workers minimizes that cost. Hard to say without knowing much about the data in question, but my r…

"large binaries" is >64 bytes. https://www.erlang.org/doc/efficiency_guide/binaryhandling.h...

Re: We used Elixir's Observer to hunt down bottlenecks

#40

Post by benmmurphy »

sprite wrote: post #36
throwawaymaths wrote: post #29

Earlier quoted context omitted.

Same. I'm not clustered yet, but I plan on it before EOY and that would be amazing. I think route53 has some internal routing capabilities, but some of the setup looks scary, or am I just being silly?

Not sure if it's helpful but I got clustering working on ECS mostly following this guide: https://towardsaws.com/an-elixir-migration-to-microservices-...

just beware service discovery via DNS in AWS will return up to 8 addresses. so if you have more than 8 nodes you will get a random subsample for each request. depending on how your clustering works this may or may not be a problem. you can use the web api if you need to handle more than 8 hosts.
Post reply on HN