Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We used Elixir's Observer to hunt down bottlenecks

blog.sequin.io

Post reply on HN
Print view View original

21–30 of 43 posts

Re: We used Elixir's Observer to hunt down bottlenecks

#21

Post by mattbaker »

losvedir wrote: post #2

This is really cool. We use Elixir at work, but we mostly use it in a "traditional web app" (i.e. non-Elixir) way, of Docker containers deployed to independent AWS instances. So I'm always intrigued by some of the more BEAM-specific things that folks do, like using `observer` on a remote (production??) node here, or distributed Elixir where the nodes communicate with each other, or "hot" code updates. How do companie…

For us we have our app deployed to $N containers with a load balancer in front (pretty standard stuff I think?)

In Erlang/Elixir you can actually override how instances of the BEAM find each other (instead of the standard EPMD daemon), so we have a module that does some DNS queries, finds the IPs of the other containers and says “hi, here’s your cluster, discovery done.” (Your setup may preclude all that, I know this all depends on how a system’s architected.)

After doing that we were free to use all of Erlang’s cool cluster stuff! In our case we have in-memory caches for a few things, and if a given instance does a lookup because of a cache miss it broadcasts a message to all the other nodes saying “I just looked up $expensive_thing, here’s its value” so they don’t have to do the lookup themselves, they just cache that value, so you end up with a little distributed cache with a few lines of code. In our case, btw, these cache entries are short lived and a little inconsistency does us no harm if one of our instances misses the message, networks are networks, but it’s been great!

Anyway, I think it’s super cool and I’d encourage you to play around if you get the chance.

Also the observer is just amazing. We’ve debugged some pretty weird memory and cpu usage issues with it, I have some internal blog posts, maybe I should see if I could make them public.

Re: We used Elixir's Observer to hunt down bottlenecks

#22

Post by mattbaker »

busterarm wrote: post #9
ananthakumaran wrote: post #5

recon and observer_cli are the tools I reach out first to debug any issues in production. In any other language, I usually think about how to reproduce the issue locally. With Elixir, I just get into a remote shell in the affected machine and live debug the issue, and there are cases where we applied hotfix by using eval right there from the shell. The idea of the remote shell itself is alien to most languages.

And unfortunately the kind of thing that compliance flags as a big no-no once you've got any kind of filing or privacy requirements.

Still wildly useful debugging things locally too!

Re: We used Elixir's Observer to hunt down bottlenecks

#23

Post by filmor »

conradfr wrote: post #11

A related anecdote: some months ago I had a memory leak inside a (greatly duplicated) genserver while repeatedly calling a lib[0] function inside it, that would result in the server basically crashing after a while. I never understood what in that lib was causing the leak but I fixed it (or more accurately mitigated it) by wrapping the call in a Task.async/1 Maybe that will help someone else one day. [0] https://hexd…

It was probably leaking refc binaries, see for example https://ferd.github.io/recon/recon.html#bin_leak-1.

Running the function (which probably parses large binaries) in a separate process ensures that it's properly garbage collected in time.

Re: We used Elixir's Observer to hunt down bottlenecks

#24

Post by conradfr »

filmor wrote: post #23
conradfr wrote: post #11

A related anecdote: some months ago I had a memory leak inside a (greatly duplicated) genserver while repeatedly calling a lib[0] function inside it, that would result in the server basically crashing after a while. I never understood what in that lib was causing the leak but I fixed it (or more accurately mitigated it) by wrapping the call in a Task.async/1 Maybe that will help someone else one day. [0] https://hexd…

It was probably leaking refc binaries, see for example https://ferd.github.io/recon/recon.html#bin_leak-1 . Running the function (which probably parses large binaries) in a separate process ensures that it's properly garbage collected in time.

Interesting thanks.

Yes that could be it.

Re: We used Elixir's Observer to hunt down bottlenecks

#26

Post by _acco »

cpursley wrote: post #25

Sequin is really cool! Are y'all listening postgres WAL?

Thanks! We considered using Postgres' WAL but decided not to for the time being.

Our solution now uses trigger functions. These trigger functions fire whenever a create/update/delete happens on a Sequin table. They insert a row into a log table. That log table is processed by our workers to send changes to the upstream API.

The advantage of using trigger functions + a log table are all about ease of use and compatibility: our customers don't have to do anything fancy to setup Sequin, we just need a role with `create` privileges in the database. The log table also makes it easy for both them and us to debug issues, as the stream of changes that we captured is right there in the database.

Re: We used Elixir's Observer to hunt down bottlenecks

#27

Post by cpursley »

_acco wrote: post #26
cpursley wrote: post #25

Sequin is really cool! Are y'all listening postgres WAL?

Thanks! We considered using Postgres' WAL but decided not to for the time being. Our solution now uses trigger functions. These trigger functions fire whenever a create/update/delete happens on a Sequin table. They insert a row into a log table. That log table is processed by our workers to send changes to the upstream API. The advantage of using trigger functions + a log table are all about ease of use and compatibi…

Very cool.

I'm using Elixir to listen to change events via https://github.com/cpursley/walex (which I basically ripped off from Supabase).

Re: We used Elixir's Observer to hunt down bottlenecks

#28

Post by JohnCurran »

mattbaker wrote: post #21
losvedir wrote: post #2

This is really cool. We use Elixir at work, but we mostly use it in a "traditional web app" (i.e. non-Elixir) way, of Docker containers deployed to independent AWS instances. So I'm always intrigued by some of the more BEAM-specific things that folks do, like using `observer` on a remote (production??) node here, or distributed Elixir where the nodes communicate with each other, or "hot" code updates. How do companie…

For us we have our app deployed to $N containers with a load balancer in front (pretty standard stuff I think?) In Erlang/Elixir you can actually override how instances of the BEAM find each other (instead of the standard EPMD daemon), so we have a module that does some DNS queries, finds the IPs of the other containers and says “hi, here’s your cluster, discovery done.” (Your setup may preclude all that, I know this…

Can you speak more to how you bypass EPMD and send the IPs of the containers to each other? That would be great for a problem we’re seeing where I work

Re: We used Elixir's Observer to hunt down bottlenecks

#29

Post by throwawaymaths »

JohnCurran wrote: post #28
mattbaker wrote: post #21

Earlier quoted context omitted.

For us we have our app deployed to $N containers with a load balancer in front (pretty standard stuff I think?) In Erlang/Elixir you can actually override how instances of the BEAM find each other (instead of the standard EPMD daemon), so we have a module that does some DNS queries, finds the IPs of the other containers and says “hi, here’s your cluster, discovery done.” (Your setup may preclude all that, I know this…

Can you speak more to how you bypass EPMD and send the IPs of the containers to each other? That would be great for a problem we’re seeing where I work

Same. I'm not clustered yet, but I plan on it before EOY and that would be amazing. I think route53 has some internal routing capabilities, but some of the setup looks scary, or am I just being silly?

Re: We used Elixir's Observer to hunt down bottlenecks

#30

Post by lvass »

davidw wrote: post #14

> Second, we passed one particularly large data structure from a manager to a pool of dedicated worker processes. This meant we were reincurring the memory cost of this data structure for each worker process. We couldn't eliminate the repetition, but reducing the data to its bare essentials before passing it down to the workers minimizes that cost. Hard to say without knowing much about the data in question, but my r…

Just confirmed it works.

iex>term = for _ iex>bin = :erlang.term_to_binary(term)

iex>for _ x = bin; :timer.sleep(1000000) end)

# Memory usage exploded in line below

iex>for _ x = term; :timer.sleep(1000000) end)

Post reply on HN