Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elixir saves Pinterest $2M a year in server costs

paraxial.io

Post reply on HN
Print view View original

291–300 of 481 posts

Re: Elixir saves Pinterest $2M a year in server costs

#291

Post by slotrans »

sph wrote: post #64
threeseed wrote: post #17

Earlier quoted context omitted.

> When our notifications system was running on Java, it was on 30 c32.xl instances. When we switched over to Elixir, we could run on 15. Would be curious to know how they tried to optimise the Java stack. Because on every benchmark I've seen the JVM is faster in every which way than Elixir. Except for memory where often people will over-provision the JVM rather than look at where their code might be over-allocating o…

The advantages of Elixir are not performance-related. There is a lot of focus on raw performance on web-related services, when in reality most of their running time is spent waiting for IO. If there are two things the BEAM excels at, is IO and turning almost any problem into half a dozen processes that are scheduled and run in parallel, if not geographically-distributed, with 1/50th the effort of any other language.…

> Handling requests over separate threads, like 99% of languages do, still isn't enough if all the business logic runs on the same thread.

When would this EVER be the case? I'm struggling to imagine a scenario where logic doesn't/can't run in the worker thread.

Re: Elixir saves Pinterest $2M a year in server costs

#292

Post by moomoo11 »

I use Go and it gives me a good sweet spot between cost and perf.

I’m nowhere close to performance requirements of Pinterest. But my auth service runs on a single 1vcpu, 512mb ram, and 10gb ssd. I use leveldb and swap on the 10gb. I’ve benchmarked it to handle 8-9k rps while delivering 150ms max response time. Not bad for a few bucks a month.

Re: Elixir saves Pinterest $2M a year in server costs

#293

Post by slotrans »

dimgl wrote: post #290

Why is everyone so angry and skeptical in these comments? I agree the article is operating on old information, but what's going on here?

It's a vendor post, written to sell you something. Accordingly, it lacks any nuance or perspective and makes claims it doesn't even attempt to back up.

Re: Elixir saves Pinterest $2M a year in server costs

#294

Post by princevegeta89 »

coldtea wrote: post #3

By the title alone, this means nothing. How much would it cost otherwise? What is the percentage savings? In TFA, it gets better though: "Steve: That’s pretty easy. When I started on the spam team, we had close to 1,400 servers running. When we converted several parts to Elixir, we reduced that by around 95%. One of the systems that ran on 200 Python servers now runs on four Elixir servers (it can actually run on two…

The problem with Elixir is that it is such a foreign language to most of the junior developers and a radical shift to dig into coming from object-oriented and other higher-level languages. This is worsened by the fact that there are not many jobs for Elixir in addition to Development Tooling, IDE Support.

To someone who starts their job on an Elixir codebase, it is just not a smooth onboarding at all. While the performance aspect is unparalleled compared to most of the popular scripting languages in the last decade, the price to pay to settle into Elixir seems huge to me.

Re: Elixir saves Pinterest $2M a year in server costs

#295

Post by pessimizer »

ch4s3 wrote: post #284
javanonymous wrote: post #267

Earlier quoted context omitted.

They wrote it from scratch with the benefit of all the knowledge they had gathered after running the old system for years. A 2X improvement would not be surprising to me, even if they had rewritten it in the same language. . According to others in this discussion they also made architecture changes (DB, Kafka etc.). Do we know if that improved the performance? There is no objective way we can tell if Elixir had any p…

Not to just toss around anecdotes, but I once rewrote an email service in elixir for a company from a literal sketch on a piece of printer paper describing what their old system did. The new service ran on 1 server vs half a dozen and was both faster at crunching through their mail queue and used far fewer resources. Some tasks are embarrassingly parallelization and the BEAM excels at those tasks. Sure you might want…

Whatsapp took over the world running on Erlang/BEAM, with barely any servers and a few engineers. I honestly don't know what could be a better success story than that, but Discord has also done pretty well. The BEAM + Rust combination is looking scarily effective right now.

Re: Elixir saves Pinterest $2M a year in server costs

#296

Post by panick21_ »

threeseed wrote: post #17
coldtea wrote: post #3

By the title alone, this means nothing. How much would it cost otherwise? What is the percentage savings? In TFA, it gets better though: "Steve: That’s pretty easy. When I started on the spam team, we had close to 1,400 servers running. When we converted several parts to Elixir, we reduced that by around 95%. One of the systems that ran on 200 Python servers now runs on four Elixir servers (it can actually run on two…

> When our notifications system was running on Java, it was on 30 c32.xl instances. When we switched over to Elixir, we could run on 15. Would be curious to know how they tried to optimise the Java stack. Because on every benchmark I've seen the JVM is faster in every which way than Elixir. Except for memory where often people will over-provision the JVM rather than look at where their code might be over-allocating o…

And of course you can just go and buy a much faster JVM from Azul rather then rewrite everything.

Re: Elixir saves Pinterest $2M a year in server costs

#297

Post by fidotron »

throwawaymaths wrote: post #256
fidotron wrote: post #210

Earlier quoted context omitted.

The beam just garbage collects each actor separately, and it so happens that much of the time your actor has finished before a gc happened so you never see the cleanup. The beam also has a spectacular failure mode: OOM whenever messages come in at a higher rate than they are processed. The lack of backpressure mechanisms mean a huge amount of beam language developers spend way too much time recreating their own way f…

I think that's developers using GenServer.cast when they should be using call. Call gives you back pressure mechanism.

TBH at the load we had we got substantial savings by eventually replacing usage of gen_server as well, though that probably isn’t a good idea much of the time.

The OOMs were largely being caused by calls to and from other services (i.e. kafka) so the answer proved to be in controlling the rate at which things come in and out at the very edge.

From what I saw I got the impression the Beam devs assumed memory and CPU usage go together so a system that is under load memory wise would also be CPU wise, but this isn’t the case if your fan out and gather involves holding large* values on which the response is based, even if for tiny amounts of time.

EDIT: *large meaning "surprisingly small" if you're coming from other universes.

Re: Elixir saves Pinterest $2M a year in server costs

#298

Post by corytheboyd »

dimgl wrote: post #290

Why is everyone so angry and skeptical in these comments? I agree the article is operating on old information, but what's going on here?

It’s hacker news, hooray! Personally I don’t mind the skepticism. I glean insight from it that I would not have on my own. Rarely it’s downright malevolent, but the tone is usually about the same.

I agree with the skeptics about this article, it’s a flashy headline that can’t be accurate, per the points being raised.

Re: Elixir saves Pinterest $2M a year in server costs

#299

Post by panick21_ »

zaphar wrote: post #238
Thaxll wrote: post #199

Earlier quoted context omitted.

I wrote the same kind of tool in both Elixir and Go, the performance difference was huge between the two, Elixir / Erlang is pretty slow overall.

The go runtime has similar capabilities as the BEAM runtime when it comes to concurrent workloads. Go has the benefit of being a typesafe compiled language which gives it speed benefits. But using either one of them instead of Java is probably going to be a huge win for most teams on concurrent workloads.

And you can achieve better performance with JVM as well. Its just about how you write that Java (or use Clojure) or something.

Re: Elixir saves Pinterest $2M a year in server costs

#300

Post by jongjong »

danjc wrote: post #108

These extreme claims strain credibility. "We built this thing really shabbily and look how much we saved using a different stack. Oh and we also architected it properly second time around."

Agreed. I get triggered whenever I see a headline like "We moved from tech A to tech B and reduced/increased C by X%"

No, that's not what happened! What happened is you put more thought into it the second time.

I hate this management BS; if the project fails, blame the engineers, if the project succeeds, praise the tech... They just do everything to turn engineers into commodities and most of us just play along.

Post reply on HN