Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elixir saves Pinterest $2M a year in server costs

paraxial.io

Post reply on HN
Print view View original

311–320 of 481 posts

Re: Elixir saves Pinterest $2M a year in server costs

#311

Post by eropple »

Name_Chawps wrote: post #310
jeswin wrote: post #151

Earlier quoted context omitted.

> ... with 1/50th the effort of any other language. To junior developers reading this, such statements are never true.

I'm a junior Elixir dev. I write concurrent code without much extra effort compared to non-concurrent code. Elixir makes it easy.

You can do that easily in modern Java--even for older JVMs, tools like Netty and later Vertx have been around forever. Or in Node, even more easily.

Elixir/BEAM do have some benefits that are worth considering for many projects. But they absolutely are not special in this regard, and that's the junior-developer trap about which the person to whom you replied was referring.

Re: Elixir saves Pinterest $2M a year in server costs

#312

Post by no_wizard »

princevegeta89 wrote: post #308
no_wizard wrote: post #303

Earlier quoted context omitted.

sounds like a feature not a bug. Poor onboarding is a cultural problem in my experience, not really a technology one. When you get junior (or even non junior) developers onboarded in a new language, you have a unique opportunity to break them of bad habits and expand horizons. Yes, there is a cost to it as it extends in the short term the time it takes to get developers ramped up, however the long tail payoff is huge

>>> the long tail payoff is huge In what terms, exactly?

assuming that the long tail is well executed

Anything the business can control for: architectural designs, server costs, approaches to building out features / services for the business etc.

When you can mold someone's experience via a new language to model a domain, they become very efficient to it, when they have no prior notions to fall back on.

How many times have developers gone down the wrong path because of X did it this way? type thinking. When you can sufficiently remove that so all that is left to think about is the problem space, you do make more gains around that problem space.

My thesis from (albeit anecdotal) experience, is that when you have developers working in a new paradigm (often, this corresponds with a new language) you have better chances at establishing these things than having to consistently try and override a developers prior notions about how something should work / look.

The trade off is higher ramp times and slower on-boarding, of course. In the short term, it can be more costly.

Re: Elixir saves Pinterest $2M a year in server costs

#313

Post by pessimizer »

pclmulqdq wrote: post #184
andrewstuart wrote: post #155

200 servers running Python. I bet it was an architecture problem more than a language problem. I’d be expecting to leave most of that in place and fix the hotspots that are dragging performance down. Maybe some compiled code in certain places. Almost certainly a detailed analysis would have yielded 20 things that could be tuned. Posts like this are almost something to be ashamed of “we rewrote an entire subsystem bec…

And they got a 50x reduction in server cost - 200 servers down to 4. I have never heard of a performance optimization project with such substantial gains without a complete rewrite.

It's important to remember the BEAM is a monster. I'll believe almost anything after I've seen big projects run on such small resources. I've been busy believing anything about BEAM since Whatsapp got so big on nothing, never mind Discord.

Re: Elixir saves Pinterest $2M a year in server costs

#315

Post by no_wizard »

panick21_ wrote: post #296
threeseed wrote: post #17

Earlier quoted context omitted.

> When our notifications system was running on Java, it was on 30 c32.xl instances. When we switched over to Elixir, we could run on 15. Would be curious to know how they tried to optimise the Java stack. Because on every benchmark I've seen the JVM is faster in every which way than Elixir. Except for memory where often people will over-provision the JVM rather than look at where their code might be over-allocating o…

And of course you can just go and buy a much faster JVM from Azul rather then rewrite everything.

then I have to pay Azul for the faster JVM and I may still not sufficiently cut my server costs, or the cost of the JVM paces the reduction I have in server costs.

2 million a year is several developers compensation saved every year. It also opens the door to more savings down the road, potentially, as existing workloads may discover they can use the same approaches to reduce cost / overhead.

Re: Elixir saves Pinterest $2M a year in server costs

#316

Post by pdimitar »

panick21_ wrote: post #306
pdimitar wrote: post #196

Earlier quoted context omitted.

There is a reason: the BEAM is almost not prone to huge GC pauses. Bigger load results in every actor responding very slightly slower. Nothing else. Many other systems don't have this property. They fall down under pressure. Gosh, a huge chunk of HN is always so dismissive. At least read up a bit beforehand, man. The criticisms should be informed and benefit the readers, not only express a generic skepticism.

If and that's a very, very, very big if, the current open GC are really leading to to much pauses. Then you can go to Azul and buy a better VM and GC, further improving the performance compared to BEAM.

Yep, agreed, I am just listing possibilities. More often than not a performance loss in Erlang/Elixir is caused by GC pauses but you can do a lot to reduce or outright eliminate those.

Re: Elixir saves Pinterest $2M a year in server costs

#317

Post by pessimizer »

kramerger wrote: post #131
capableweb wrote: post #120

Earlier quoted context omitted.

You seem to be wrong, at least based on the quote you chose to use here, as Steve doesn't even mention anything related to functional programming?

Elixir is a functional language for the Erlang VM. Most C++/Java/Rust/... programmers wpuld have hard time understanding it. But if you are in a bubble where everyone uses Haskell and talks about Monads, then OPs statement may be valid

Rust screams Erlang influence, but maybe it's just me. IIRC, in the beginning they even tried to put in a green threading VM for tail calls, which seems insane for something that you want to be a systems language.

Re: Elixir saves Pinterest $2M a year in server costs

#318

Post by fud101 »

pdimitar wrote: post #228
fidotron wrote: post #210

Earlier quoted context omitted.

The beam just garbage collects each actor separately, and it so happens that much of the time your actor has finished before a gc happened so you never see the cleanup. The beam also has a spectacular failure mode: OOM whenever messages come in at a higher rate than they are processed. The lack of backpressure mechanisms mean a huge amount of beam language developers spend way too much time recreating their own way f…

I can see how you would think that, yes. In practice I haven't noticed it except in super rare cases where processes (actors) hold on to huge binaries / strings -- which is one of the weak points of BEAM's GC.

I've been bit by this. In reality you need to know your shit when it comes to tuning the Beam and GC to achieve decent performance under load without triggering OOM.

Re: Elixir saves Pinterest $2M a year in server costs

#319

Post by Volundr »

eropple wrote: post #311
Name_Chawps wrote: post #310

Earlier quoted context omitted.

I'm a junior Elixir dev. I write concurrent code without much extra effort compared to non-concurrent code. Elixir makes it easy.

You can do that easily in modern Java--even for older JVMs, tools like Netty and later Vertx have been around forever. Or in Node, even more easily. Elixir/BEAM do have some benefits that are worth considering for many projects. But they absolutely are not special in this regard, and that's the junior-developer trap about which the person to whom you replied was referring.

Java, kind of with Akka or similar, although even with that one always be aware of blocking. Loom should help. Node: not really unless something dramatic has changed. Using 32 cores is going to require 32 separate node (OS level) processes, and your on your own for providing communication between them, plus callbacks aren't near as intuitive as the BEAM process model (think green threads)

Re: Elixir saves Pinterest $2M a year in server costs

#320

Post by adamgamble »

"The combined effect of better architecture and Elixir saved Pinterest..." I appreciate that they also call out better architecture here. People often read these kinds of headlines and think "Oh Elixir is better than python" or whatever.

I remember when twitter had fail whales constantly and they rewrote it from ruby to java I think. At the time everyone assumed it was all ruby's fault, and that might be partly true. But it's also true that the engineers who rebuilt it understood the problem much better now, and knew the major pain points. They also completely changed the architecture to be better suited to the problem. I submit that a lot of rewrites could happen in the same language and still have major gains.

All that being said Elixir is great, and particularly well suited to these kinds of problems.

Post reply on HN