Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elixir saves Pinterest $2M a year in server costs

paraxial.io

Post reply on HN
Print view View original

321–330 of 481 posts

Re: Elixir saves Pinterest $2M a year in server costs

#321

Post by xk_id »

cmrdporcupine wrote: post #97
xk_id wrote: post #60

Earlier quoted context omitted.

I was never sure where all the hate is coming from. Pinterest is actually one of my favourite apps, that I use regularly – and I use very few apps. I actually really love it and after years I still find new uses for it. As far as I can tell, they have a pretty loyal user base.

The hate comes from them hiding where the original content is from. They hijack search results and then pull you into an endless circle of harvested content, making it difficult to find the link to the original source of the material.

this is not true? every single pin has a big button underneath which takes you to the original webpage where it was saved from. and if it was manually uploaded by the user and lacks this kind of metadata, you can scan it visually and it shows you all the other uploads on pinterest which match it approximately, and usually one of the matches has better metadata.

Re: Elixir saves Pinterest $2M a year in server costs

#322

Post by steveklabnik »

pessimizer wrote: post #317
kramerger wrote: post #131

Earlier quoted context omitted.

Elixir is a functional language for the Erlang VM. Most C++/Java/Rust/... programmers wpuld have hard time understanding it. But if you are in a bubble where everyone uses Haskell and talks about Monads, then OPs statement may be valid

Rust screams Erlang influence, but maybe it's just me. IIRC, in the beginning they even tried to put in a green threading VM for tail calls, which seems insane for something that you want to be a systems language.

Early Rust was pretty expressly influenced by Erlang.

The green threading runtime was removed in 2014, shortly before the 1.0.0 release in May 2015.

The reason for the waning influence was exactly that: couldn’t be reconciled with systems use cases effectively.

Re: Elixir saves Pinterest $2M a year in server costs

#323

Post by no_wizard »

javanonymous wrote: post #307
neeleshs wrote: post #253

Earlier quoted context omitted.

Yep. We got a 10x improvement on throughput for our backend runtime, which was in Java, by moving to a better architecture for performance hotspots.. using Java again. In a rewrite with a different design/architecture, that new design typically accounts for most gains, rather than language. A language may make some parts of that rewrite simpler.

I've gotten 100x improvement with no code change by just adding an index in the database table. An inexperienced developer might have blamed the database and insisted on moving to NoSQL because of "web scale". If they got the chance to rewrite it, they could have pointed to the performance increase as a proof that they were right.

They really should teach benchmark training more widely in the industry. Even though I'm readily here to sing the praises of Elixir when warranted, nothing beats actually profiling end to end the workload(s) in question that need to be improved. Sometimes, it really is optimizing the database that matters most, like adding an index (or using window functions or stored procedures, as in many cases I've had in the past)

Re: Elixir saves Pinterest $2M a year in server costs

#324

Post by ozr »

masklinn wrote: post #129
_m8fo wrote: post #100

Earlier quoted context omitted.

[flagged]

It’s not even remotely similar. Node’s cluster is just bog-standard OS subprocesses running their own event loop. To spread work over multiple cores with the cluster (or even worker-threads) modules you have to do so explicitely and manually. It’s essentially the same model you get with pthreads, or java, or python. BEAM is a completely different model, the “processes” are internal tasks, which the runtime will sched…

I'm unfamiliar with BEAM. How does this compared to goroutines? Obviously they won't migrate between machines, but concurrency feels very easy and ergonomic in Go.

Re: Elixir saves Pinterest $2M a year in server costs

#325

Post by cmrdporcupine »

xk_id wrote: post #321
cmrdporcupine wrote: post #97

Earlier quoted context omitted.

The hate comes from them hiding where the original content is from. They hijack search results and then pull you into an endless circle of harvested content, making it difficult to find the link to the original source of the material.

this is not true? every single pin has a big button underneath which takes you to the original webpage where it was saved from. and if it was manually uploaded by the user and lacks this kind of metadata, you can scan it visually and it shows you all the other uploads on pinterest which match it approximately, and usually one of the matches has better metadata.

this has been inconsistent over the years. maybe they 'fixed' it but it wasn't not the case before. they were extremely parasitical. you'd click the button and it would just take you to more pins

Re: Elixir saves Pinterest $2M a year in server costs

#326

Post by pdimitar »

fud101 wrote: post #318
pdimitar wrote: post #228

Earlier quoted context omitted.

I can see how you would think that, yes. In practice I haven't noticed it except in super rare cases where processes (actors) hold on to huge binaries / strings -- which is one of the weak points of BEAM's GC.

I've been bit by this. In reality you need to know your shit when it comes to tuning the Beam and GC to achieve decent performance under load without triggering OOM.

Not denying it, yeah, it's a blind spot and it's one of the more advanced topics when you start having actually loaded nodes in production.

Re: Elixir saves Pinterest $2M a year in server costs

#327

Post by wahnfrieden »

sanitycheck wrote: post #16

I wonder how much of their traffic is angry people who ended up there by mistake. They could save a lot more than $2M if they just set robots.txt to disallow everything.

Why on earth would they do that? It’s intentional and drives growth for them

Re: Elixir saves Pinterest $2M a year in server costs

#328

Post by SanderNL »

pdimitar wrote: post #225
SanderNL wrote: post #205

Earlier quoted context omitted.

That does not in any way explain a drop of 95%, which IMO is ridiculous and points to other issues. The system they created now is totally different from the one they had. It’s more efficient by an insane margin. Choice of language seems like it would have trouble breaking the top 5 major reasons.

Why is it ridiculous? I migrated Rails apps to Elixir before, we reduced from 15 servers to 3, and 1 was basically "if crap hits the fan", we could have gotten away with 2 easily. It's worrying that a supposedly high-quality forum like HN receives comments with no substance. If you have an actual counter-argument, let's discuss. If not, well, not an interesting exchange.

You keep insisting HN is not providing comments up to your standards. Let’s not go there OK? I just disagree with your analysis. I think it is too simplistic to state GCs cause this and Elixir somehow magically causes 95% efficiency boosts.

All I’m saying is that if you can drop 1330 servers just like that, there might be something more going on than Python’s slowness.

This is from experience. I have seen people create slow and fast systems with just about any tech. I can make Elixir crawl, I can assure you of that.

I have seen Python apps use 10 servers and reduced it to one as well. Same tech, just a more efficient mindset. It’s IMO a bit too simplistic to say systems with GCs fall over when under load.

Re: Elixir saves Pinterest $2M a year in server costs

#329

Post by xk_id »

caskstrength wrote: post #107
xk_id wrote: post #60

Earlier quoted context omitted.

I was never sure where all the hate is coming from. Pinterest is actually one of my favourite apps, that I use regularly – and I use very few apps. I actually really love it and after years I still find new uses for it. As far as I can tell, they have a pretty loyal user base.

> I was never sure where all the hate is coming from. Pinterest is actually one of my favourite apps, that I use regularly – and I use very few apps. I'm not looking for an _app_ when searching in google, I'm looking for a link to a _web site_ with original image.

google search is worthless now, not just for images. i only ever use it with the `site:` setting or the research subsomain (these are rhe only ones that seem to have retained a functional version of google's algorithm). i use pinterest itself for general image discoverability and it works really well.

Re: Elixir saves Pinterest $2M a year in server costs

#330

Post by anonymoushn »

astrobe_ wrote: post #121
TacticalCoder wrote: post #23

Earlier quoted context omitted.

They rewrote, which is known to help too. Going from 30 to 15 instances is not bad but it's very likely that a Java-to-Java rewrite would have helped go down too. The big one however is going from 200 Python servers to 4 Erlang ones: a 50x reduction is quite something and a Python-to-Python rewrite would not have allowed to achieved a 50x gain: > All this is possible because Elixir, and the Erlang platform underneath…

> The big one however is going from 200 Python servers to 4 Erlang ones If your rewrite or refactor gives you a 10X in performance, that's not an optimization, but a bugfix. Unless you are a researcher who have just found a revolutionary algorithm.

Why? Most software is at least 10x slower than necessary for reasons that don't require inventing new algorithms.
Post reply on HN