Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elixir saves Pinterest $2M a year in server costs

paraxial.io

Post reply on HN
Print view View original

331–340 of 481 posts

Re: Elixir saves Pinterest $2M a year in server costs

#331

Post by ch4s3 »

pessimizer wrote: post #295
ch4s3 wrote: post #284

Earlier quoted context omitted.

Not to just toss around anecdotes, but I once rewrote an email service in elixir for a company from a literal sketch on a piece of printer paper describing what their old system did. The new service ran on 1 server vs half a dozen and was both faster at crunching through their mail queue and used far fewer resources. Some tasks are embarrassingly parallelization and the BEAM excels at those tasks. Sure you might want…

Whatsapp took over the world running on Erlang/BEAM, with barely any servers and a few engineers. I honestly don't know what could be a better success story than that, but Discord has also done pretty well. The BEAM + Rust combination is looking scarily effective right now.

I'd be willing to accept the argument that Whatsapp happened to have assembled an uncommonly good team, but it is a signal.

Yeah the BEAM with some Rust NIFs is a great combo. I'd definitely consider it in the future for many types of problems and anything involving a HTTP or GraphQL interface.

Re: Elixir saves Pinterest $2M a year in server costs

#332

Post by cpursley »

sph wrote: post #64
threeseed wrote: post #17

Earlier quoted context omitted.

> When our notifications system was running on Java, it was on 30 c32.xl instances. When we switched over to Elixir, we could run on 15. Would be curious to know how they tried to optimise the Java stack. Because on every benchmark I've seen the JVM is faster in every which way than Elixir. Except for memory where often people will over-provision the JVM rather than look at where their code might be over-allocating o…

The advantages of Elixir are not performance-related. There is a lot of focus on raw performance on web-related services, when in reality most of their running time is spent waiting for IO. If there are two things the BEAM excels at, is IO and turning almost any problem into half a dozen processes that are scheduled and run in parallel, if not geographically-distributed, with 1/50th the effort of any other language.…

Which libraries are you using for the crawler or are you building from scratch? I’d love to hear more about your approach!

Re: Elixir saves Pinterest $2M a year in server costs

#333

Post by codesnik »

xk_id wrote: post #60
dazc wrote: post #53

Earlier quoted context omitted.

Google are returning results for image search without the overhead cost. This is the only logical reason I can think of why Pinterest still exists.

I was never sure where all the hate is coming from. Pinterest is actually one of my favourite apps, that I use regularly – and I use very few apps. I actually really love it and after years I still find new uses for it. As far as I can tell, they have a pretty loyal user base.

It'd be fine if it wouldn't spam google images results, and resulting images are often completely cut out of context, or origin link is not working.

Re: Elixir saves Pinterest $2M a year in server costs

#334

Post by xk_id »

cmrdporcupine wrote: post #325
xk_id wrote: post #321

Earlier quoted context omitted.

this is not true? every single pin has a big button underneath which takes you to the original webpage where it was saved from. and if it was manually uploaded by the user and lacks this kind of metadata, you can scan it visually and it shows you all the other uploads on pinterest which match it approximately, and usually one of the matches has better metadata.

this has been inconsistent over the years. maybe they 'fixed' it but it wasn't not the case before. they were extremely parasitical. you'd click the button and it would just take you to more pins

that only happens when the image was manually uploaded by a user (i.e from gallery). there's not much they can do about it. in that case you can scan it to find visual matches and hopefully one of them has metadata about the original source (but it's not perfect and you have to scroll a bit).

Re: Elixir saves Pinterest $2M a year in server costs

#335

Post by pdimitar »

SanderNL wrote: post #328
pdimitar wrote: post #225

Earlier quoted context omitted.

Why is it ridiculous? I migrated Rails apps to Elixir before, we reduced from 15 servers to 3, and 1 was basically "if crap hits the fan", we could have gotten away with 2 easily. It's worrying that a supposedly high-quality forum like HN receives comments with no substance. If you have an actual counter-argument, let's discuss. If not, well, not an interesting exchange.

You keep insisting HN is not providing comments up to your standards. Let’s not go there OK? I just disagree with your analysis. I think it is too simplistic to state GCs cause this and Elixir somehow magically causes 95% efficiency boosts. All I’m saying is that if you can drop 1330 servers just like that, there might be something more going on than Python’s slowness. This is from experience. I have seen people crea…

Sure, if you want to expand the discussion to "everyone can make every tech stack act badly" then you might have had an argument. I don't find that argument compelling however -- it's borderline meaningless.

Also nobody used the word "magically" before you did. Note that.

What's your argument exactly? That Elixir is overrated? Or something else?

Furthermore, I am not insisting on my standards of the quality of comments. I am under the impression that's the expected quality of comments on HN at large.

Re: Elixir saves Pinterest $2M a year in server costs

#336

Post by xk_id »

codesnik wrote: post #333
xk_id wrote: post #60

Earlier quoted context omitted.

I was never sure where all the hate is coming from. Pinterest is actually one of my favourite apps, that I use regularly – and I use very few apps. I actually really love it and after years I still find new uses for it. As far as I can tell, they have a pretty loyal user base.

It'd be fine if it wouldn't spam google images results, and resulting images are often completely cut out of context, or origin link is not working.

if the origin link isn't working, that means it was taken offline by the original source. in that case, imo it's a good thing that at least the image was archived on their servers. especially since each one has a comments section where someone might post information about it.

Re: Elixir saves Pinterest $2M a year in server costs

#337

Post by Volundr »

ozr wrote: post #324
masklinn wrote: post #129

Earlier quoted context omitted.

It’s not even remotely similar. Node’s cluster is just bog-standard OS subprocesses running their own event loop. To spread work over multiple cores with the cluster (or even worker-threads) modules you have to do so explicitely and manually. It’s essentially the same model you get with pthreads, or java, or python. BEAM is a completely different model, the “processes” are internal tasks, which the runtime will sched…

I'm unfamiliar with BEAM. How does this compared to goroutines? Obviously they won't migrate between machines, but concurrency feels very easy and ergonomic in Go.

Disclaimer: I'm not very experienced in Go.

Go routines were pretty directly inspired by Erlang processes, so in terms of primitives I'd say they are very similar, aside from go lacking the distributed features you already mentioned.

Where Erlang/Elixir add value beyond go routines is what OTP (kind of the standard library) provides on top. Pre-built abstractions like GenServer for long running processes, Genstage for producer/consumer pipelines and supervision trees for making sure your processes haven't crashed and restarting them and their dependents if they have.

Re: Elixir saves Pinterest $2M a year in server costs

#338

Post by lifeisstillgood »

0xpgm wrote: post #34
lifeisstillgood wrote: post #27

I argue every project has to go through three phases - no idea what's wrong - No idea what's right - We know what was right, but the world has changed now.

Could you elaborate? I'm finding it hard to connect this to the topic.

Yeah. There is a phase of discovery - we simply don't know what (some combination of) the project goals, the user needs, the market, the business model and the architectures possible. By doing something we find out. often our guesses are good. Often the company politics mean we build some atrocious monster. The point here is we are elimating what's wrong (the choices are /right/ not (provably) wrong/ (provably) wrong/ and not even wrong (as a famous physicist would mark his students)

Once this is done (maybe proof of concept, maybe research maybe years of fighting the politics) we can build out something that's not wrong.

If we time it well and get it right it's a unicorn. It's Facebook. if we time it right but get it merely not wrong, it's friendster or myspace.

And of course by the time we have learnt all that, mobile hits and we have to relearn

Re: Elixir saves Pinterest $2M a year in server costs

#339

Post by SanderNL »

pdimitar wrote: post #335
SanderNL wrote: post #328

Earlier quoted context omitted.

You keep insisting HN is not providing comments up to your standards. Let’s not go there OK? I just disagree with your analysis. I think it is too simplistic to state GCs cause this and Elixir somehow magically causes 95% efficiency boosts. All I’m saying is that if you can drop 1330 servers just like that, there might be something more going on than Python’s slowness. This is from experience. I have seen people crea…

Sure, if you want to expand the discussion to "everyone can make every tech stack act badly" then you might have had an argument. I don't find that argument compelling however -- it's borderline meaningless. Also nobody used the word "magically" before you did. Note that. What's your argument exactly? That Elixir is overrated? Or something else? Furthermore, I am not insisting on my standards of the quality of commen…

Good sir, I am not looking for trouble. I concur, my argument is nebulous at best.

To be clear, I think Elixir is marvelous. This post spiked my interest in it actually. Sorry if I come across ignorant. That’s because I am.

Re: Elixir saves Pinterest $2M a year in server costs

#340

Post by pdimitar »

SanderNL wrote: post #339
pdimitar wrote: post #335

Earlier quoted context omitted.

Sure, if you want to expand the discussion to "everyone can make every tech stack act badly" then you might have had an argument. I don't find that argument compelling however -- it's borderline meaningless. Also nobody used the word "magically" before you did. Note that. What's your argument exactly? That Elixir is overrated? Or something else? Furthermore, I am not insisting on my standards of the quality of commen…

Good sir, I am not looking for trouble. I concur, my argument is nebulous at best. To be clear, I think Elixir is marvelous. This post spiked my interest in it actually. Sorry if I come across ignorant. That’s because I am.

Thank you for recognizing that we were going nowhere. Apologies if my tone was sharp.

I am not evangelizing tech -- I am a polyglot and I use what I find is best suited for a job, and Elixir happens to cover quite a lot of ground. That's all really. I also use Rust and Golang quite a bit.

I simply get ticked off when people start demeaning something without seriously working with it or even reading a bit beforehand. Sorry if I mistakenly put you in that group.

Post reply on HN