Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hotswapping Haskell

simonmar.github.io

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Hotswapping Haskell

#21

Post by Terribledactyl »

teraflop wrote: post #2

I don't want to knock the technical achievement here -- it's a cool hack -- but I'm really surprised that it was deemed to be the best choice for a production system. In the first place, "we can't compile our code on every change because it takes too long" is a really awful situation to be in. Are developers not building and testing their changes before deploying them? Can Facebook not afford a continuous integration…

I think the main benefit is the middle point. It sounds like they have programs with huge memory footprints and (I’m guessing) caches that take a while to warm up. This lets them avoid that. Fraud detection is probably time sensitive and slow responses aren’t acceptable.

Re: Hotswapping Haskell

#22

Post by cakoose »

cies wrote: post #7
jjnoakes wrote: post #6

Earlier quoted context omitted.

I agree. Fun read and cool hack, but it definitely feels like they are stretching to justify the more fun of the two options (spend time on this or spend time fixing the root cause).

Glad you guys both can make a better trade off than the engineers that actually have their hands on the problem. /s You're reading a blog post, you do not know all they have tried, nor the various intricacies they're dealing with.

Yeah, my initial reaction was "I can see how these design decisions might make sense, but the blog post is horrible."

These kinds of designs typically emerge over a long and windy history and, for someone who was part of that process, it's difficult to coherently describe the final state to an outsider. Good textbook authors have this skill. Most tech blog authors do not. (I think that part of the problem is that people don't respect just how difficult it actually is.)

My guess: restarting a large fleet of processes is a pain. The rollout will typically be throttled to avoid connection churn, among other things. For risky code changes, you probably want a slow rollout anyway, but if you're just tweaking abuse detection rules (almost just a config change), it's nice to have your changes take effect more quickly. Dynamic loading seems like one reasonable way to achieve that goal.

Tangent: people, please stop making analogies to mechanical engineering feats that are WAY more difficult than what you did [1]. People have been loading shared libraries forever; it's like adding an AUX port, not swapping out the engine. It's not even in the same league as Ksplice or as the JVM's dynamic loading/deoptimization.

[1] http://jensimmons.com/post/jan-4-2017/replacing-jet-engine-w...

Re: Hotswapping Haskell

#23

Post by andy_ppp »

rdtsc wrote: post #9

Hotswapping just like security is one of those things that is hard to bolt on later, unless it is built in deeply into the very core of the language / runtime. Erlang (and Elixir) define hotswapping very well. It is a standard way to upgrade code in production in some places. And even with it being well defined it is still very hard and there are enough corner cases to handle. But when used correctly, it is really ma…

Couldn't agree more, when the article said "Starting and tearing down millions of heavy processes a day would create undue churn on other infrastructure" I just thought, yes, I best you'd struggle to create an architecture so monolithic in Erlang or Elixir. Just one of the many benefits of course... add on the the number of process you can create on one machine while maintaining throughput...

Re: Hotswapping Haskell

#24

Post by jimbokun »

teraflop wrote: post #2

I don't want to knock the technical achievement here -- it's a cool hack -- but I'm really surprised that it was deemed to be the best choice for a production system. In the first place, "we can't compile our code on every change because it takes too long" is a really awful situation to be in. Are developers not building and testing their changes before deploying them? Can Facebook not afford a continuous integration…

"In the first place, "we can't compile our code on every change because it takes too long" is a really awful situation to be in."

Isn't this exactly the problem Go was invented to solve?

Re: Hotswapping Haskell

#25

Post by goialoq »

teraflop wrote: post #2

I don't want to knock the technical achievement here -- it's a cool hack -- but I'm really surprised that it was deemed to be the best choice for a production system. In the first place, "we can't compile our code on every change because it takes too long" is a really awful situation to be in. Are developers not building and testing their changes before deploying them? Can Facebook not afford a continuous integration…

JVM hotswapping is ages old, but it's usually used only in testing, not production.

Re: Hotswapping Haskell

#26

Post by JonCoens »

teraflop wrote: post #2

I don't want to knock the technical achievement here -- it's a cool hack -- but I'm really surprised that it was deemed to be the best choice for a production system. In the first place, "we can't compile our code on every change because it takes too long" is a really awful situation to be in. Are developers not building and testing their changes before deploying them? Can Facebook not afford a continuous integration…

I should have emphasized the speed of deployment being a first order concern more. We certainly can (and do) build our code for every change, but not at the speed that we want to be updating.

We use a monorepo for all of the benefits it has, and deploying fast business logic updates this way helps mitigate one of its downsides (particularly when you've maximally parallelized the build). I've found https://danluu.com/monorepo/ to give a quick overview of how chopping up the repo would have separate downsides.

The section about "Sticky Shared Objects" speaks directly to mutable state across code modifications, just with a Haskell-minded focus.

Re: Hotswapping Haskell

#27

Post by goialoq »

Terribledactyl wrote: post #21
teraflop wrote: post #2

I don't want to knock the technical achievement here -- it's a cool hack -- but I'm really surprised that it was deemed to be the best choice for a production system. In the first place, "we can't compile our code on every change because it takes too long" is a really awful situation to be in. Are developers not building and testing their changes before deploying them? Can Facebook not afford a continuous integration…

I think the main benefit is the middle point. It sounds like they have programs with huge memory footprints and (I’m guessing) caches that take a while to warm up. This lets them avoid that. Fraud detection is probably time sensitive and slow responses aren’t acceptable.

They could transfer the cache data from one (old) server instance to another (new) one.

Re: Hotswapping Haskell

#28

Post by hood_syntax »

forkerenok wrote: post #17
teraflop wrote: post #2

I don't want to knock the technical achievement here -- it's a cool hack -- but I'm really surprised that it was deemed to be the best choice for a production system. In the first place, "we can't compile our code on every change because it takes too long" is a really awful situation to be in. Are developers not building and testing their changes before deploying them? Can Facebook not afford a continuous integration…

I remember at Standard Chartered they have a Haskell monolith project of a few million LoC and they are relying on incremental building. [1] I wonder why that wasn't an option for facebook. [1] From podcast: http://www.haskellcast.com/episode/002-don-stewart-on-real-w...

I'm not 100% sure, but they (Standard Chartered) do use a custom compiler. That might explain the difference

Re: Hotswapping Haskell

#29

Post by elihu »

teraflop wrote: post #2

I don't want to knock the technical achievement here -- it's a cool hack -- but I'm really surprised that it was deemed to be the best choice for a production system. In the first place, "we can't compile our code on every change because it takes too long" is a really awful situation to be in. Are developers not building and testing their changes before deploying them? Can Facebook not afford a continuous integration…

It kind of sounds like they're running into some limitations of GHC: it tends to take a long time to compile stuff, and it tends to generate some very big binaries. For most applications, those aren't major problems but in their use case (hundreds of thousands of lines of code deployed to many servers) it is an issue so they're working around it. That allows them to keep working in the language they prefer and are productive in, which is great.

Improving GHC compile times and reducing the binary size would be better, but presumably a lot of work has already gone into those problems and if it were easy someone would have done it by now. As for myself, I really like using Haskell and I'm glad whenever I hear about it being used in industry.

Re: Hotswapping Haskell

#30

Post by kornish »

jimbokun wrote: post #24
teraflop wrote: post #2

I don't want to knock the technical achievement here -- it's a cool hack -- but I'm really surprised that it was deemed to be the best choice for a production system. In the first place, "we can't compile our code on every change because it takes too long" is a really awful situation to be in. Are developers not building and testing their changes before deploying them? Can Facebook not afford a continuous integration…

"In the first place, "we can't compile our code on every change because it takes too long" is a really awful situation to be in." Isn't this exactly the problem Go was invented to solve?

It was one of them. However, given the other writing/talks Facebook has put out about their usage of Haskell and Haxl, Go is probably not a good fit for their use case due to language expressivity concerns (not declarative enough, not enough type safety, not syntactically flexible enough for writing DSLs).
Post reply on HN