Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How And Why We Switched from Erlang to Python (2011)

engineering.mixpanel.com

Post reply on HN
Print view View original

131–140 of 150 posts

Re: How And Why We Switched from Erlang to Python (2011)

#131

Post by collyw »

ummonk wrote: post #118
bradleyjg wrote: post #54

Earlier quoted context omitted.

In general language diversity at a company is a big negative. Obviously sometimes there are even bigger negatives that mandate overriding that rule--you aren't going to use Java on the front end or javascript on the backend, that would be madness--but you never want two languages in the same niche. That's a cost with no benefit. If you have some developer that wants to write something in a new (for your company) lang…

Why would it be madness to us javascript on the backend?

Because there are far better languages to choose from.

Re: How And Why We Switched from Erlang to Python (2011)

#132

Post by nathan_long »

bradleyjg wrote: post #54
lightcatcher wrote: post #27

I was an intern at Mixpanel at the time (but not the author of this article or involved in the discussed rewrite). For context, the company was then My recollection of events (of 8 years ago): The Erlang endpoint was written a year or two earlier as someone's first Erlang project, in a tiny org with no Erlang experience. The endpoint worked well enough and people moved onto other projects in the mostly Python codebas…

In general language diversity at a company is a big negative. Obviously sometimes there are even bigger negatives that mandate overriding that rule--you aren't going to use Java on the front end or javascript on the backend, that would be madness--but you never want two languages in the same niche. That's a cost with no benefit. If you have some developer that wants to write something in a new (for your company) lang…

> In general language diversity at a company is a big negative

I agree. It really increases the difficulty of understanding the whole system. But it's a negative that has to be considered alongside potential benefits.

> If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy.

I get what you're saying; resume-driven development can ruin your systems. But at some point between "we wrote this new banking system in COBOL" and "every developer who knows COBOL and could maintain our old system is now dead", there needs to be an accepted way to experiment.

Re: How And Why We Switched from Erlang to Python (2011)

#133

Post by UweSchmidt »

cellularmitosis wrote: post #121
andy_ppp wrote: post #109

Earlier quoted context omitted.

The problem here is not understanding your database not about making a system in an unsupported and probably undeployable thing to prove how clever you are...

> "to prove how clever you are" I see this a lot and I just don't get it. Is a mechanic who uses air tools trying to "prove how clever he is"? Knowing more than one tool allows you to choose the right tool for the job. Assuming arrogance and a motive of "proving cleverness" sounds like a great way to pick a fight.

There are no parallels in the real world to how programmers chose and use languages, frameworks and tools.

Re: How And Why We Switched from Erlang to Python (2011)

#134

Post by cellularmitosis »

collyw wrote: post #128
cellularmitosis wrote: post #105

Earlier quoted context omitted.

Some dev: "Hey, I noticed one of our more complex API endpoints had some pretty bad response latency, so I dug into it a bit. It turns out our PHP/Symfony/Doctrine stack was generating over a hundred SQL queries, all of which were being performed sequentially because we don't have a good per-request concurrency story in this tech stack. So I wrote an alternate implementation in Clojure which has a great concurrency s…

Rather than making thousands of requests in parallel, wouldn't reducing the number of queries be the more obvious choice.

Both would be ideal.

Re: How And Why We Switched from Erlang to Python (2011)

#135

Post by cellularmitosis »

delian66 wrote: post #119
cellularmitosis wrote: post #105

Earlier quoted context omitted.

Some dev: "Hey, I noticed one of our more complex API endpoints had some pretty bad response latency, so I dug into it a bit. It turns out our PHP/Symfony/Doctrine stack was generating over a hundred SQL queries, all of which were being performed sequentially because we don't have a good per-request concurrency story in this tech stack. So I wrote an alternate implementation in Clojure which has a great concurrency s…

Another developer: "...our stack was generating over a hundred SQL queries, all of which were being performed sequentially because we don't have a good per-request concurrency story in this tech stack. I just bypassed the ORM and wrote a slightly more complex SQL query, so that there is no more N+1 query problem in the backend. The latency of this end point was cut by several orders of magnitude, and our database is…

From a technical perspective, bypassing the ORM is a great solution (and if you can get the number of queries down to one, there’s no longer any need for concurrency).

Unfortunately, politically, the kind of shop which fires a dev for choosing another platform is unlikely to approve a PR which bypasses the ORM (maybe also a fireable offense!)

Re: How And Why We Switched from Erlang to Python (2011)

#136

Post by theamk »

cellularmitosis wrote: post #124
theamk wrote: post #108

Earlier quoted context omitted.

If there are no other people who can maintain that stack, maybe (firing is too extreme, I’d just ban the Clojure code in the production) Writing initial version of the code is only a small part of it. Maintaining and enhancing it is much harder. What is the company going to do if the new endpoint went down and the author is on vacation?

> just ban the Clojure code in the production A dev ran up against the limitations of one platform and solved the problem by choosing a platform which didn't have those limitations. And management responded by banning the new platform. How long do you think that dev is going to stick around in an environment like that? Ultimately this comes down to what sort of environment you want to foster. One approach leads to an…

I am not sure why you think it is "management responded" -- it is all about other people on the team.

Right now, I am responsible for maintaining one of our internal services. If someone tries to rewrite part of it in Clojure without extensive internal discussion first, I'll do all I can to stop it (and I am not "management" at all). And most people on the team will agree with me.

If the original dev would not want to stick around after that, good for them. If they do not care about teammates at all, they would likely be happier working alone anyway.

After all, single-handedly moving to a new technology and then forcing everyone else on the team to support it is impolite and annoying. Clojure is nice, but I don't want to be forced to learn it because the server went down on my shift.

(this is different if the service does not need to be up all the time, or the company is very small. But if you are talking about API endpoint being overloaded, the chances are, neither of this is true)

Re: How And Why We Switched from Erlang to Python (2011)

#137

Post by Izkata »

delian66 wrote: post #119
cellularmitosis wrote: post #105

Earlier quoted context omitted.

Some dev: "Hey, I noticed one of our more complex API endpoints had some pretty bad response latency, so I dug into it a bit. It turns out our PHP/Symfony/Doctrine stack was generating over a hundred SQL queries, all of which were being performed sequentially because we don't have a good per-request concurrency story in this tech stack. So I wrote an alternate implementation in Clojure which has a great concurrency s…

Another developer: "...our stack was generating over a hundred SQL queries, all of which were being performed sequentially because we don't have a good per-request concurrency story in this tech stack. I just bypassed the ORM and wrote a slightly more complex SQL query, so that there is no more N+1 query problem in the backend. The latency of this end point was cut by several orders of magnitude, and our database is…

Or, "here's a completely supported not-well-enough-known ORM feature that does that, that I learned about when it looked like I wasn't doing any work".

Re: How And Why We Switched from Erlang to Python (2011)

#138

Post by exelius »

Stratoscope wrote: post #88
ajross wrote: post #74

Earlier quoted context omitted.

> If they didn't build it well, you throw it away, and you're only out what you paid the temporary worker and your code review/acceptance test. Yeah, and that logic is almost inherently a violation of labor law. The whole idea behind allowing "internships" at all is that the intern is deriving value (education) from the relationship that isn't captured by wages alone. The test for whether it's legal involves how much…

I think you may be mistaken about this. If it helps any, I wasn't one of your downvoters; I upvote comments that may be incorrect but lead to an interesting and informative discussion. I live with an HR exec and read some of this thread to her while she was cutting rhubarb for something she's baking. She said (paraphrased), "Are you kidding? This is a perfect project for a summer intern. They won't have to take three…

There is immense value to the intern just for being around; the biggest things software dev interns learn isn’t how to code: it’s how to code as part of a team. So check-ins, CI/CD, automated testing, daily meeting cadence, architecture discussions, backlog grooming, code reviews... all things they don’t teach you in school, and all things that you need to know to develop software in the real world.

Most kids right out of school are honestly pretty decent coders. The internship is more about the professional skills than the technical ones.

Re: How And Why We Switched from Erlang to Python (2011)

#139

Post by teilo »

bradleyjg wrote: post #54
lightcatcher wrote: post #27

I was an intern at Mixpanel at the time (but not the author of this article or involved in the discussed rewrite). For context, the company was then My recollection of events (of 8 years ago): The Erlang endpoint was written a year or two earlier as someone's first Erlang project, in a tiny org with no Erlang experience. The endpoint worked well enough and people moved onto other projects in the mostly Python codebas…

In general language diversity at a company is a big negative. Obviously sometimes there are even bigger negatives that mandate overriding that rule--you aren't going to use Java on the front end or javascript on the backend, that would be madness--but you never want two languages in the same niche. That's a cost with no benefit. If you have some developer that wants to write something in a new (for your company) lang…

My company is an example of this, and in a way that is quite different from that laid out in the article. We actually began with a Python implementation for our integration / API server. But this was well before asyncio and eventlet and kestral. It didn't scale well, and we quickly outgrew it.

We implemented its replacement in Elixir, a solution we are still running. It is written by a developer we still have, is well organized, follows proper functional paradigms, and fully leverages OTP/Genserver. It is resilient, and so far as the one two developers we have working in it at present, easy to maintain. We love it.

But we also are switching back to Python. The reason is simple: We only have two Elixir developers (really only 1.5), and we have far more work than they can complete. And when something breaks (usually because something changed in the various systems being integrated), we have one guy who can fix it, and get our production queues flowing again.

We have looked for more Elixir developers. They are rare beasts. But we have a lot of Python developers. And now that we have asyncio/eventlet/etc., the landscape has drastically changed. In Python, the various teams can create their own worker modules, and do so more efficiently because they know their own workloads far better.

Language diversity IS a problem when it impedes development and puts production at risk.

Re: How And Why We Switched from Erlang to Python (2011)

#140

Post by bcbrown »

teilo wrote: post #139
bradleyjg wrote: post #54

Earlier quoted context omitted.

In general language diversity at a company is a big negative. Obviously sometimes there are even bigger negatives that mandate overriding that rule--you aren't going to use Java on the front end or javascript on the backend, that would be madness--but you never want two languages in the same niche. That's a cost with no benefit. If you have some developer that wants to write something in a new (for your company) lang…

My company is an example of this, and in a way that is quite different from that laid out in the article. We actually began with a Python implementation for our integration / API server. But this was well before asyncio and eventlet and kestral. It didn't scale well, and we quickly outgrew it. We implemented its replacement in Elixir, a solution we are still running. It is written by a developer we still have, is wel…

> We have looked for more Elixir developers. They are rare beasts. But we have a lot of Python developers.

Was there any consideration of training up the Python developers on Elixir?

Post reply on HN