Go + Services = One Goliath Project • Hacker News Board

Re: Go + Services = One Goliath Project

#391

Post by asdfasgasdgasdg » Sun, Dec 22, 2019, 1:58 PM UTC

andy_ppp wrote: post #330

scottrogowski wrote: post #296

Earlier quoted context omitted.

A company I am affiliated with made a decision to rewrite their code in microservices-oriented architecture thinking it would only take one year. Now we're 7 years into the transition and starting to come up against some hard deadlines that threaten revenue streams. It seems obvious to everyone except the leadership and the architects that this has been an unmitigated disaster. Other comments on this thread seem to i…

Yes, the micro services + Golang vanity project because you think you’re Google. I really don’t think people understand error states in distributed systems very well, putting possible network partitions everywhere is not a great idea. I would strongly suggest trying a Golang monolith first and seeing if there are one or two heavily used services that need splitting off. Also monorepo. Always.

The really fascinating thing about this tendency is that Google itself never completed nor even really started the wholesale transition of programs and services to Golang/microservices. Google does have services that are micro with respect to the overall codebase. But they aren't what most people out there in the wider world would think of as micro. And Golang remains a niche language at Google, perhaps more popular than server python, but far smaller in usage than Java or C++.

Re: Go + Services = One Goliath Project

#392

Post by d_burfoot » Sun, Dec 22, 2019, 2:17 PM UTC

I heard about this project from a friend who works at KA. I am concerned about the strategy, and I think the following approach would yield better results:

1. Write in Go an exact reimplementation of the current Python codebase. Use the same database schema, front-end HTML/JS, test suite, and so on. To whatever extent possible, use the same names for classes and functions. Check the reimplementation correctness by using a comparison tool that calls both the Python and Go version of a page/function/search and making sure that they produce the same results.

2. Change the production code over to the Go version, perhaps using a ramping strategy where X% of servers are running the Go code, and you gradually increase X, while monitoring vital statistics like server load and response time.

3. Now that the production site is running Go, incrementally split off components into their own services.

This approach leads you to the same destination, but with a lot less risk. It is very unhealthy to have a situation where the production site is running one codebase but all the developers are working on another codebase. Note that you will realize the benefits of Go (performance, type safety) after step 2, which is much sooner than OP's plan.

Joel Spolsky's classic essay about how you should never do full codebase rewrites is worth reviewing:

https://www.joelonsoftware.com/2000/04/06/things-you-should-...

Re: Go + Services = One Goliath Project

#393

Post by bborud » Sun, Dec 22, 2019, 2:18 PM UTC

bcrosby95 wrote: post #305

bborud wrote: post #262

Earlier quoted context omitted.

Start with a monolith that has clear internal APIs that are designed so they can later be made into network APIs. This gives you the development speed of a monolith while maintaining an options for the future. When you do break things out into separate services: try to make as few of them as possible and maintain the ability to build as a monolith. Forget everything you have heard about micro services. Most of it is…

This. If you can't design a well segmented monolith, you can't design a well segmented system of microservices either. The microservices will just be buggier and much harder to fix after the fact.

Design is best evolved. If you can get a lot of that done while you still are able to run a well structured system as a monolith you can save a lot of time. It is cheaper to change an interface in Java/Golang than a REST API.

Re: Go + Services = One Goliath Project

#394

Post by stult » Sun, Dec 22, 2019, 2:21 PM UTC

kodablah wrote: post #247

purple-again wrote: post #9

We turned our monolith into a bunch of micro services almost 6 years ago to the day. For a long time I was very happy with the new pattern but over the years the weight of keeping everything updated along with the inevitable corners that fall behind and have...questionable..security due to how long they sit neglected has really left me wondering if I am happy with it after all. I would love hear some thoughts from ot…

I've found there's a happy middleground. You need medium-sized-services that still share code libraries. For many companies, this is often 7 or 8. The key is to combine like business units/features, not necessarily fragmenting at every visible code boundary. A deployed "service" can really just be several HTTP paths and/or gRPC services in one repo. You still get to keep decent separation of work, deployment, version…

I’ve accidentally landed on an architecture like this and I’m actually pretty happy about it. It was driven by a desire to kill a large monolith slowly, by extracting key features into separate services. Sold to the customer as microservices because sexy fad of the moment. Our real motivation was we had way too much trouble recruiting in the monolith stack (.NET) and had a surplus of embedded C++, python, and JS engineers. Anyway, turns out our teams naturally self-organize around four or five domain+language clusters that effectively form separate services that are too large to really be micro, but too dissimilar to play nicely together in a monolith. Eg, python data science module wrapped in a Flask API providing physics calcs, legacy C# service providing simple data models via REST, JS react front end served from a separate node service, a weird C++/python hybrid used for an embedded device sim service, etc. It’s not what I planned as the tech lead/architect, but I think we organically reached what is really the best approach for our team. Definitely an element of Conway’s law in action, but in a good way. We are making the most of our organizational structure rather than fighting against it. Would this scale to google levels? No, probably not. But we don’t need it to, and it’s incredibly unlikely we ever would given our specific business.

Re: Go + Services = One Goliath Project

#395

Post by AnonymousPlanet » Sun, Dec 22, 2019, 2:28 PM UTC

stult wrote: post #389

AnonymousPlanet wrote: post #385

Earlier quoted context omitted.

I agree that code quality is a big factor. But what is good code quality in Python? In our case, the oldest code is the most "pythonic" and is at the same time the worst to maintain. The better code mitigates the drawbacks of dynamic typing and by that moves away from the pythonic standard you see in many libraries. But even if you nail the types to the board (e.g. assert isinstance(...)), use (the somewhat weak) Myp…

What specific “pythonic” habits have you found are more difficult to maintain? Asking out of genuine curiosity, not to challenge the premise. I work with a lot of data science people that really emphasize being pythonic, but coming from the software/static typing side of the house I always find their code style and architecture a little concerning, and I’m not sure if I’m just not getting it or if they really are wri…

One of the biggest footguns is method naming. Most Python libraries will gladly use generic method names like "add()" or "getName()". The moment you need to rename the method or change the signature, you will have a hard time telling it apart from all the other method calls by the same name. No type inference will save you here because type inference is incomplete and will never let you find all the callers.

What you should do is use unique names. But that will give you ugly code like myFoobar.foobar_addBar(). The kind of code that makes the pythonic crowd cringe.

Another problem is making code too generic with regards to what types it consumes, instead of nailing it down to the few types you're ever going to use here. This makes it hard to reason about your code months and years down the line. How is this method used in the rest of the code base? Do all callers expect an int? What if my method now happens to return a float?

And there's also abuse of duck typing. Throw around a lot of objects, sprinkling methods and other members on to them as you go. Then when you consume the object, just look if it has the method you want to call. This makes any kind of type checking and static type inference useless.

And then there's a whole lot of Python 2 libraries where you get the feeling that the authors didn't give too much thought about whether they are dealing with str or unicode. The method might just call .encode(...) on one of its arguments without being too sure what it is.

And every one of the mistakes that result from the above practices might only pop up when your code has already been shipped to the customer site.

Re: Go + Services = One Goliath Project

#396

Post by jordanbeiber » Sun, Dec 22, 2019, 2:31 PM UTC

d_burfoot wrote: post #392

I heard about this project from a friend who works at KA. I am concerned about the strategy, and I think the following approach would yield better results: 1. Write in Go an exact reimplementation of the current Python codebase. Use the same database schema, front-end HTML/JS, test suite, and so on. To whatever extent possible, use the same names for classes and functions. Check the reimplementation correctness by us…

Is moving directly to #3 not an option?

I’m thinking that defining parts that can be moved to separate services, and start consuming these could be a way to organically transition to a new architecture.

Re: Go + Services = One Goliath Project

#397

Post by stult » Sun, Dec 22, 2019, 2:35 PM UTC

bborud wrote: post #262

purple-again wrote: post #9

We turned our monolith into a bunch of micro services almost 6 years ago to the day. For a long time I was very happy with the new pattern but over the years the weight of keeping everything updated along with the inevitable corners that fall behind and have...questionable..security due to how long they sit neglected has really left me wondering if I am happy with it after all. I would love hear some thoughts from ot…

Start with a monolith that has clear internal APIs that are designed so they can later be made into network APIs. This gives you the development speed of a monolith while maintaining an options for the future. When you do break things out into separate services: try to make as few of them as possible and maintain the ability to build as a monolith. Forget everything you have heard about micro services. Most of it is…

Could not agree more. I recently led a refactor of a monolithic .NET MVC app and took this exact approach. We made all of the controllers thin, with almost no logic at all beyond specifying the route and dependency injection. Then redirected the request to a “service”, which originally was just a reworked combination of the old controller/model logic hidden behind a common service interface. Then, slowly we replaced the C# services with microservices. So we went from ball of spaghetti to monolithic service oriented architecture lite to actual microservices with the monolith converted into an API gateway. If we didn’t have independent motives for going to microservices, sticking to the clean and well organized internal APIs of the refactored monolith would have been totally fine.

Re: Go + Services = One Goliath Project

#398

Post by jordanbeiber » Sun, Dec 22, 2019, 2:37 PM UTC

NicoJuicy wrote: post #357

ako wrote: post #325

Earlier quoted context omitted.

Indeed, microservices is mostly about scaling development.

Microservices is changing dev complexity to ops. That's why most companies are promoting devs to do ops . So then you have devops

It think in ways this is true.

Containerization, autoscaling, service discovery, tracing, metrics and monitoring et al - lot of it is required to do larger scale, distributed systems. Even if you do not call them microservices.

Re: Go + Services = One Goliath Project

#399

Post by krenel » Sun, Dec 22, 2019, 2:41 PM UTC

d_burfoot wrote: post #392

I heard about this project from a friend who works at KA. I am concerned about the strategy, and I think the following approach would yield better results: 1. Write in Go an exact reimplementation of the current Python codebase. Use the same database schema, front-end HTML/JS, test suite, and so on. To whatever extent possible, use the same names for classes and functions. Check the reimplementation correctness by us…

100% agree. We've just finished to roll out our implementation in Go migrating a subsystem from PHP and receiving around 150req/second and demultiplexing those request to 1500-2000req/second to legacy backends.

The key to the success of the project was that the API was an exact match, and we could compare both implementations for exact requests. The deploy strategy of the new version:

- Reply the real traffic to the new Go service comparing the results with the old one - Then implement a toggle feature than enabled different traffic sources to use one backend or the other - Keep changing backends to the new system and ensure that metrics were unaffected

Having e2e and integration tests for the Golang project was of a huge help, since we could fix all differences using TDD.

Although we changed some of the implementations to take advantage of Go constructs, just a 1-to-1 replacement would have had a huge performance impact.

Re: Go + Services = One Goliath Project

#400

Post by AnonymousPlanet » Sun, Dec 22, 2019, 2:43 PM UTC

StavrosK wrote: post #390

AnonymousPlanet wrote: post #388

Earlier quoted context omitted.

We tried 2to3 and it gave us poor results. But probably because we deviated too far from being pythonic. The str/unicode misery is one of the biggest gripes I have with Python. I'm glad this unpleasant knot has been mostly untied in Python 3. I came to the conclusion that the transition would have been much easier if Python 3 just concentrated on the separation of bytes and (unicode) strings. The other features could…

I'm not sure that's true because, as I said, most of it is automatic and worked well with 2to3, leaving us to deal pretty much only with Unicode. I'd certainly prefer to only have to do this upgrade once.

How do you automatically go from sort( ... some elaborate compare function ...) to sort(key=some completely different function). Yes, there's a wrapper, but it makes the code more convoluted instead of transforming it to the key-paradigm. And if you want to sort by several keys, now you will have to call sort several times.

How do you automatically infer the intention of somedict.keys()? Is it going to be used as a list or as an iterator?

Those are just off the top of my head. I don't remember all the cases where 2to3 tripped over and produced garbage. But there were too many cases to put actual faith in automatic conversion.

It might work if your code is kind of new and homogenic. But looking at how much trouble Dropbox had, even with all the tooling and Guidos they could muster, I have the feeling that your positive experience with 2to3 might rather be the exception than the rule for old and big code bases.