Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go + Services = One Goliath Project

engineering.khanacademy.org

Post reply on HN
Print view View original

401–410 of 449 posts

Re: Go + Services = One Goliath Project

#401

Post by KhalPanda »

d_burfoot wrote: post #392

I heard about this project from a friend who works at KA. I am concerned about the strategy, and I think the following approach would yield better results: 1. Write in Go an exact reimplementation of the current Python codebase. Use the same database schema, front-end HTML/JS, test suite, and so on. To whatever extent possible, use the same names for classes and functions. Check the reimplementation correctness by us…

Sounds nice in theory, but won't you always be playing catch-up? Or will development(/bugfixes) halt on the existing python version? Would it ever be acceptable to the business to commence such a rewrite without any show of value until after completion?

(Speaking as someone working at a firm that did choose (before my time) to start a ground-up rewrite than ran about 3 years over estimate).

Re: Go + Services = One Goliath Project

#402

Post by sethammons »

d_burfoot wrote: post #392

I heard about this project from a friend who works at KA. I am concerned about the strategy, and I think the following approach would yield better results: 1. Write in Go an exact reimplementation of the current Python codebase. Use the same database schema, front-end HTML/JS, test suite, and so on. To whatever extent possible, use the same names for classes and functions. Check the reimplementation correctness by us…

Having done this kind of migration/rewrite multiple times, the way you succeed is starting with acceptance tests that verify functionality from the API layer that is implementation agnostic.

After the tests are in place, you break off small portions into microservises and ensure tests pass.

You pass a small percentage of traffic through the new arch, fixing bugs and leveraging telemetry.

You eventually slide all traffic to the new architecture.

You want to get something receiving traffic asap as to start getting feedback. Often this means taking something smaller or simpler out of the lagacy codebase first.

Doing a complete, side by side replica is a recipe for disaster. Think MVP. We've even introduced traffic routing based on feature sets so you can route users who don't use edge case features the new arch which has yet to incorporate the those features while keeping others on the old arch.

Re: Go + Services = One Goliath Project

#403

Post by StavrosK »

AnonymousPlanet wrote: post #400
StavrosK wrote: post #390

Earlier quoted context omitted.

I'm not sure that's true because, as I said, most of it is automatic and worked well with 2to3, leaving us to deal pretty much only with Unicode. I'd certainly prefer to only have to do this upgrade once.

How do you automatically go from sort( ... some elaborate compare function ...) to sort(key=some completely different function). Yes, there's a wrapper, but it makes the code more convoluted instead of transforming it to the key-paradigm. And if you want to sort by several keys, now you will have to call sort several times. How do you automatically infer the intention of somedict.keys()? Is it going to be used as a l…

> How do you automatically go from sort( ... some elaborate compare function ...) to sort(key=some completely different function). Yes, there's a wrapper

Yes, there's a wrapper. You use it, add a comment "this is wrapped in the migration to 3" and move on.

> How do you automatically infer the intention of somedict.keys()? Is it going to be used as a list or as an iterator?

If it's being iterated on first thing, it's an iterator. If list methods are called on to it, it's a list. This just hasn't been a problem for us, sure, it took some looking at, but it wasn't more than 30 seconds per case.

> I have the feeling that your positive experience with 2to3 might rather be the exception than the rule for old and big code bases.

Maybe so, but the codebase was ten years old and hundreds of thousands of lines.

Re: Go + Services = One Goliath Project

#404

Post by danenania »

andy_ppp wrote: post #356
pm90 wrote: post #337

Earlier quoted context omitted.

I can easily ask the vice versa of such a bland question. Answer that first if you want a real answer instead of trying to flame.

It’s not bland it’s direct. What you’re doing is conflating personal preference with actual language features. Most developers who aren’t us would say python, being a higher level, dynamic scripting language rather than Golang which is lower level and extremely explicit about the data your program is using. Iteration in go is simply harder as you have to be more explicit rather than sketching something out. Changing…

Dynamic languages generally enable faster iteration in the early stages of a project, but once you have a large, mature codebase, static types allow you to work faster and produce fewer bugs. A more performant language will also run your test suite faster which can have a big impact as a project gets very large.

Also, while I agree Go’s error handling isn’t very elegant, it does force you to explicitly consider every potential error, which in my experience makes uncaught errors far less likely than a language with bubbling exceptions.

Re: Go + Services = One Goliath Project

#405

Post by sethammons »

dangoor wrote: post #211
purple-again wrote: post #9

We turned our monolith into a bunch of micro services almost 6 years ago to the day. For a long time I was very happy with the new pattern but over the years the weight of keeping everything updated along with the inevitable corners that fall behind and have...questionable..security due to how long they sit neglected has really left me wondering if I am happy with it after all. I would love hear some thoughts from ot…

Our approach to services at Khan Academy is likely a bit different from most. We're sticking with a monorepo (the code for all services lives in one repository). We have a single go.mod file at the top of the repo, so all services use the _same versions_ of dependencies. We're still building out our deployment system to better support multiple services, but we're planning to redeploy all of the services when library…

I'm in an org that runs multiple services in Go. The dependencies on library stuff has been a very minimal need. I think you are optimizing for an imaginary problem. With microservises, a team should have non breaking API versions running and work with teams to transition to a new API version when needed. If the underlying uuid lib or kaftka lib changes, teams may or may not need to update, but they can do so on their own time.

Re: Go + Services = One Goliath Project

#406

Post by sethammons »

andy_ppp wrote: post #331
pm90 wrote: post #287

Earlier quoted context omitted.

* It’s not just that Go is faster to run but also faster to iterate on. If python can be neither, its offering little benefit. * they moved from a monolith to a microservices architecture; concern that any of the services in the request path could add latency just because of the overall runtime speed is slow is a legitimate one. * their primary deployment method is Google App Engine where you are billed by CPU used.…

Go is faster to iterate on is absolutely false. Where do you get this idea from?

Having worked in larger async twisted Python and Go, the experience of our teams is that Go is faster to iterate on by a long shot. We've replaced most of the old Python. We just brought on some new devs on my team. They were able to make new contributions to the Go stuff in short order. The Twisted Python, not so much.

Re: Go + Services = One Goliath Project

#407

Post by krenel »

andy_ppp wrote: post #356
pm90 wrote: post #337

Earlier quoted context omitted.

I can easily ask the vice versa of such a bland question. Answer that first if you want a real answer instead of trying to flame.

It’s not bland it’s direct. What you’re doing is conflating personal preference with actual language features. Most developers who aren’t us would say python, being a higher level, dynamic scripting language rather than Golang which is lower level and extremely explicit about the data your program is using. Iteration in go is simply harder as you have to be more explicit rather than sketching something out. Changing…

IMHO the affirmation of

> Go is faster to iterate

holds true considering the total lifespan of the project. Golang is more explicit thus requiring more time to define every type but I've never refactored so fast and safe a codebase. In Python the fact that is dynamic makes it more difficult to safely iterate over it (is more statically-typed Vs dynamically typed). About error handling, it's not perfect, but the code is readable, easy to follow and easy to reason about.

> Iteration in go is simply harder as you have to be more explicit rather than sketching something out.

I will completely agree that prototyping in Python is way faster. Python is my preferred language for throwaway/prototype code.

> Where do you get this idea from?

It's solely based in my experience. I hope it contributes to the conversation.

Re: Go + Services = One Goliath Project

#408

Post by IshKebab »

ako wrote: post #325
twblalock wrote: post #282

Earlier quoted context omitted.

In my experience, the biggest benefit of microservices is decoupling teams . Developer productivity is very hard to maintain in a monolithic app as the number of developers increases and the legacy code piles up. Breaking up services and giving each dev team control over their own codebases enables them to develop their own products at their own pace. If you only have one dev team, microservices are a lot less attrac…

Indeed, microservices is mostly about scaling development.

This is nonsense. You can already do that via libraries. The choice of RPC vs local procedure calls has no effect on scaling development.

IMO the only reason to use micro-services is the one they mentioned - you can have different parts of your system running on different machines so they can be spun up independently. But I think most people aren't "web scale" enough to need that anyway.

Re: Go + Services = One Goliath Project

#409

Post by Thaxll »

apta wrote: post #380
Thaxll wrote: post #239

Earlier quoted context omitted.

I used both Java and Go in production and yes Go can be faster or Java it depends but Go usually use between 3 and 10x less memory. https://www.techempower.com/benchmarks/ ( real world benchmark with networking / serialization ect ... ) Get over it yes Go can be faster than Java it's not a secret.

> real world benchmark Highly debatable. Even then, you see that golang ranks in the 102nd place for the plaintext benchmark, almost 6x slower than Netty, which is very established in the JVM world. Same with the JSON benchmark, golang is in the 126th place, ~3.5x slower than Netty and Vertx. > but Go usually use between 3 and 10x less memory. The JVM by default will use whatever memory is assigned to it. This makes…

You didn't read the benchmark properly, plaintext benchmark it's 6th.

Re: Go + Services = One Goliath Project

#410

Post by tony »

dangoor wrote: post #260
tony wrote: post #240

You can start porting your live codebase to python 3 right now - no downtime at all. You can have a 2+3 compatible codebase live in a week or two, it'll get you about 98% the way there [1] My projects (e.g. https://tmuxp.git-pull.com , https://unihan-etl.git-pull.com ) are both python 2/3 compatible. I learned by reading through other projects like Sphinx, Flask, Werkzeug, and SQLAlchemy: - Flask: https://github.com/…

I appreciate all of the links and the fact that, for a lot of folks, this is the state of Python 2 to 3 migration today. What you're suggesting is, imho, the best path for most (and I appreciate you mentioning the pickle incompatibility because that is a thing which would trip people up when trying to make the move). If we would have been able to have a 2&3 compatible codebase live in a week or two, we absolutely wou…

Thanks for your reply to my reply! Sure, it may take more than two weeks, and maybe golang is a good choice for your stack. Maybe a better way of saying it is python 2/3 compatibility can be gradually adopted while maintaining.

Also, I can't speak for the technical specifics of your project, but I would like to speak a bit from my prior experiences:

> Incompatible changes in some libraries we use

What are those libraries? There may be python2/3 forks available on pypi. For instance, on Peergrade we went from pyPdf -> pyPdf2, boto -> boto3 (that one is a lot of work). We were still able to stick on the python 2 codebase, but gain python 3 forward compatibility.

In the case of very specific packages, we had to do forks. We had to move some patches from a python 2 only library and port them into a python 3 only library, then do one of those version constraint things.

> App Engine first gen to second gen changes (which are for the better, but still a big deal)

I'm not familiar with app engine so can't speak to it.

> the choice of storing some pickles permanently

Can you clarify this?

I can't speak for it without seeing it, but would it be possible to serialize the data to json, use python import strings (https://devel.tech/tips/n/djms3tTe/how-django-uses-deferred-...), then write a migration to port the old pickles over, and have something more portable?

It's possible, depending on what's being stored, it'd also help you migrate to golang if the data stored in it could be consumed by a go service.

> plus a need to really verify unicode handling all over the place (especially in a 10 year old codebase)

Yes I had a lot of pitfalls with this one. Areas to look out for are hashing functions. They are very strict in whether they're dealing with bytes or strings.

Are you already using unicode_literals? Those can be implemented gradually in a current codebase.

How is your test suite? Sometimes having test coverage, even naively, can also act as a smoke test to catch unicode issues.

Selenium tests can get a lot of coverage for very cheap to verify behavior at a high level.

> Moving to Go is more work than moving to Python 3, but in our particular case it's not as much more work as people might expect.

I think the services + golang part is smart, and can't speak for the specifics of your codebase.

I will say in hindsight, I feel the effort / hours I put into upgrading a Python 2 codebase -> 2/3 and eventually 3-only made it much easier to breath. Cleaner syntax, no unicode headaches, and no need to have the lingering prospect of a language exodus looming over the head.

One negative aspect we haven't talked about those with "gradual" python 2/3 migrating is breakages. There are refactors that end up being done that when pushed out risk breaking - and when they're purely internal code changes. There is a business case to eliminate the tech debt because the value equation: A python 2 codebase, even with warts to it, is meeting an EOL in the next few days (https://pythonclock.org/).

Even assuming golang + microservices is the final destination. Amortized, on the projects I've been on, moving to python 2/3 (or 3-only) paid back. If there's opportunities where the python codebase could be split into apps / separate wsgi entry points, then have golang services replace them later - that could be an option. Even without golang, the (probably huge?) refactors involved in moving to python 3 and being "(micro?)service ready" has benefits.

Post reply on HN