Live data from Hacker News

How we rolled out one of the largest Python 3 migrations

blogs.dropbox.com

121–130 of 141 posts

Re: How we rolled out one of the largest Python 3 migrations

#121
post #88

Earlier quoted context omitted.

They're also fast and small, which while not always a big matter is very satisfying for me.

I work in a Python shop. The Docker images we build are nearly 1 GB. I just built a Go service whose image is only 2.5MB. Admittedly it’s much simpler than the Python apps, but even a complex Go app would never reach the size of our Python app for a number of reasons: 1. Python apps require a distro base image while Go can run on scratch 2. Python images ship with the full standard library; not just the bits you impo…

[deleted]

Re: How we rolled out one of the largest Python 3 migrations

#122
post #88

Earlier quoted context omitted.

They're also fast and small, which while not always a big matter is very satisfying for me.

I work in a Python shop. The Docker images we build are nearly 1 GB. I just built a Go service whose image is only 2.5MB. Admittedly it’s much simpler than the Python apps, but even a complex Go app would never reach the size of our Python app for a number of reasons: 1. Python apps require a distro base image while Go can run on scratch 2. Python images ship with the full standard library; not just the bits you impo…

[deleted]

Re: How we rolled out one of the largest Python 3 migrations

#123
post #88

Earlier quoted context omitted.

They're also fast and small, which while not always a big matter is very satisfying for me.

I work in a Python shop. The Docker images we build are nearly 1 GB. I just built a Go service whose image is only 2.5MB. Admittedly it’s much simpler than the Python apps, but even a complex Go app would never reach the size of our Python app for a number of reasons: 1. Python apps require a distro base image while Go can run on scratch 2. Python images ship with the full standard library; not just the bits you impo…

While it can't compete with 2.5MB, python:3.6-alpine (which includes points 1 and 2) weighs less than 100MB. You need a lot of Python code to get to 1GB.

Re: How we rolled out one of the largest Python 3 migrations

#124
post #117

Earlier quoted context omitted.

I work in a Python shop. The Docker images we build are nearly 1 GB. I just built a Go service whose image is only 2.5MB. Admittedly it’s much simpler than the Python apps, but even a complex Go app would never reach the size of our Python app for a number of reasons: 1. Python apps require a distro base image while Go can run on scratch 2. Python images ship with the full standard library; not just the bits you impo…

I agree, I'm not a python fan. Python has a few good libraries I can't find in other languages, but it's slow, bad for multi core utilization, hard to distribute, is very wasteful with how many dependencies need to be included, has a terrible package manager and I prefer static strong typing.

Have you tried mypy? It's a pretty good static typing system for Python - arguably better than Go's.

Re: How we rolled out one of the largest Python 3 migrations

#125
post #42

Earlier quoted context omitted.

What would you consider the main cause(s) of that?

The job of a programmer is to write code, so that’s what they do. I doubt there are very many people at Dropbox whose job it is to remove code!

The classic story: https://www.folklore.org/StoryView.py?story=Negative_2000_Li...

Re: How we rolled out one of the largest Python 3 migrations

#126
post #117

Earlier quoted context omitted.

I agree, I'm not a python fan. Python has a few good libraries I can't find in other languages, but it's slow, bad for multi core utilization, hard to distribute, is very wasteful with how many dependencies need to be included, has a terrible package manager and I prefer static strong typing.

Have you tried mypy? It's a pretty good static typing system for Python - arguably better than Go's.

TL;DR - I continue to root for Python's typing story, but it's just not there yet.

I have, and I wanted to like it. On its face it seems like it should be a lot better than Go's--after all, it supports generics and union types! But it falls over in trivial cases, like recursive types (i.e., there's no way to model tree structures such as JSON or linked lists). A few other hard/impossible/confusing things come to mind:

1. How do you declare a typevar for a certain scope. If I define a type parameter `T` for function `foo`, I only want `T` to be scoped to `foo`. I don't want the type checker getting confused with `T`s for other functions/classes/etc. 2. What is the signature for a function that takes args/kwargs? 3. It straight up doesn't work with popular libraries like SQLAlchemy (last I checked, these were simply not supported because the likes of SQLAlchemy are "too magical"--this is a fair take, but frustratingly limiting for users).

These are just a few because my memory is poor, but I run into these sorts of things by the dozens every time I try to use mypy. It's just not ready for prime time. Go's type system is limiting, but its limitations are much more predictable and even less limiting (it turns out recursive types and poor-man's union types are quite a bit better than first-class, non-recursive union types, for example).

Re: How we rolled out one of the largest Python 3 migrations

#127

Earlier quoted context omitted.

I work in a Python shop. The Docker images we build are nearly 1 GB. I just built a Go service whose image is only 2.5MB. Admittedly it’s much simpler than the Python apps, but even a complex Go app would never reach the size of our Python app for a number of reasons: 1. Python apps require a distro base image while Go can run on scratch 2. Python images ship with the full standard library; not just the bits you impo…

While it can't compete with 2.5MB, python:3.6-alpine (which includes points 1 and 2) weighs less than 100MB. You need a lot of Python code to get to 1GB.

Fair point. Our largest Python image has only 255 Mb of Python dependencies and ~50 Mb of source code. If we could use alpine (our compliance auditors strongly prefer centos base images), it would only be ~400 Mb. This is easily an order of magnitude bigger than an equivalent Go program, but still quite a lot better.

Re: How we rolled out one of the largest Python 3 migrations

#128

Earlier quoted context omitted.

While it can't compete with 2.5MB, python:3.6-alpine (which includes points 1 and 2) weighs less than 100MB. You need a lot of Python code to get to 1GB.

Fair point. Our largest Python image has only 255 Mb of Python dependencies and ~50 Mb of source code. If we could use alpine (our compliance auditors strongly prefer centos base images), it would only be ~400 Mb. This is easily an order of magnitude bigger than an equivalent Go program, but still quite a lot better.

Is that really just Python code?? Must be over a million lines, no? I've worked with Odoo, which is a bit of a kitchen-sink (ERP, CRM, POS, sales, accounting, invoice, stock management, manufacturing control, website builder, marketing and a bunch more) and its Python code weighs just 15MB, the rest is JavaScript or data files.

Re: How we rolled out one of the largest Python 3 migrations

#129

Earlier quoted context omitted.

I work in a Python shop. The Docker images we build are nearly 1 GB. I just built a Go service whose image is only 2.5MB. Admittedly it’s much simpler than the Python apps, but even a complex Go app would never reach the size of our Python app for a number of reasons: 1. Python apps require a distro base image while Go can run on scratch 2. Python images ship with the full standard library; not just the bits you impo…

While it can't compete with 2.5MB, python:3.6-alpine (which includes points 1 and 2) weighs less than 100MB. You need a lot of Python code to get to 1GB.

My favorite Python library is pandas. If I only include this with pyinstaller I already get a >500MB executable.

IMHO unacceptably large, but that's because Python doesn't know which parts of pandas it might need to execute the program.

Re: How we rolled out one of the largest Python 3 migrations

#130

Earlier quoted context omitted.

Have you tried mypy? It's a pretty good static typing system for Python - arguably better than Go's.

TL;DR - I continue to root for Python's typing story, but it's just not there yet. I have, and I wanted to like it. On its face it seems like it should be a lot better than Go's--after all, it supports generics and union types! But it falls over in trivial cases, like recursive types (i.e., there's no way to model tree structures such as JSON or linked lists). A few other hard/impossible/confusing things come to mind…

It's also tacked on. It's a bit too optional. If one team member doesn't care about typing he makes all his colleagues do the work to make his code work with mypy.
Post reply on HN