Live data from Hacker News

How we rolled out one of the largest Python 3 migrations

blogs.dropbox.com

131–140 of 141 posts

Re: How we rolled out one of the largest Python 3 migrations

#131
post #129

Earlier quoted context omitted.

While it can't compete with 2.5MB, python:3.6-alpine (which includes points 1 and 2) weighs less than 100MB. You need a lot of Python code to get to 1GB.

My favorite Python library is pandas. If I only include this with pyinstaller I already get a >500MB executable. IMHO unacceptably large, but that's because Python doesn't know which parts of pandas it might need to execute the program.

[deleted]

Re: How we rolled out one of the largest Python 3 migrations

#132
post #129

Earlier quoted context omitted.

While it can't compete with 2.5MB, python:3.6-alpine (which includes points 1 and 2) weighs less than 100MB. You need a lot of Python code to get to 1GB.

My favorite Python library is pandas. If I only include this with pyinstaller I already get a >500MB executable. IMHO unacceptably large, but that's because Python doesn't know which parts of pandas it might need to execute the program.

Code pruning is a problem in Go too; as soon as your codebase or any of your dependencies use reflection, it gets disabled for the whole binary.

A simpler alternative would be for Numpy and Pandas to provide its features as subpackages, like Airflow does: https://airflow.readthedocs.io/en/latest/installation.html

Re: How we rolled out one of the largest Python 3 migrations

#133
post #118

Earlier quoted context omitted.

Does go not have exception handling?

Go does not have exception handling at all. It is a proposal they're seriously considering for go2: https://go.googlesource.com/proposal/+/master/design/go2draf... See the bits about error handling and error values.

What was the reason the go designers didn't have exception handling?

Re: How we rolled out one of the largest Python 3 migrations

#134
post #129

Earlier quoted context omitted.

My favorite Python library is pandas. If I only include this with pyinstaller I already get a >500MB executable. IMHO unacceptably large, but that's because Python doesn't know which parts of pandas it might need to execute the program.

Code pruning is a problem in Go too; as soon as your codebase or any of your dependencies use reflection, it gets disabled for the whole binary. A simpler alternative would be for Numpy and Pandas to provide its features as subpackages, like Airflow does: https://airflow.readthedocs.io/en/latest/installation.html

I’m almost certain that only applies to individual compilation units and not the whole AST. In other words, if I use reflection in my main package, code pruning still works on dependencies, which is quite a lot better than the Python situation.

Re: How we rolled out one of the largest Python 3 migrations

#135

Earlier quoted context omitted.

Fair point. Our largest Python image has only 255 Mb of Python dependencies and ~50 Mb of source code. If we could use alpine (our compliance auditors strongly prefer centos base images), it would only be ~400 Mb. This is easily an order of magnitude bigger than an equivalent Go program, but still quite a lot better.

Is that really just Python code?? Must be over a million lines, no? I've worked with Odoo, which is a bit of a kitchen-sink (ERP, CRM, POS, sales, accounting, invoice, stock management, manufacturing control, website builder, marketing and a bunch more) and its Python code weighs just 15MB, the rest is JavaScript or data files.

It's not just souce code--it's also docs and test code and other things that are tedious to omit given our current Docker image hierarchy and repository structure. Our docs are largely Sphinx docs in Python docstrings; a decent minifier could probably reduce this, but it's probably not worthwhile for a ~5% improvement on overall image size.

Re: How we rolled out one of the largest Python 3 migrations

#136
post #118

Earlier quoted context omitted.

Go does not have exception handling at all. It is a proposal they're seriously considering for go2: https://go.googlesource.com/proposal/+/master/design/go2draf... See the bits about error handling and error values.

What was the reason the go designers didn't have exception handling?

Read Rob Pike's post on why: https://commandcenter.blogspot.com/2012/06/less-is-exponenti...

The go2 proposal shows that in this respect, he was woefully wrong.

Re: How we rolled out one of the largest Python 3 migrations

#137

Earlier quoted context omitted.

We can simply start by asking what does “syncing files” include? Watching files. Keeping backup of files. Keeping conflicts resolved. Watching Selective Sync files and folders. Watching Smart Sync files and folders. Notifications for synced files. Etc. etc. There’s way more the client does than what I mention.

I don't think there is much code for conflict-resolving in dropbox. Usually in case of conflicts it renames one of the involved files and add a message about conflict and the date to the name and moves on.

Perhaps there's lots of code to minimize the number of times a real conflict occurs?

Re: How we rolled out one of the largest Python 3 migrations

#138

Earlier quoted context omitted.

Python being EOL sounds scary but actually won't matter to Apple. They already apply custom patches, they can carry on running python 2.7 forever, with minor bug fixes where really required.

> they can carry on running python 2.7 forever would they do this? does anything macOS internal depend on python 2.7?

the `xattr` terminal command relies on the system installed Python, though this seems to be the only example.

If you look at the source in `/usr/bin/xattr` it does some work to deal with different versions of Python. All the work ultimately gets handled by the xattr module preinstalled with the system Python. This module has Apple's copyright in it and is different than the `xattr` module on pypi.

Wonder how this Python one-off in macOS came to be.

Re: How we rolled out one of the largest Python 3 migrations

#139
post #117

Earlier quoted context omitted.

I agree, I'm not a python fan. Python has a few good libraries I can't find in other languages, but it's slow, bad for multi core utilization, hard to distribute, is very wasteful with how many dependencies need to be included, has a terrible package manager and I prefer static strong typing.

Have you tried mypy? It's a pretty good static typing system for Python - arguably better than Go's.

I'd like type annotations to be used to optimise performance. After all, if it has been statically verified that a particular variable is always an instance of class X, why not use that to optimise code?

This is an argument for type annotations to be integrated into every dynamically typed language, rather than tacked on via an external tool.

Re: How we rolled out one of the largest Python 3 migrations

#140
post #48

> However, rather than use the native toolchains directly, such as Xcode for macOS, we delegated the creation of platform-compliant binaries to py2exe for Windows, py2app for macOS, and bbfreeze for Linux. I wish the authors of more Python tools would deploy standalone applications. I do not like having to maintain various sets of Python installers/package managers (because every Python tool seems to use a different…

Good point. I never understood why the developers of every language don't make it trivial to build a .exe or .app file that users can double-click to run. Seems like the Python team is penalising developers for using Python :)

It can internally use an interpreter, JIT, bytecode or full native code like C. That's a different discussion. Just don't make it a pain to distribute.

Post reply on HN