Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we rolled out one of the largest Python 3 migrations

blogs.dropbox.com

Post reply on HN
Print view View original

111–120 of 141 posts

Re: How we rolled out one of the largest Python 3 migrations

#111

Post by goerz »

fabioz wrote: post #101
Rotareti wrote: post #97

Earlier quoted context omitted.

I hope some day some big corp decides to write a decent (open source) Python to C compiler, which makes optional optimizations based on type annotations. I find this project: https://github.com/Nuitka/Nuitka very interesting, but its written and maintained only by a single person and I never got it to work with any of my apps.

Cython is a pretty good Python to C compiler and in its latest release is using type annotations... the thing is, you shouldn't need to compile your whole program (compiling is really slow and interpreted Python is fast enough for most of the code).

I was under the impression that Cython is a different language (a subset / dialect of Python). So, not a Python to C compiler.

Re: How we rolled out one of the largest Python 3 migrations

#112

Post by weberc2 »

Already__Taken wrote: post #105
NLips wrote: post #99

Earlier quoted context omitted.

Being charitable, I think the parent means that the python developer sticks all the dependencies in a directory and creates a bash script to set PYTHONPATH and launch. The user receives a directory rather than an executable, but only has to use the bash script, rather than worry about any of the Python in the directory.

Now your lovingly created cross-platform app is back to linux only because you used bash to invoke it.

Use a Python script instead.

Re: How we rolled out one of the largest Python 3 migrations

#113

Post by weberc2 »

carlmr wrote: post #88
oweiler wrote: post #86

Earlier quoted context omitted.

That's why most of the tools I use in my day to day work are Go or Rust binaries.

They're also fast and small, which while not always a big matter is very satisfying for me.

I work in a Python shop. The Docker images we build are nearly 1 GB. I just built a Go service whose image is only 2.5MB. Admittedly it’s much simpler than the Python apps, but even a complex Go app would never reach the size of our Python app for a number of reasons:

1. Python apps require a distro base image while Go can run on scratch

2. Python images ship with the full standard library; not just the bits you import

3. In Python, if you add a dependency only to use 1 function or variable, you still end up with the whole dependency in your Docker image, while I’m pretty sure Go’s linker strips unused code.

Re: How we rolled out one of the largest Python 3 migrations

#114

Post by weberc2 »

kccqzy wrote: post #76
mr_toad wrote: post #48

> However, rather than use the native toolchains directly, such as Xcode for macOS, we delegated the creation of platform-compliant binaries to py2exe for Windows, py2app for macOS, and bbfreeze for Linux. I wish the authors of more Python tools would deploy standalone applications. I do not like having to maintain various sets of Python installers/package managers (because every Python tool seems to use a different…

Even when I am a python developer, there is still a difference between (a) this dependency that I explicitly rely on and that I need to sort out packaging issues for, and (b) code I treat as a black box and simply expect to work. More often than not, a typical developer's python environment tends towards https://xkcd.com/1987/

That xkcd is missing virtualenv, pyenv, pipenv, venv, and poetry.

Re: How we rolled out one of the largest Python 3 migrations

#115

Post by jnwatson »

sandGorgon wrote: post #70

Is there any sample code that illustrates this kind of python embedding versus one of the freezer scripts ? We used to build desktop software using pyqt and freeze it. I wonder how that entire toolchain looks like in this new way.

See https://docs.python.org/3.7/extending/embedding.html.

The idea behind embedding is you might have a Python shell in a larger app. But you can also use it to tightly control the execution of the interpreter.

Re: How we rolled out one of the largest Python 3 migrations

#116

Post by oblio »

weberc2 wrote: post #112
Already__Taken wrote: post #105

Earlier quoted context omitted.

Now your lovingly created cross-platform app is back to linux only because you used bash to invoke it.

Use a Python script instead.

Nah, he should use some turtles instead.

Re: How we rolled out one of the largest Python 3 migrations

#117

Post by carlmr »

weberc2 wrote: post #113
carlmr wrote: post #88

Earlier quoted context omitted.

They're also fast and small, which while not always a big matter is very satisfying for me.

I work in a Python shop. The Docker images we build are nearly 1 GB. I just built a Go service whose image is only 2.5MB. Admittedly it’s much simpler than the Python apps, but even a complex Go app would never reach the size of our Python app for a number of reasons: 1. Python apps require a distro base image while Go can run on scratch 2. Python images ship with the full standard library; not just the bits you impo…

I agree, I'm not a python fan. Python has a few good libraries I can't find in other languages, but it's slow, bad for multi core utilization, hard to distribute, is very wasteful with how many dependencies need to be included, has a terrible package manager and I prefer static strong typing.

Re: How we rolled out one of the largest Python 3 migrations

#118

Post by SEJeff »

samfisher83 wrote: post #33
SEJeff wrote: post #20

Earlier quoted context omitted.

I'm not sure why it is downvoted. This is true. I write both Golang and Python as part of my day job. I love both languages, but this is definitely the truth. Go is more verbose. It also gives you the wonder of the compiler telling you about doing stupid things. That doesn't make it a worse language.

Does go not have exception handling?

Go does not have exception handling at all. It is a proposal they're seriously considering for go2:

https://go.googlesource.com/proposal/+/master/design/go2draf...

See the bits about error handling and error values.

Re: How we rolled out one of the largest Python 3 migrations

#119

Post by cjhanks »

lugg wrote: post #67
cjhanks wrote: post #55

Earlier quoted context omitted.

This is possible and actually pretty trivial. You install all 3rd party dependencies into some directory. The command line entry point is then a simple BASH script which sets the PYTHONPATH to the appropriate installation location and then does the appropriate exec call. You then have a functionally portable python installation.

Pretty trivial? Really? I think as soon as you start writing all caps trivial has gone right out the window. Trivial would be application dependencies managed by the system package manager. I'm at the point where I won't touch a python app or library that can't be installed via Pacman. It's just not worth my time. And... python is still the only thing on arch that gives me the shits everytime it upgrades.

It takes approximately a year to learn to walk after you are born. But almost everybody figures it out. And once they know, they practically never forget. So I don't know, is it not trivial?

I have never had to support all of the mobile environments of DropBox nor the scale, so I cannot claim that my 99% solution would ever meet their 99.999% requirements. But I have been able to package Python apps for Mac OSX, WinVista, Win7, Ubuntu, and CentOS at the same time using that strategy.

Re: How we rolled out one of the largest Python 3 migrations

#120

Post by lathiat »

vemv wrote: post #50

I hope this kills (or helps killing) the 100+ thread count I have always seen in macOS. It surpasses any other thread count from far more important/sophisticated processes. I'd say that's a waste (if not abuse) of the system's resources and scheduling system.

I wouldn't be that worried about 100 threads, when you have 10s of thousands you can think about worrying.
Post reply on HN