Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster Python with Guido van Rossum

softwareatscale.dev

Post reply on HN
Print view View original

231–240 of 251 posts

Re: Faster Python with Guido van Rossum

#231

Post by BiteCode_dev »

Abishek_Muthian wrote: post #227
BiteCode_dev wrote: post #143

Earlier quoted context omitted.

A trap for your dream world were you suddenly get google size ? Because I have a 1 million unique users video streaming service still running python 2.7, using a few servers. The thing has a mobile version serving a different media on the fly, encodes user uploaded videos, features comments, tagging, and even has machine learning detection of content now. It is still maintained by one single person, and he is not a p…

> Because I have a 1 million unique users video streaming service still running python 2.7, using a few servers. Congratulations. •How many servers 2, 5, 10? •How many pure python libraries did you have to replace with a Cython extension after you realized that it cannot support your scale? •How many external dependencies? How many of them did you replace after they were abandoned? Surely not all actively developed l…

> •How many servers 2, 5, 10?

7

But the question is rather, whats the cost / ratio ? The 7 servers costs 7 less times money that they bring. The solo dev makes $150k/years with it, and doesn't need to work full time, have flexible hours and its own business.

> How many pure python libraries did you have to replace with a Cython extension after you realized that it cannot support your scale?

Many. Being extensible with compiled extension is a feature of Python. Numpy and co are part of the ecosystem. That's an awesome thing about it: I can code in Python and get c speed anyway.

> •How many external dependencies? How many of them did you replace after they were abandoned? Surely not all actively developed library are still using Python2.7.

Many. And the project still runs, with so little resource, bring the money in and serve its purpose, after 10 years of service.

> So he cannot upgrade the project to Python3.x along with all matching dependencies either, So the project stays with all bugs and vulnerabilities.

Sure he can, it's just an expense he won't do. Bugs are workaround, vulnerabilities are shielded by other parts of the system. Essential things like injections are dealt with. Some trade off are made, like always.

> So only Google size is a benchmark for scale? If I were to start a web project which has just 10 concurrent users I would choose Go because I know I can develop the entire project with little to no dependencies,

No you can't. Not if you want the first version to come out in a few months. Because you need a scrapper, and a bdd, and registration, and auth, and permissions, and screenshotting, and upload, and videos encoding, with background tasks, and load balancing all the videos and pics, tagging, i18n, xss protection, and search, etc. So either you pay to use cloud services to do all those things for you, or you leverage a rich ecosystem. Or you take a year to do it. Even if Go were the best language in the world, it can't beat 10 years of django plugins, 25 years of python modules and the iterative speed of a dynamic syntax. Well, it can, with AWS :)

In fact, Go is a very niche language. If you take it out of the I/O and concurrency specialty, it's quite average. Rust is better at raw perfs. At same perf, java has a better ecosystem. For simplicity and easy compilation, you have nim and zig that are more expressive. Resilience ? Erlang. And for speed of dev, python is better.

So for our use case it is not were it would shine, because:

> it can scale to N users without touching the code and with the confidence that the number of servers required will be 1 server only run the python code, the others are for the video hosting and the db. Python is not the bottleneck in this site. It almost never is in web dev. nginx serves the content, postgres and redis deals with the data.

Re: Faster Python with Guido van Rossum

#232

Post by m4rtink »

bborud wrote: post #230
m4rtink wrote: post #226

Earlier quoted context omitted.

I'm not sure one can really end up with one true way of packaging per language - sooner or later someone will think "this is too complicated, I can do it better and simpler!" and you end up with another way of software distribution for that language (which is just as complicated as the previous one, once the author actually notices all the usecases). This is why I prefer using the target OS dependent package manageme…

It is often more subtle. For instance, for languages that compile to binaries, you have already solved about 95% of the problem. The remaining is mostly things like static vs dynamic linking and how you actually get the binary from a distribution point into some directory on a hard-drive (which is, in part, a separate problem domain - except when it isn't :-)). Then you have languages that are "half way there", like…

Good point about scientists being quite terrible and software mainteinability - I've encountered it quite a few times myself and from what I've heard its not much better elsewhere.

I blame the publish-or-perish mechanics and grant approval process not actually motivating the participants in any way to write maintainable or reusable code. Like, I don't say they should make it good enough for industry to consume, but at least for the followup scientist to build in the work to run!

But thinking about it, even in the computer science course I studied there was hardly any emphasis on software maintenance, cove versioning or even how to collaborate with others - all non technical courses were basically about being and analyst and planning how to write a project, with zero interaction with others.

As for self contained deployment mechanisms - I do agree that effectively packing everything into a single static compiled binary like AFAIK Go and Rust effectively do has a lot of benefits. But comming from the distro background (Fedora) it also terrifies my quite a bit!

With dynamic linking you can patch CVEs by rebuilding the system library everyone uses (and not just encryption libraries can get CVEs!), you can recompile shared libraries with hardening flags and you actually know if something no longer builds from source next time rebuilding one of the parts fails.

Compared to that a developer provided massive static binary is quite a significant black box that can have multiple unpatched CVEs, compiled in fixed version of patched libraries or even non-publicly available code (meaning the binary can't be in-depndently rebuilt from source).

This scenario kinda describes a fully open source project - I guess for proprietary stuff a lot of these downsides does not really apply though out of necessity.

Re: Faster Python with Guido van Rossum

#233

Post by disgruntledphd2 »

wenc wrote: post #124
ctur wrote: post #4

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…

As someone with nearly two decades of experience with Python and who has deployed it in production in high impact scenarios, I’d like to nuance the above for people who think Python is only good for prototyping and that you have to rewrite in a “proper” programming language. This is the kind of blanket thinking to avoid. I think the rewriting part is only necessary if there’s some characteristic in your use case that…

> For many data science projects, maintaining large code bases in Python is often the optimal decision (rather than reinventing the wheel and writing your own data frame and machine learning libraries, or using immature poorly maintained ones in other languages — for data manipulation and scientific algorithms, these libraries are highly optimized in Python anyway since the underlying code is in C or Fortran)

Or you could use R, instead of the half-baked immature clones in Python :) I'm only slightly joking here, even if Python is much much better for string processing, it's much less useful for DS tasks (unless it's NLP or DL, to be fair).

That being said, Python is the second best DS language, as well as second best for everything else, so it can be a good choice for these projects.

Re: Faster Python with Guido van Rossum

#234

Post by Decabytes »

abdabab wrote: post #220
Decabytes wrote: post #61

Python will always be important to me. It's the first programming language that I've learned, and it allows me to support my family. It's awesome that almost any problem I have, there is a library or wrapper ready made for me to use. I can google just about any error and get a solution right away. But I've found that I really dislike maintaining larger Python codebases. Whenever I've tried to develop packages or libr…

> And when it comes to personal growth as a programmer, I find it's just too easy to stack ready made libraries on top of libraries, without ever coding things from scratch. This is working as intended. This is how everything should be.

I think it is as well. Python is a batteries included language. But just like I have to hide my phone sometimes to get myself to use it less, I like to drop down into a language that is more niche so I'm less likely to find YouTube videos and Stack Overflow answers to my problems. I've found being forced to read the language reference, and just allowing myself to be stuck on a problem until I've thought my way out of it, helpful for my own personal growth. I don't have the discipline to do this in Python

Re: Faster Python with Guido van Rossum

#235

Post by bborud »

m4rtink wrote: post #232
bborud wrote: post #230

Earlier quoted context omitted.

It is often more subtle. For instance, for languages that compile to binaries, you have already solved about 95% of the problem. The remaining is mostly things like static vs dynamic linking and how you actually get the binary from a distribution point into some directory on a hard-drive (which is, in part, a separate problem domain - except when it isn't :-)). Then you have languages that are "half way there", like…

Good point about scientists being quite terrible and software mainteinability - I've encountered it quite a few times myself and from what I've heard its not much better elsewhere. I blame the publish-or-perish mechanics and grant approval process not actually motivating the participants in any way to write maintainable or reusable code. Like, I don't say they should make it good enough for industry to consume, but a…

I see your point about using shared libraries to fix problems in multiple applications at once, but there is a flip side to that: what you're running isn't what you built so you may not be able to predict what your software will do when installed on different users computers.

Re: Faster Python with Guido van Rossum

#236

Post by igouy »

KronisLV wrote: post #223
igouy wrote: post #211

Earlier quoted context omitted.

Thanks. > "more real world scenario" "realistic" I'm of the opinion that "Real programs may not be representative either". http://www.larcenists.org/Twobit/bmcrock.temp.html We have to show that they are or discover they are not.

In this context I believe that if a benchmark does exactly that your software would do in a production environment (shuffling JSON around and accessing a DB), then such a benchmark is probably a rough indicator of how well any stack would work, as long as the source code is also written in an idiomatic fashion. Of course, if you can, take benchmarks with a grain of salt and ideally do some prototyping and load testin…

> … does exactly that your software would do…

Is "exactly" really the situation, or is it that the benchmark does something kind-of like one of the use cases?

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: Faster Python with Guido van Rossum

#237

Post by igouy »

makapuf wrote: post #221
igouy wrote: post #209

Earlier quoted context omitted.

re is an ordinary standard Python library — https://docs.python.org/3/library/re.html pcre2 is not — https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

re depends ultimately of this c file https://github.com/python/cpython/blob/main/Modules/_sre.c

And? CPython is implemented in C.

Here's some of a comment from _operator.c

"This module exports a set of functions implemented in C corresponding\n\ to the intrinsic operators of Python. For example, operator.add(x, y)\n\ is equivalent to the expression x+y."

Why wouldn't you expect that CPython regex would "ultimately" be written in C?

Re: Faster Python with Guido van Rossum

#238

Post by igouy »

CraigJPerry wrote: post #213
igouy wrote: post #156

Earlier quoted context omitted.

> Each example just calls out to a very fast c library… No. https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Your python 3 one isnt the fastest python 3 solution, it’s this one which uses pcre2

That's correct, the Python program that uses pcre2 is faster.

The slower program is an example from that website of a regex program which does not " just calls out to a very fast c library (pcre2)…".

Re: Faster Python with Guido van Rossum

#239

Post by brundolf »

manfre wrote: post #217
brundolf wrote: post #177

Earlier quoted context omitted.

To add to this: any time somebody at my company who doesn't work on our Python services every day has to dip into them for a one-off task, it takes a day or two of troubleshooting their environment just to get to where they can start working with the actual code. And this isn't necessarily from scratch: any Python environment that's sat still for more than a few weeks becomes inevitably broken. It's a huge time-sink…

I recommend learning to use tox and virtualenvs to avoid this headache. A properly laid out project (of any language) should have a fast time from checkout to running tests. For most python projects, that can be as simple as `pip install tox & tox`

We use pip and virtual envs; not sure about tox

Re: Faster Python with Guido van Rossum

#240

Post by fasterbynight »

prionassembly wrote: post #192
fasterbynight wrote: post #188

Earlier quoted context omitted.

I don't know about other languages, but in Julia I've heard people often say that loops end up faster than the equivalent vectorized code. So while this is true for Python/Matlab I don't think it is good universal advice. That said, matrix notation can sometimes be the more readable way of expressing a calculation.

I might have used an early beta of Julia circa 2018 or something, but the chorus that it performs like $static_fastlang doesn't match the experience I had.

I'd don't know what you mean by "performs like" but it's definitely significantly closer in runtime speed after the first run to compiled languages than interpreted (in particular Python/Matlab).

Anyways, my point was more in response to this from the person I responded to: "my instinct is to look for opportunities to vectorize by rewriting loops into matrix notation on paper and then expressing them as array calculations". In some languages (Julia in particular) that is slower than the equivalent loop based code.

Post reply on HN