Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python numbers every programmer should know

mkennedy.codes

Post reply on HN
Print view View original

71–80 of 191 posts

Re: Python numbers every programmer should know

#71

Post by oivey »

fooker wrote: post #67
oivey wrote: post #59

Earlier quoted context omitted.

People generally aren’t rolling their own matmuls or joins or whatever in production code. There are tons of tools like Numba, Jax, Triton, etc that you can use to write very fast code for new, novel, and unsolved problems. The idea that “if you need fast code, don’t write Python” has been totally obsolete for over a decade.

Yes, that's what I said. If you are writing performance sensitive code that is not covered by a popular Python library, don't do it unless you are a megacorp that can put a team to write and maintain a library.

It isn’t what you said. If you want, you can write your own matmul in Numba and it will be roughly as fast as similar C code. You shouldn’t, of course, for the same reason handrolling your own matmuls in C is stupid.

Many problems can performantly solved in pure Python, especially via the growing set of tools like the JIT libraries I cited. Even more will be solvable when things like free threaded Python land. It will be a minority of problems that can’t be, if it isn’t already.

Re: Python numbers every programmer should know

#72

Post by hibikir »

nutjob2 wrote: post #52
thundergolfer wrote: post #48

A lot of people here are commenting that if you have to care about specific latency numbers in Python you should just use another language. I disagree. A lot of important and large codebases were grown and maintained in Python (Instagram, Dropbox, OpenAI) and it's damn useful to know how to reason your way out of a Python performance problem when you inevitably hit one without dropping out into another language, whic…

> A lot of important and large codebases were grown and maintained in Python How does this happen? Is it just inertia that cause people to write large systems in a essentially type free, interpreted scripting language?

Small startups end up writing code in whatever gets things working faster, because having too large a codebase with too much load is a champagne problem.

If I told you that we were going to be running a very large payments system, with customers from startups to Amazon, you'd not write it in ruby and put the data in MongoDB, and then using its oplog as a queue... but that's what Stripe looked like. They even hired a compiler team to add type checking to the language, as that made far more sense than porting a giant monorepo to something else.

Re: Python numbers every programmer should know

#73

Post by [deleted] »

its-summertime wrote: post #69
Quothling wrote: post #41

Earlier quoted context omitted.

Why? I've build some massive analytic data flows in Python with turbodbc + pandas which are basically C++ fast. It uses more memory which supports your point, but on the flip-side we're talking $5-10 extra cost a year. It could frankly be $20k a year and still be cheaper than staffing more people like me to maintain these things, rather than having a couple of us and then letting the BI people use the tools we provid…

From the complete opposite side, I've built some tiny bits of near irrelevant code where python has been unacceptable, e.g. in shell startup / in bash's PROMPT_COMMAND, etc. It ends up having a very painfully obvious startup time, even if the code is nearing the equivalent of Hello World time python -I -c 'print("Hello World")' real 0m0.014s time bash --noprofile -c 'echo "Hello World"' real 0m0.001s

[deleted]

Re: Python numbers every programmer should know

#74

Post by kingstnap »

zelphirkalt wrote: post #6

I doubt there is much to gain from knowing how much memory an empty string takes. The article or the listed numbers have a weird fixation on memory usage numbers and concrete time measurements. What is way more important to "every programmer" is time and space complexity, in order to avoid designing unnecessarily slow or memory hungry programs. Under the assumption of using Python, what is the use of knowing that you…

I disagree. Performance is a leaky abstraction that *ALWAYS* matters.

Your cognition of it is either implicit or explicit.

Even if you didn't know for example that list appends was linear and not quadratic and fairly fast.

Even if you didn't give a shit if simple programs were for some reason 10000x slower than they needed to be because it meets some baseline level of good enough / and or you aren't the one impacted by the problems inefficacy creates.

Library authors beneath you would still know and the APIs you interact with and the pythonic code you see and the code LLMS generate will be affected by that leaky abstraction.

If you think that n^2 naive list appends is a bad example its not btw, python string appends are n^2 and that has and does affect how people do things, f strings for example are lazy.

Similarly a direct consequence of dictionaries being fast in Python is that they are used literally everywhere. The old Pycon 2017 talks from Raymond talk about this.

Ultimately what the author of the blog has provided is this sort of numerical justification for the implicit tacit sort of knowledge performance understanding gives.

Re: Python numbers every programmer should know

#75

Post by AtlasBarfed »

Demiurge wrote: post #39
fooker wrote: post #5

Counterintuitively: program in python only if you can get away without knowing these numbers. When this starts to matter, python stops being the right tool for the job.

I agree. I've been living off Python for 20 years and have never needed to know any of these numbers, nor do I need them now, for my work, contrary to the title. I also regularly use profiling for performance optimization and opt for Cython, SWIG, JIT libraries, or other tools as needed. None of these numbers would ever factor into my decision-making.

.....

You don't see any value in knowing that numbers?

Re: Python numbers every programmer should know

#76

Post by bathtub365 »

fooker wrote: post #5

Counterintuitively: program in python only if you can get away without knowing these numbers. When this starts to matter, python stops being the right tool for the job.

These basically seem like numbers of last resort. After you’ve profiled and ruled out all of the usual culprits (big disk reads, network latency, polynomial or exponential time algorithms, wasteful overbuilt data structures, etc) and need to optimize at the level of individual operations.

Re: Python numbers every programmer should know

#78

Post by i_am_a_peasant »

thundergolfer wrote: post #48

A lot of people here are commenting that if you have to care about specific latency numbers in Python you should just use another language. I disagree. A lot of important and large codebases were grown and maintained in Python (Instagram, Dropbox, OpenAI) and it's damn useful to know how to reason your way out of a Python performance problem when you inevitably hit one without dropping out into another language, whic…

our build system is written in python, and i’d like it not to suck but still stay in python, so these numbers very much matter.

Re: Python numbers every programmer should know

#79

Post by snakepit »

This is helpful. Someone should create a similar benchmark for the BEAM. This is also a good reminder to continue working on snakepit [1] and snakebridge [2]. Plenty remains before they're suitable for prime time.

[1] https://hex.pm/packages/snakepit [2] https://hex.pm/packages/snakebridge

Re: Python numbers every programmer should know

#80

Post by willseth »

oivey wrote: post #62
willseth wrote: post #42

Earlier quoted context omitted.

Knowing that an empty string is 41 bytes or how many ns it takes to do arithmetic operations is not general knowledge.

How is it not general knowledge? How do you otherwise gauge if your program is taking a reasonable amount of time, and, if not, how do you figure out how to fix it?

You gauge with metrics and profiles, if necessary, and address as needed. You don’t scrutinize every line of code over whether it’s “reasonable” in advance instead of doing things that actually move the needle.
Post reply on HN