Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python numbers every programmer should know

mkennedy.codes

Post reply on HN
Print view View original

81–90 of 191 posts

Re: Python numbers every programmer should know

#81

Post by dekhn »

its-summertime wrote: post #69
Quothling wrote: post #41

Earlier quoted context omitted.

Why? I've build some massive analytic data flows in Python with turbodbc + pandas which are basically C++ fast. It uses more memory which supports your point, but on the flip-side we're talking $5-10 extra cost a year. It could frankly be $20k a year and still be cheaper than staffing more people like me to maintain these things, rather than having a couple of us and then letting the BI people use the tools we provid…

From the complete opposite side, I've built some tiny bits of near irrelevant code where python has been unacceptable, e.g. in shell startup / in bash's PROMPT_COMMAND, etc. It ends up having a very painfully obvious startup time, even if the code is nearing the equivalent of Hello World time python -I -c 'print("Hello World")' real 0m0.014s time bash --noprofile -c 'echo "Hello World"' real 0m0.001s

What exactly do you need 1ms instead of 14ms startup time in a shell startup? The difference is barely perceptible.

Most of the time starting up is time spent seartching the filesystem for thousands of packages.

Re: Python numbers every programmer should know

#82

Post by oivey »

willseth wrote: post #80
oivey wrote: post #62

Earlier quoted context omitted.

How is it not general knowledge? How do you otherwise gauge if your program is taking a reasonable amount of time, and, if not, how do you figure out how to fix it?

You gauge with metrics and profiles, if necessary, and address as needed. You don’t scrutinize every line of code over whether it’s “reasonable” in advance instead of doing things that actually move the needle.

These are the metrics underneath it all. Profiles tell you what parts are slow relative to others and time your specific implementation. How long should it take to sum together a million integers?

Re: Python numbers every programmer should know

#83

Post by dekhn »

Aurornis wrote: post #24

A meta-note on the title since it looks like it’s confusing a lot of commenters: The title is a play on Jeff Dean’s famous “Latency Numbers Every Programmer Should Know” from 2012. It isn’t meant to be interpreted literally. There’s a common theme in CS papers and writing to write titles that play upon themes from past papers. Another common example is the “_____ considered harmful” titles.

That doc predates 2012 significantly.

From what I've been able to glean, it was basically created in the first few years Jeff worked at Google, on indexing and serving for the original search engine. For example, the comparison of cache, RAM, and disk: determined whether data was stored in RAM (the index, used for retrieval) or disk (the documents, typically not used in retrieval, but used in scoring). Similarly, the comparison of California-Netherlands time- I believe Google's first international data cetner was in NL and they needed to make decisions about copying over the entire index in bulk versus serving backend queries in the US with frontends in the NL.

The numbers were always going out of date; for example, the arrival of flash drives changed disk latency significantly. I remember Jeff came to me one day and said he'd invented a compression algorithm for genomic data "so it can be served from flash" (he thought it would be wasteful to use precious flash space on uncompressed genomic data).

Re: Python numbers every programmer should know

#84

Post by cycomanic »

oivey wrote: post #62
willseth wrote: post #42

Earlier quoted context omitted.

Knowing that an empty string is 41 bytes or how many ns it takes to do arithmetic operations is not general knowledge.

How is it not general knowledge? How do you otherwise gauge if your program is taking a reasonable amount of time, and, if not, how do you figure out how to fix it?

But these performance numbers are meaningless without some sort of standard comparison case. So if you measure that e.g. some string operation takes 100ns, how do you compare against the numbers given here? Any difference could be due to PC, python version or your implementation. So you have to do proper benchmarking anyway.

Re: Python numbers every programmer should know

#85

Post by sjducb »

It’s missing the time taken to instantiate a class.

I remember refactoring some code to improve readability, then observing something that was previously a few microseconds take tens of seconds.

The original code created a large list of lists. Each child list had 4 fields each field was a different thing, some were ints and one was a string.

I created a new class with the names of each field and helper methods to process the data. The new code created a list of instances of my class. Downstream consumers of the list could look at the class to see what data they were getting. Modern Python developers would use a data class for this.

The new code was very slow. I’d love it if the author measured the time taken to instantiate a class.

Re: Python numbers every programmer should know

#86

Post by lifeisstillgood »

sjducb wrote: post #85

It’s missing the time taken to instantiate a class. I remember refactoring some code to improve readability, then observing something that was previously a few microseconds take tens of seconds. The original code created a large list of lists. Each child list had 4 fields each field was a different thing, some were ints and one was a string. I created a new class with the names of each field and helper methods to pro…

I went to the doctor and I said “It hurts when I do this”

The doctor said, “don’t do that”.

Edit: so yeah a rather snarky reply. Sorry. But it’s worth asking why we want to use classes and objects everywhere. Alan Kay is well known for saying object orientated is about message passing (mostly by Erlang people).

A list of lists (where each list is four different types repeated) seems a fine data structure, which can be operated on by external functions, and serialised pretty easily. Turning it into classes and objects might not be a useful refactoring, I would certainly want to learn more before giving the go ahead.

Re: Python numbers every programmer should know

#87

Post by TuringTest »

AtlasBarfed wrote: post #75
Demiurge wrote: post #39

Earlier quoted context omitted.

I agree. I've been living off Python for 20 years and have never needed to know any of these numbers, nor do I need them now, for my work, contrary to the title. I also regularly use profiling for performance optimization and opt for Cython, SWIG, JIT libraries, or other tools as needed. None of these numbers would ever factor into my decision-making.

..... You don't see any value in knowing that numbers?

As others have pointed out, Python is better used in places where those numbers aren't relevant.

If they start becoming relevant, it's usually a sign that you're using the language in a domain where a duck-typed bytecode scripting-glue language is not well-suited.

Re: Python numbers every programmer should know

#88

Post by willseth »

oivey wrote: post #82
willseth wrote: post #80

Earlier quoted context omitted.

You gauge with metrics and profiles, if necessary, and address as needed. You don’t scrutinize every line of code over whether it’s “reasonable” in advance instead of doing things that actually move the needle.

These are the metrics underneath it all. Profiles tell you what parts are slow relative to others and time your specific implementation. How long should it take to sum together a million integers?

It literally doesn’t matter unless it impacts users. I don’t know why you would waste time on non problems.

Re: Python numbers every programmer should know

#89

Post by NekkoDroid »

dekhn wrote: post #81
its-summertime wrote: post #69

Earlier quoted context omitted.

From the complete opposite side, I've built some tiny bits of near irrelevant code where python has been unacceptable, e.g. in shell startup / in bash's PROMPT_COMMAND, etc. It ends up having a very painfully obvious startup time, even if the code is nearing the equivalent of Hello World time python -I -c 'print("Hello World")' real 0m0.014s time bash --noprofile -c 'echo "Hello World"' real 0m0.001s

What exactly do you need 1ms instead of 14ms startup time in a shell startup? The difference is barely perceptible. Most of the time starting up is time spent seartching the filesystem for thousands of packages.

> What exactly do you need 1ms instead of 14ms startup time in a shell startup?

I think as they said: when dynamically building a shell input prompt it starts to become very noticable if you have like 3 or more of these and you use the terminal a lot.

Re: Python numbers every programmer should know

#90

Post by calmbonsai »

You absolutely do not need to know those absolute numbers--only the relative costs of various operations.

Additionally, regardless of the code you can profile the system to determine where the "hot spots" are and refactor or call-out to more performant (Rust, Go, C) run-times for those workflows where necessary.

Post reply on HN