What would be the explanation for an int taking 28 bytes but a list of 1000 ints taking only 7.87KB?
That appears to be the size of the list itself, not including the objects it contains: 8 bytes per entry for the object pointer, and a kilo-to-kibi conversion. All Python values are "boxed", which is probably a more important thing for a Python programmer to know than most of these numbers. The list of floats is larger, despite also being simply an array of 1000 8-byte pointers. I assume that it's because the int arr…
It was. I updated the results to include the contained elements. I also updated the float list creation to match the int list creation.
A lot of people here are commenting that if you have to care about specific latency numbers in Python you should just use another language. I disagree. A lot of important and large codebases were grown and maintained in Python (Instagram, Dropbox, OpenAI) and it's damn useful to know how to reason your way out of a Python performance problem when you inevitably hit one without dropping out into another language, whic…
> A lot of important and large codebases were grown and maintained in Python
How does this happen? Is it just inertia that cause people to write large systems in a essentially type free, interpreted scripting language?
A lot of people here are commenting that if you have to care about specific latency numbers in Python you should just use another language. I disagree. A lot of important and large codebases were grown and maintained in Python (Instagram, Dropbox, OpenAI) and it's damn useful to know how to reason your way out of a Python performance problem when you inevitably hit one without dropping out into another language, whic…
I think both points are fair. Python is slow - you should avoid it if speed is critical, but sometimes you can’t easily avoid it.
I think the list itself is super long winded and not very informative. A lot of operations take about the same amount of time. Does it matter that adding two ints is very slightly slower than adding two floats? (If you even believe this is true, which I don’t.) No. A better summary would say “all of these things take about the same amount of time: simple math, function calls, etc. these things are much slower: IO.” And in that form the summary is pretty obvious.
A lot of people here are commenting that if you have to care about specific latency numbers in Python you should just use another language. I disagree. A lot of important and large codebases were grown and maintained in Python (Instagram, Dropbox, OpenAI) and it's damn useful to know how to reason your way out of a Python performance problem when you inevitably hit one without dropping out into another language, whic…
> A lot of important and large codebases were grown and maintained in Python How does this happen? Is it just inertia that cause people to write large systems in a essentially type free, interpreted scripting language?
It’s very natural. Python is fantastic for going from 0 to 1 because it’s easy and forgiving. So lots of projects start with it. Especially anything ML focused. And it’s much harder to change tools once a project is underway.
A lot of people here are commenting that if you have to care about specific latency numbers in Python you should just use another language. I disagree. A lot of important and large codebases were grown and maintained in Python (Instagram, Dropbox, OpenAI) and it's damn useful to know how to reason your way out of a Python performance problem when you inevitably hit one without dropping out into another language, whic…
> A lot of important and large codebases were grown and maintained in Python How does this happen? Is it just inertia that cause people to write large systems in a essentially type free, interpreted scripting language?
It's very simple. Large systems start as small systems.
A lot of people here are commenting that if you have to care about specific latency numbers in Python you should just use another language. I disagree. A lot of important and large codebases were grown and maintained in Python (Instagram, Dropbox, OpenAI) and it's damn useful to know how to reason your way out of a Python performance problem when you inevitably hit one without dropping out into another language, whic…
> A lot of important and large codebases were grown and maintained in Python How does this happen? Is it just inertia that cause people to write large systems in a essentially type free, interpreted scripting language?
It’s a nice and productive language. Why is that incomprehensible?
Why? I've build some massive analytic data flows in Python with turbodbc + pandas which are basically C++ fast. It uses more memory which supports your point, but on the flip-side we're talking $5-10 extra cost a year. It could frankly be $20k a year and still be cheaper than staffing more people like me to maintain these things, rather than having a couple of us and then letting the BI people use the tools we provid…
> you need to know these numbers on Python to know when to actually build something in C People usually approach this the other way, use something like pandas or numpy from the beginning if it solves your problem. Do not write matrix multiplications or joins in python at all. If there is no library that solves your problem, it's a great indication that you should avoid python. Unless you are willing to spend 5 man-ye…
People generally aren’t rolling their own matmuls or joins or whatever in production code. There are tons of tools like Numba, Jax, Triton, etc that you can use to write very fast code for new, novel, and unsolved problems. The idea that “if you need fast code, don’t write Python” has been totally obsolete for over a decade.
What would be the explanation for an int taking 28 bytes but a list of 1000 ints taking only 7.87KB?
That appears to be the size of the list itself, not including the objects it contains: 8 bytes per entry for the object pointer, and a kilo-to-kibi conversion. All Python values are "boxed", which is probably a more important thing for a Python programmer to know than most of these numbers. The list of floats is larger, despite also being simply an array of 1000 8-byte pointers. I assume that it's because the int arr…
That's impressive how you figured out the reason for the difference in list of floats vs list of ints container size, framed as an interview question that would have been quite difficult I think