Python Serialization Performance
kurtsp.com
Python Serialization Performance
1–10 of 20 posts
Re: Python Serialization Performance
#2TL;DR: Things get faster if you disable validation and parse dates using more specialized code. Duh.
It's humorous when someone who presumably cares about performance tells you they use Python. Python is a wonderful language, but performance is not what it is designed for. Basically _anything_ that does not require an interpreter to run will be 10-30x faster on the same hardware, and most will also consume less RAM and be able to use more than one core on the system efficiently. It used to be that Python's lack of performance didn't matter because disks and networks were so slow things were IO bound. In more and more cases that's just not true anymore. You could be easily reading at 1GB+/sec and pushing 10-20Gbps to NICs, depending on the hardware.
Re: Python Serialization Performance
#3Seems like a good use case for protocol buffers [1].
Re: Python Serialization Performance
#4TL;DR: Things get faster if you disable validation and parse dates using more specialized code. Duh. It's humorous when someone who presumably cares about performance tells you they use Python. Python is a wonderful language, but performance is not what it is designed for. Basically _anything_ that does not require an interpreter to run will be 10-30x faster on the same hardware, and most will also consume less RAM a…
It depends what "using python" means though. Cython is pretty good at optimizing basic code. Numpy will process your matrices and vectors using specialised libraries faster than most manual C approaches. Shedskin will give you a nice code framework which you can optimise in parts that matter. (insert other specialised examples)
CPython is slow as an interpreter, true. "Programming in Python" may or may not be many times slower than compiling the comparable code in other language. Depends what you're doing and how you're doing it.
Also, I care about performance in any language to some extent. If I can write a backup bash script that takes 2h, or write one that takes 20min, I do care about performance and will choose the second one. Why shouldn't I?
Re: Python Serialization Performance
#5> Disable Schematics Validation ... There's a cool Python trick to create an object while skipping the __init__ function.
Yes, but now every single time you update the library which provided the base class, you need to re-verify that __init__ doesn't do anything new. May be worth the tradeoff, but it really should be noted.
Re: Python Serialization Performance
#6TL;DR: Things get faster if you disable validation and parse dates using more specialized code. Duh. It's humorous when someone who presumably cares about performance tells you they use Python. Python is a wonderful language, but performance is not what it is designed for. Basically _anything_ that does not require an interpreter to run will be 10-30x faster on the same hardware, and most will also consume less RAM a…
It depends what "using python" means though. Cython is pretty good at optimizing basic code. Numpy will process your matrices and vectors using specialised libraries faster than most manual C approaches. Shedskin will give you a nice code framework which you can optimise in parts that matter. (insert other specialised examples) CPython is slow as an interpreter, true. "Programming in Python" may or may not be many ti…
Is Shedskin still actively maintained? I haven't looked in a long time but last I saw it hadn't gotten any updates in a couple years (maybe I have corrupted memory though ;)
Anyhow, the relatively new Nuitka project seems to be aiming to tackle the python-to-c++ compiler problem, and seems to have a lot of promise. Really good compatibility, apparently decent speedups, and cross platform support. Works into Python3 too. I have a lot of hope!
Re: Python Serialization Performance
#7Seems like a good use case for protocol buffers [1]. [1] https://developers.google.com/protocol-buffers/?hl=en
It might be that they consider it important to be able to interrogate the in-memory or on-disk representations without the help of a decoding step. Protobufs are great for getting objects into a nice compact format to throw on the wire, but god help you if you end up with multiple actors filling up a queue with inconsistently encoded objects.
Re: Python Serialization Performance
#8TL;DR: Things get faster if you disable validation and parse dates using more specialized code. Duh. It's humorous when someone who presumably cares about performance tells you they use Python. Python is a wonderful language, but performance is not what it is designed for. Basically _anything_ that does not require an interpreter to run will be 10-30x faster on the same hardware, and most will also consume less RAM a…
It depends what "using python" means though. Cython is pretty good at optimizing basic code. Numpy will process your matrices and vectors using specialised libraries faster than most manual C approaches. Shedskin will give you a nice code framework which you can optimise in parts that matter. (insert other specialised examples) CPython is slow as an interpreter, true. "Programming in Python" may or may not be many ti…
Even Cython will be several times slower than a carefully tuned C/C++, Java, C# or Go program for most practical problems. And at least in the case of C/C++ it'll also likely use several times more RAM. Now for a company like Uber it may not matter if something is 3x slower and uses 3x the RAM, just throw more hardware at the problem, but if you're going to introduce typing into Python, you might as well go with a language where typing is not optional, and which, more importantly, has been used in production by thousands of teams over the past decade or more. Of the languages I listed, only C++ is not really viable for most programmers.
As to caring about perf, you shouldn't care about it until you have to. Take that 2h vs 20min example, for instance. If you only need to run it a few times and there's plenty of time available, who cares how long it takes. If the 2h one is easier to write that's by all means what you should do. OTOH if you're under severe time constraints and need to run it every hour, then obviously 2h script won't do the job. Or alternatively if 20 min script takes the same time to write as 2h one, then of course you should go with it. All too often I see people optimizing things that don't matter one iota, simply because they like things to be fast. Something gets executed once a day and runs for 5 minutes? Let's spend two weeks making it complete in 30 seconds. As long as the employer is paying, why not.
Re: Python Serialization Performance
#9Earlier quoted context omitted.
It depends what "using python" means though. Cython is pretty good at optimizing basic code. Numpy will process your matrices and vectors using specialised libraries faster than most manual C approaches. Shedskin will give you a nice code framework which you can optimise in parts that matter. (insert other specialised examples) CPython is slow as an interpreter, true. "Programming in Python" may or may not be many ti…
Even Cython will be several times slower than a carefully tuned C/C++, Java, C# or Go program for most practical problems. And at least in the case of C/C++ it'll also likely use several times more RAM. Now for a company like Uber it may not matter if something is 3x slower and uses 3x the RAM, just throw more hardware at the problem, but if you're going to introduce typing into Python, you might as well go with a la…
Cython is the same speed as a carefully tuned C/C++ program; carefully tuned, Cython maps directly to C/C++. I've done many benchmarks of low level numerical code when implementing SageMath (which uses Cython for several 100K lines of code).
Re: Python Serialization Performance
#10TL;DR: Things get faster if you disable validation and parse dates using more specialized code. Duh. It's humorous when someone who presumably cares about performance tells you they use Python. Python is a wonderful language, but performance is not what it is designed for. Basically _anything_ that does not require an interpreter to run will be 10-30x faster on the same hardware, and most will also consume less RAM a…
> You could be easily reading at 1GB+/sec and pushing 10-20Gbps to NICs, depending on the hardware.
Python is easily able to push this much data but I have the impression that the problem are the performance-hogging libraries. In my case I had to write my own HTTP client implementation for Python to get such speeds. Python is not the problem you just need to avoid unnecessary LoCs.