321–326 of 326 posts
Earlier quoted context omitted.
Tried a memmap?
I still wish to find a good tutorial about memmap. The doc about it is very formal. Something with clear use cases, patterns, gotchas and best practice would probably make it more popular.
Earlier quoted context omitted.
If you use master instead of the current release that example goes from about two mins to just under one minute for my machine. Also, if you use .subst(‘y’, ’n’) instead of a regex it runs in under 9 seconds locally. Thats still much slower than perl 5 (which locally takes less than half a second) but they’re making great strides at improving performance.
So, doing another simple change to my code brought the runtime down to a few ms. Doing: time yes | head -n1000000 | perl6 --profile -e 'for $*IN.readchars { .subst("y", "n").print }’ > /dev/null says it took 86 ms. Which is pretty decent I’d say.
time echo "#include
main(n){char*b,*s=n=0;getdelim(&s,&n,-1,stdin);for(b=s;*s;++s)*s=='y'&&*s='n';puts(b);}">s.c|yes|head -n1000000|tcc -w -run s.c>/dev/null
real 0m0.028s
user 0m0.023s
sys 0m0.019s
Earlier quoted context omitted.
So, doing another simple change to my code brought the runtime down to a few ms. Doing: time yes | head -n1000000 | perl6 --profile -e 'for $*IN.readchars { .subst("y", "n").print }’ > /dev/null says it took 86 ms. Which is pretty decent I’d say.
Sure, but that's still kinda defeating the purpose of providing functionality that's easy to remember and type. Might as well do time echo "#include main(n){char*b,*s=n=0;getdelim(&s,&n,-1,stdin);for(b=s;*s;++s)*s=='y'&&*s='n';puts(b);}">s.c|yes|head -n1000000|tcc -w -run s.c>/dev/null real 0m0.028s user 0m0.023s sys 0m0.019s 28 milliseconds!
Hopefully more edge cases will be discovered and fixed as the implementation matures as well.
Earlier quoted context omitted.
Could you post the code for that fib program I couldn't find it anywhere.
It is in the Gilectomy branch of Larry Hastings github project. ( https://github.com/larryhastings/gilectomy ) I also pasted the fib test to pastebin: https://pastebin.com/Ryyb2K7V
Interestingly doing the same on Cpython using the multiprocessing module was ~2x slower than jython/threads. More interestingly pypy with multiprocessing was ~5x faster than jython/threads.
$ time jython fib.py 40
real 1m11.247s
user 6m14.130s
sys 0m3.012s
$ time python fib.py 40
real 2m4.067s
user 11m46.103s
sys 0m2.352s
$ time pypy fib.py 40
real 0m21.040s
user 1m51.461s
sys 0m1.892sEarlier quoted context omitted.
> Multiprocess parallelism isn't always appropriate Using Python isn't always appropriate.
Are you saying that because a language is missing something, when considering a fix for that thing, the existence of other languages/solutions is an argument against that fix?
Earlier quoted context omitted.
This looks very interesting. I am reading the docs https://pythonhosted.org/joblib/parallel.html#manual-managem... and it looks like it would help a lot (possibly solve the issue). Do you have any experience using this in production?
DAMN. I just did a basic test and it kinnda just worked?!? I created a test dataframe of 100M rows X 10 cols which took up ~2.3G and then used joblib.dump within the on_starting hook which is run when the gunicorn master starts up. Then loaded that df in with joblib.load within the worker and the total memory consumption was practically flat. Then I bumped up the number of workers to 20 and still flat. That is actual…