Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

Post reply on HN
Print view View original

101–110 of 336 posts

Re: Python’s Weak Performance Matters

#101

Post by shele »

loeg wrote: post #74
banku_brougham wrote: post #46

I’d like to thank the author for sharing a very practical view of problem solving in the data science space. Can I suggest julia? Its very easy to understand coming from python, and performant code can be had usually in easy to read implementation of the expressions in whatever paper you are basing your work upon.

Anytime someone brings up Julia, I think of Dan Luu's review of the language: https://danluu.com/julialang/

The way Dan Luu's post is used on HN resembles a thought terminating cliche.

Re: Python’s Weak Performance Matters

#102

Post by narimiran »

If you would like 10-100x faster performance than Python, but would like to keep the easy-to-read code, give Nim [0] a try.

I do all my work in Python, and I've been using Nim in last couple of months - it took me a week or two until I was able to be productive in Nim.

Don't expect Python's large ecosystem, nor some Python goodies, but if you're looking for a readable, writable, high-performance post-Python language - Nim is the way to go!

[0] https://nim-lang.org/

Re: Python’s Weak Performance Matters

#103

Post by fxmc »

luispedrocoelho wrote: post #64
pletnes wrote: post #52

Earlier quoted context omitted.

That’s easy with python, too, in a lot of number crunching cases. Numpy with MKL will use all your cores, as will e.g dask and other libraries built on numpy. Farming out embarassingly parallel work to threads or processes is also easy.

If I can fit the code into numpy-like structure, then Python is typically fine. The issue is when I cannot.

Have you given dask a try? It gives you out-of-core arrays with numpy semantics and distributed computing.

Re: Python’s Weak Performance Matters

#104

Post by PyComfy »

cturner wrote: post #24

Where are the main blowouts in python performance? For example, is it compilation, evaluation overhead, or memory management?

Naive interpretation of the bytecode (not even pre-decoded, just a switch statement). And almost everything is resolved in the dynamic environment. for example,

    a = foo.bar(b)
is actually

    ldict = locals()

    ldict['a'] = ldict['foo'].__getattribute__('bar')(ldict['b'])

Re: Python’s Weak Performance Matters

#105

Post by kalefranz »

luispedrocoelho wrote: post #64
pletnes wrote: post #52

Earlier quoted context omitted.

That’s easy with python, too, in a lot of number crunching cases. Numpy with MKL will use all your cores, as will e.g dask and other libraries built on numpy. Farming out embarassingly parallel work to threads or processes is also easy.

If I can fit the code into numpy-like structure, then Python is typically fine. The issue is when I cannot.

Also look into numba as a jit decorator for python functions.

Re: Python’s Weak Performance Matters

#106

Post by ptype »

I love Python. The language is a joy, the eco system is fantastic. But yes, let’s be honest, if you can not vectorise your code it is slow, and I think that will be its downfall eventually.

I’m excited about Julia, I hope it gains popularity and the eco system grows. Until then, and in particular until the data frames story can compete with pandas, it Python with Cython for me, but I’d rather skip the Cython if it was not necessary for performance.

Any early adopters running Julia in production with stories to share?

Re: Python’s Weak Performance Matters

#107

Post by eesmith »

indubitable wrote: post #87
eesmith wrote: post #61

Earlier quoted context omitted.

I know little on this topic, but as you were looking for some feedback I thought I would response. Because the Microsoft version was closed source for a long time? It was released in 2000. You give 2014 as the open source date, which was only 4 years ago. Woolvalley was referring to the implementation, so bringing up the specification is not that relevant. The ECMA specification does (did?) not include ASP.NET, ADO.N…

Much appreciated! One clarification, since my own language was sloppy. C# did not get open sourced in 2014. That was the date that Microsoft open sourced just about everything (and that's a weasel word there - to my knowledge, it is everything) that wasn't already open source. Things like ADO/ASP/Windows Forms/etc were open sourced back in 2008, along with Microsoft's implementation of their framework libraries, whic…

You wrote "Things like ADO/ASP/Windows Forms/etc were open sourced back in 2008".

Could you explain that further? It appears to contradict this ZDNet article from 2014, http://www.zdnet.com/article/microsoft-to-open-source-more-o... , which says:

> Microsoft to open source more of .NET, and bring it to Linux, Mac OS X

> Microsoft is porting its server-side .NET stack to Linux and Mac OS X, and is making more of that stack available as open source. ...

> In April 2014, Microsoft announced plans to open source a number of its developer technologies , including ASP.NET, the Roslyn .NET compiler platform, the .NET Micro Framework, .NET Rx and the VB and C# programming languages. ...

> Microsoft is not planning to open source the client side .NET stack, which means certain pieces like the Windows Presentation Foundation (WPF) and Windows Forms won't be going open source,

Here's another link from 2014, this describing how .NET Core was made open source. https://blogs.msdn.microsoft.com/dotnet/2014/11/12/net-core-... . It says:

> As a .NET developer you were able to build & run code on more than just Windows for a while now, including Linux, MacOS, iOs and Android.

> The challenge is that the Windows implementation has one code base while Mono has a completely separate code base. The Mono community was essentially forced to re-implement .NET because no open source implementation was available. Sure, the source code was available since Rotor but we didn’t use an OSI approved open source license, which made Rotor a non-starter. Customers have reported various mismatches, which are hard to fix because neither side can look at the code of the other side. This also results in a lot of duplicated work in areas that aren’t actually platform specific.

This would seem to contradict your statement that "Things like ADO/ASP/Windows Forms/etc were open sourced back in 2008, along with Microsoft's implementation of their framework libraries, which Mono rapidly integrated into their implementation."

Here's another page, describing the difficulties before .NET Core, http://www.c-sharpcorner.com/article/why-is-dot-net-core-imp... :

> Previously, 'cross-platform' with Microsoft was a joke - it was cross-platform but only within the Microsoft Windows operating system family. .NET Core brings the true cross-platform compatibility, which means you can have one single source code base on Windows, Mac, and Linux. This is a huge deal, especially between Windows and Linux - it gives you more choice for deployment, hosting, and scaling.

> By making its fundamental codebase open source, Microsoft is giving .NET developers an incredible opportunity to enter into areas with their existing skills which were previously locked off to them. The opportunities presented are only going to start to emerge over the next months and years - it's well worth your while checking it out and taking .NET Core for a spin.

Now, certainly it's possible that you are correct, and these are all parts of the misinformation derived from the early days of C# and .Net.

If so, could you provide some references? Otherwise it's very easy for me to conclude that you misremember the historical details.

Re: Python’s Weak Performance Matters

#108

Post by bjoli »

willtim wrote: post #35
matt_wulfeck wrote: post #2

> The result is that I find myself doing more and more things in Haskell, which lets me write high-level code with decent performance (still slower than what I get if I go all the way down to C++, but with very good libraries). This strikes me as an odd conclusion to come to if speed was the main motivator.

GHC Haskell is an advanced optimising compiler, which can get very near the speed of C and C++. However, to write fast programs, one must use the right data structures and algorithms. Often this means array-based strings and streaming IO, unfortunately many Haskell textbooks don't tend to cover this.

All comparisons of Haskell and C I have seen where Haskell has been near the performance of C(++) has always compared a highly optimised Haskell program to a moderately fast c program.

I have found that most managed languages generally come within 2-5 times slower than C and C++. Which is good enough for me.

Re: Python’s Weak Performance Matters

#109

Post by peterwaller »

aldanor wrote: post #85
luispedrocoelho wrote: post #36

Earlier quoted context omitted.

OP here. Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode. Python has the lowest TimeToWriteCode, but very high TimeToRunCode. C++ has lowest TimeToRunCode, but high TimeTowWriteCode. Haskell is often a good compromise for me. Also, with Haskell, it can be very easy to take advantage of 20 CPU cores, while I don't have as much familiarity with high-level C++ threading libraries.

@ the OP - not to sound hostile, but you write code (like in the example here [1]) that is bound to be slow, just from a glance at it. vstacking, munging with pandas indices (and pandas in general), etc; in order for it to be fast, you want pure numpy, with as little allocations happening as possible. I help my coworkers “make things faster” with snippets like this all the time. If you provide me with a self-containe…

> you write code (like in the example here [1]) that is bound to be slow, just from a glance at it > [1] https://git.embl.de/costea/metaSNV/blob/master/metaSNV_post.....

Given his code you referenced, could you elaborate on what makes it look slow at a glance, and how you might speed it up? :)

Re: Python’s Weak Performance Matters

#110

Post by cromat3 »

shakna wrote: post #19
ChrisSD wrote: post #11

I don't think it's true to say that Python's core developers are uninterested in performance. Speeding up Python is a hard problem. He mentions PyPy but even that has only managed modest performance gains in some areas (and not without tradeoffs). He suggests JavaScript as a comparison but doesn't elaborate on how they're comparable beyond the superficial (they're both dynamic scripting languages). I get that he's fr…

Viper is an interesting approach on speeding up Python. It's developed for MicroPython, which does give them room for breaking changes, but has trade-offs. Arithmetic is much faster, but dictionary lookups take much longer compared to CPython. Viper is a code-emitter from a large subset of Python, and even allows for inline assembly. But it's only for a few architectures at the moment, like ARM and x86.

Viper is called Zerynth today: https://wiki.python.org/moin/Zerynth
Post reply on HN