Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Still Use Python for High Performance Scientific Computing

nbviewer.jupyter.org

Post reply on HN
Print view View original

151–158 of 158 posts

Re: Why I Still Use Python for High Performance Scientific Computing

#151

Post by makeset »

baldfat wrote: post #145
makeset wrote: post #129

Earlier quoted context omitted.

Yeah, lots of squinting needed, or a terribly shallow understanding of FP to say that. Merely providing first-class functions and map/reduce idioms does not a functional language make. Python has those, so is it also FP really? R is as much a FP language as Haskell or OCaml are imperative "at heart."

> "terribly shallow understanding of FP to say that" That was very condescending and I am sure don't know who Hadley Wickham is? I wouldn't call his understanding "Shallow."

No, I meant that he was squinting. What is condescending is you making assumptions about whom I don't know and what I'm unfamiliar with.

Re: Why I Still Use Python for High Performance Scientific Computing

#152

Post by codygman »

KirinDave wrote: post #146
codygman wrote: post #137

Earlier quoted context omitted.

> it's basically a few really awesome modern features in a massive graveyard of failed academic initiatives that are now enshrined in the lore of the language because one or two useful libraries used them. Err... can you qualify this? Based on my knowledge of Haskell this isn't a fair assessment at all.

Based on my knowledge, it's very fair. How many language extensions are available vs placed in common use? Where is the guidance on which to use? You WILL use at least 3 I can think of off hand very commonly. And how many failed lens libraries preceeded the current (quite good) dominant library that still show up on google and even get included in? How many people still run versions of Yesod using now-unfavored abstr…

>

Alright, I'll give you that there are a ton of GHC extensions after getting a list of them, all 106 as of GHC 7.10[0].

This does make me wonder why having so many extensions was never an issue for me.

> The Haskell community really got its shit together over the last 3 years. Cabal got a lot better, a bunch of key libraries got good, our compiler is fixed some major bugs

Agreed.

> I love Haskell, I really do. I wish I could ship more code with Haskell, and like it as well. But the OP asked about why Haskell didn't take off. Its ecosystem was in what we might call a state of growth and flux at the time a lot of stream and batch computation systems were shipping.

Fair enough.

> Oh and uh, I think Haskell's community is full of some truly toxic people, an intersection of some of the ugliest and most arrogant personalities of the Scala world who's primary education strategy can only be called "negging."

I've seen this personality type in other languages, but not Haskell yet. I too abhor the "negging" type, as well as the subculture that term arose from.

0: all ghc extensions

    OverlappingInstances	
    UndecidableInstances	
    IncoherentInstances	
    DoRec	
    RecursiveDo	
    ParallelListComp	
    MultiParamTypeClasses	
    MonomorphismRestriction	
    FunctionalDependencies	
    Rank2Types	
    RankNTypes	
    PolymorphicComponents	
    ExistentialQuantification	
    ScopedTypeVariables	
    PatternSignatures	
    ImplicitParams	
    FlexibleContexts	
    FlexibleInstances	
    EmptyDataDecls	
    CPP	
    KindSignatures	
    BangPatterns	
    TypeSynonymInstances	
    TemplateHaskell	
    ForeignFunctionInterface	
    Arrows	
    Generics	
    ImplicitPrelude	
    NamedFieldPuns	
    PatternGuards	
    GeneralizedNewtypeDeriving	
    ExtensibleRecords	
    RestrictedTypeSynonyms	
    HereDocuments	
    MagicHash	
    TypeFamilies	
    StandaloneDeriving	
    UnicodeSyntax	
    UnliftedFFITypes	
    InterruptibleFFI	
    CApiFFI	
    LiberalTypeSynonyms	
    TypeOperators	
    RecordWildCards	
    RecordPuns	 (Deprecated, use NamedFieldPuns instead)
    DisambiguateRecordFields	
    TraditionalRecordSyntax	
    OverloadedStrings	
    GADTs	
    GADTSyntax	
    MonoPatBinds	
    RelaxedPolyRec	
    ExtendedDefaultRules	
    UnboxedTuples	
    DeriveDataTypeable	
    DeriveGeneric	
    DefaultSignatures	
    InstanceSigs	
    ConstrainedClassMethods	
    PackageImports	
    ImpredicativeTypes	(deprecated)
    NewQualifiedOperators (deprecated)
    PostfixOperators	
    QuasiQuotes	
    TransformListComp	
    MonadComprehensions	
    ViewPatterns	
    XmlSyntax	
    RegularPatterns	
    TupleSections	
    GHCForeignImportPrim	
    NPlusKPatterns	
    DoAndIfThenElse	
    MultiWayIf	
    LambdaCase	
    RebindableSyntax	
    ExplicitForAll	
    DatatypeContexts	
    MonoLocalBinds	
    DeriveFunctor	
    DeriveTraversable	
    DeriveFoldable	
    NondecreasingIndentation	
    SafeImports	
    Safe	
    Trustworthy	
    Unsafe	
    ConstraintKinds	
    PolyKinds	
    DataKinds	
    ParallelArrays	
    RoleAnnotations	
    OverloadedLists	
    EmptyCase	
    AutoDeriveTypeable	
    NegativeLiterals	
    BinaryLiterals	
    NumDecimals	
    NullaryTypeClasses	
    ExplicitNamespaces	
    AllowAmbiguousTypes	
    JavaScriptFFI	
    PatternSynonyms	
    PartialTypeSignatures	
    NamedWildCards	
    DeriveAnyClass

Re: Why I Still Use Python for High Performance Scientific Computing

#153

Post by lmm »

msellout wrote: post #141
lmm wrote: post #48

But you can have both. In Scala I can write prototypes just as rapidly as Python, but I can run them with close-to-native performance. I can even explore interactively in a REPL but backed by the power of my company's big computer cluster, using spark-shell. The profiling capabilities are excellent, but when I spot a bottleneck I can solve it in the language directly, without needing the awkwardness of cython or of c…

> But you can have both. It seems you misread the article as saying Python created a trade-off. Perhaps the author shouldn't have tried to create a surprise ending. Regardless, the conclusion was that the Python implementation was both easier to code and more performant than the Java implementation.

From the article: "Easy development comes at a cost of course. That initial experimental implementation was terribly slow, taking thirty seconds or more to cluster only a few thousand points." Even if the final implementation in mostly-Python is fast, having a slow feedback cycle during early development can be a huge cost.

Re: Why I Still Use Python for High Performance Scientific Computing

#154

Post by lmm »

wesm wrote: post #134
lmm wrote: post #49

Earlier quoted context omitted.

Then what was "C is just a wrapper for assembly, then, right?" supposed to mean?

Drawing a parallel!

What's the parallel? Python wrapping a C library is very different from C compiling down to assembly (which is often not even how C is compiled).

Re: Why I Still Use Python for High Performance Scientific Computing

#155

Post by lmm »

srean wrote: post #135
bipin_nag wrote: post #132

Earlier quoted context omitted.

Correct me if I am wrong but Spark does not use JNI for Python. It uses py4j to allow Python programs to access Java Objects. All the core functionality is implemented in Scala/Java. py4j uses sockets to communicate. See https://www.py4j.org/about.html

That's my understanding as well. This is the reason why you are caught between the rock and a hard place on the Hadoop stack. Java code wont be efficient in terms of FLOPs and a popular and often effective escape hatch: code number crunching parts in C, C++ or Fortran, is also not very effective / convenient. Particularly so if it requires going back and forth over the bridge frequently because crossing the bridge ha…

When we say JVM code is not efficient in terms of FLOPs we're talking about a factor of 2 or so though, not really the same as the factor of 20+ you lose by implementing numeric code in pure Python. So Scala gives you a 10x faster cycle during early development. Maybe it's fast enough that you don't need that last factor of 2, or maybe just JNIing a few core operations is enough (you can also do some tricks with the advanced type system; with NumPy if it looks like Python iteration then it is, whereas with something like Breeze you can potentially have something that looks like Scala iteration but won't actually copy the data back and forth). But in the cases where you do need to push things right down into Fortran, Scala is just as good a glue language as Python is.

Re: Why I Still Use Python for High Performance Scientific Computing

#156

Post by srean »

lmm wrote: post #155
srean wrote: post #135

Earlier quoted context omitted.

That's my understanding as well. This is the reason why you are caught between the rock and a hard place on the Hadoop stack. Java code wont be efficient in terms of FLOPs and a popular and often effective escape hatch: code number crunching parts in C, C++ or Fortran, is also not very effective / convenient. Particularly so if it requires going back and forth over the bridge frequently because crossing the bridge ha…

When we say JVM code is not efficient in terms of FLOPs we're talking about a factor of 2 or so though, not really the same as the factor of 20+ you lose by implementing numeric code in pure Python. So Scala gives you a 10x faster cycle during early development. Maybe it's fast enough that you don't need that last factor of 2, or maybe just JNIing a few core operations is enough (you can also do some tricks with the…

> When we say JVM code is not efficient in terms of FLOPs we're talking about a factor of 2 or so though, not really the same as the factor of 20+

Not so in my experience. Write a matrix multiply in pure Java, and pure C, C++, Fortran. The difference would easily be north of 5X, typically more.

Regarding JNI, please see the root of the comment tree. Rarely have I seen more stinky garbage. Its hopeless if you have to go back and forth across the bridge. If that be so I might as well be on the other side of the bridge.

JVM is fantastic, but if you are doing number crunching and performance matters, then JVM is a wrong choice.

Re: Why I Still Use Python for High Performance Scientific Computing

#157

Post by yomritoyj »

codygman wrote: post #136
yomritoyj wrote: post #52

Earlier quoted context omitted.

Regarding Haskell specifically, Haskell makes it cumbersome to write code that mutates data structures, which is what you need to efficiently implement many numerical algorithms. But I'm a fan of Haskell's [REPA]( https://hackage.haskell.org/package/repa ) which provides multidimensional arrays with complete control over which operations are computed immediately and which are deferred and can be combined later with o…

> Regarding Haskell specifically, Haskell makes it cumbersome to write code that mutates data structures Can you provide some Haskell examples that are cumbersome, then provide an example that isn't in another language of your choice? Also, have you tried using lens for this?

Compare the implementation of a simple numerical program in Haskell

https://github.com/jmoy/avb-econ-hs/blob/f6d65af480c99045573...

to that in Julia

https://github.com/jesusfv/Comparison-Programming-Languages-...

Despite the Haskell program using the elegant facilities provided by REPA, the fact that mutable part (the `writePolicy`) has to be written in a different style than the pure parts since it is a monadic action seems cumbersome to me.

Re: Why I Still Use Python for High Performance Scientific Computing

#158

Post by codygman »

yomritoyj wrote: post #157
codygman wrote: post #136

Earlier quoted context omitted.

> Regarding Haskell specifically, Haskell makes it cumbersome to write code that mutates data structures Can you provide some Haskell examples that are cumbersome, then provide an example that isn't in another language of your choice? Also, have you tried using lens for this?

Compare the implementation of a simple numerical program in Haskell https://github.com/jmoy/avb-econ-hs/blob/f6d65af480c99045573... to that in Julia https://github.com/jesusfv/Comparison-Programming-Languages-... Despite the Haskell program using the elegant facilities provided by REPA, the fact that mutable part (the `writePolicy`) has to be written in a different style than the pure parts since it is a monadic acti…

I agree that the Julia looks nicer, despite the fact I usually dislike so much nesting.

> the fact that mutable part (the `writePolicy`) has to be written in a different style than the pure parts since it is a monadic action seems cumbersome to me.

It has to be written differently more because it's using the state monad. You can write monadic actions that resemble pure code, for instance:

    > replicateM_ (readFile "/etc/issue")
    ["Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n"]
Post reply on HN