Python Is Eating the World • Hacker News Board

Re: Python Is Eating the World

#821

Post by verttii » Mon, Aug 12, 2019, 6:08 PM UTC

swat535 wrote: post #736

coldtea wrote: post #684

Earlier quoted context omitted.

> You need to test anyway. Actually, if your type system is powerful enough, you don't need to test. That's the source of the "if it compiles, 99% of the time it works right" people mention about Haskell (and even more so languages like Idris etc). Type systems are tests -- just formal and compiler-enforced, not ad-hoc "whatever I felt like testing" tests, like unit tests are. From there on it's up to the power of th…

> Actually, if your type system is powerful enough, you don't need to test. That's the source of the "if it compiles, 99% of the time it works right" people mention about Haskell (and even more so languages like Idris etc). Types only eliminate certain tests. You will always have system tests, acceptance tests and unit tests. One should use types to augment their system reliability. Types will not catch logical error…

Haskell's type system most definitely does catch some of your logical errors. That's exactly why it is so revered.

An effective use of a type system such as Haskell's Hindley-Milner can result in a vastly smaller surface area for possible problems and thus can cut a big number of otherwise mandatory unit tests off your todo list.

Re: Python Is Eating the World

#822

Post by HelloNurse » Mon, Aug 12, 2019, 6:09 PM UTC

totalperspectiv wrote: post #757

theptip wrote: post #752

Earlier quoted context omitted.

The bugs you describe should both be easy to catch with unit tests. It sounds like the problem is not that you're using Python, it's that your project lacks tests. Sure, you can typo this sort of thing; but it should be apparent within seconds when your tests go red. (And nowadays, you can also use type hints to give you a warning for this kind of thing, e.g. your IDE/mypy will complain about passing a string where t…

Serious question: If you are writting unit tests to check types, why not just use a language that has a compiler that does that for you? And if you are writing python with type hints, why not just use a language that uses the types you spend time adding to make your program faster. Python is great for sharing ideas / concepts, but under some circumstances it seems irresponsible to choose it over other viable options…

Because the unit tests are not to "check types", they are to check that incorrect values (e.g. a string instead of a list of strings) do not occur. They are no different from other kinds of incorrect values, like attempting to cluster an odd number of items into pairs.

Re: Python Is Eating the World

#823

Post by mixmastamyk » Mon, Aug 12, 2019, 6:10 PM UTC

rafaelvasco wrote: post #818

A joy to write and read. But a pain to debug and maintain. Coded a big project in Python once. When the project was still small everything was perfect. When the project grew it became unsustainable. Of course I was an already experienced C# dev, just getting started with Python but the problem is there. A whole lot of errors spilling to runtime is a big pain. But that's a problem with every dynamic language at variab…

Tools, tests, and logging are the cure to those troubles. I've not experienced substantial problems with projects large or small.

If a project is known to be huge up front and/or some developers will be inexperienced choosing a language with additional guardrails would be prudent.

Re: Python Is Eating the World

#824

Post by staticassertion » Mon, Aug 12, 2019, 6:11 PM UTC

joshuamorton wrote: post #779

staticassertion wrote: post #655

Earlier quoted context omitted.

Huge overhead for process communication, weird interface, weird semantics, etc.

Is that still true with processpoolexecutor? The overhead obviously is still there, but the interface is a drop in replacement for threadpoolexecutor, which looks basically just like a multithreaded/async/future-based collect.

The interface may have improved, I haven't paid attention to mp in a long while. If I'm reaching for mp I generally just accept that I'm using the wrong language.

Re: Python Is Eating the World

#825

Post by vincent-toups » Mon, Aug 12, 2019, 6:13 PM UTC

AnimalMuppet wrote: post #764

vincent-toups wrote: post #548

If you ask me the world only needs two programming languages, Scheme and OCaml and it would be better if OCaml had s-expressions as syntax.

There are more things in the computer programming universe than are dreamed of in your philosophy. (With apologies to William Shakespeare...)

That is why we have syntax-case.

Re: Python Is Eating the World

#826

Post by verttii » Mon, Aug 12, 2019, 6:14 PM UTC

p1esk wrote: post #799

ernst_klim wrote: post #352

Earlier quoted context omitted.

> What does it mean? Have you done it in any of those language? I did. I'm doing a image processing recently and use OCaml for prototyping. I've tried python (I've used it a lot for that long time ago), I've failed, it felt to awkward. I've described my experience here [1] If you have no experience whatsoever with ML family [2], and doing all the stuff in python, you'll most likely be much more productive with python…

Right now I’m experimenting with a pretty complicated model (60+ layers of multiple types), and I plan to train it on several hundred GB of data, using 8-16 node cluster (4 GPUs per node). Does Owl have a well tested and well documented autograd library with distributed GPU support (e.g. Horovod)? With a good choice of optimizers, regularizers, normalizers, etc, so I can focus on my model and not on debugging the plu…

I feel what you're saying is that regardless of how subpar a language is compared to alternatives as long as it has community built specific libraries that solve your problems you're more productive using them than anything else.

Which is of course a fair point. A language by itself is probably not even in the top 3 considerations when choosing new tech. Stuff like runtime, ecosystem and the amount of available developers would probably be more important in most cases.

Re: Python Is Eating the World

#827

Post by stevesimmons » Mon, Aug 12, 2019, 6:15 PM UTC

Barrin92 wrote: post #635

nighthawk648 wrote: post #529

Earlier quoted context omitted.

How stable is python to run a full trading / quantitative algorithm on? I feel there are benefits for every language. I am just curious if super stable and scalable conditions can be met on python.

>How stable is python to run a full trading / quantitative algorithm on? JP Morgan operates a ~30 million LOC Python platform for trading and analytics. (related talk: https://www.youtube.com/watch?v=ZYD9yyMh9Hk ) Yes, there are very, very large working python codebases in fields out there that demand correctness. I'm honestly getting tired of the static typing circlejerk that has entered the industry.

That's my talk! Thank you for posting that :)

Python has been my main programming language since 2000, fwiw.

Re: Python Is Eating the World

#828

Post by mixmastamyk » Mon, Aug 12, 2019, 6:15 PM UTC

sunaurus wrote: post #478

Am I correct when I assume that most of the people hating on Python here are using it for medium or big projects? I feel like the dev community figured this out ages ago: Statically typed languages are easier to read, harder to write, dynamically typed languages are easier to write, harder to read (and also harder to refactor). For small projects (scripts, prototypes, etc), you don't care about readability, you just…

Looks like a lot of folks are using the wrong tool for the job. Reminds me of that onion article about chastising a dog for being a dog. Found it:

https://www.theonion.com/dog-chastised-for-acting-like-dog-1...

Re: Python Is Eating the World

#829

Post by nabdab » Mon, Aug 12, 2019, 6:16 PM UTC

ben509 wrote: post #762

takeda wrote: post #728

Earlier quoted context omitted.

Python is a dynamic language, that's what dynamic languages do, you don't have a type checker but have greater flexibility, but you don't have to settle on that, you can actually use mypy and annotate types and get best out of both worlds. > And another one, I typoed a variable name as groups_keys instead of group_keys (or vice-versa, I don't remember). Instead of just throwing an error, Python happily went along wit…

It has nothing to do with static vs. dynamic. There's no reason that being an early-binding language that a string has to be iterable itself, and the proposal to change this was only rejected as it broke too many things[1] and couldn't be automatically fixed. Point in the GP's favor: Fixing it would definitely not be a problem with an early-binding language! In fact, the nigh-impossibility of automated refactoring pu…

What’s supposed to be the surprising thing? Are you confusing pass and break and expecting it to print 1?

Re: Python Is Eating the World

#830

Post by joshuamorton » Mon, Aug 12, 2019, 6:19 PM UTC

higherkinded wrote: post #789

joshuamorton wrote: post #772

Earlier quoted context omitted.

> sound and flexible type systems Python has a more powerful type system than any mainstream language except for C++ (templates). No other mainstream language has support for explicit marking of variance of arguments, nor does any other language support literal (dependent) types, except for typescript.

And how's it helping you verify the correctness of your codebase at compile time (it still compiles the source to byte-code before evaluation)? Last time I checked, it didn't care too much about type annotations, and it didn't type-check anything before execution without additional shoehorning. >No other mainstream whatever C#, Rust. Notice that I don't even mention my man Haskell or Scala while the latter one is rea…

> And how's it helping you verify the correctness of your codebase at compile time (it still compiles the source to byte-code before evaluation)? Last time I checked, it didn't care too much about type annotations, and it didn't type-check anything before execution without additional shoehorning.

All of the python I write is checked at compile time. Much like java and javac are different tools, python and pytype/mypy are different tools. This is nothing new.

>C#, Rust. Notice that I don't even mention my man Haskell or Scala while the latter one is really big.

Neither supports dependent types.

> Uh, even Java has dependent types, anyway. Read the docs.

No, Java has generic types, not dependent types[0]. Dependent types are types that depend on the value, not the type of an argument. As a concrete example:

    @overload
    def open(
        path: _PathType, 
        mode: Literal["r", "w", "a", "x", "r+", "w+", "a+", "x+"],
    ) -> IO[Text]: ...
    @overload
    def open(
        path: _PathType,
        mode: Literal["rb", "wb", "ab", "xb", "r+b", "w+b", "a+b", "x+b"],
    ) -> IO[bytes]: ...

Java, rust, and C# don't support this. C++ does for certain types (integers). Python and typescript support this for certain types (integers, strings, enum values).

Other common examples are type-safe matrix multiplication that at compile time can ensure that the matrix sizes all match correctly. Eigen in C++ does this. Haskell and Scala can do this with some massaging, although its not natural. Java, Rust, C# just plainly cannot.

[0]: https://en.wikipedia.org/wiki/Dependent_type

[1]: https://www.python.org/dev/peps/pep-0586/