Python Is Eating the World • Hacker News Board

Re: Python Is Eating the World

#841

Post by p1esk » Mon, Aug 12, 2019, 6:57 PM UTC

verttii wrote: post #826

p1esk wrote: post #799

Earlier quoted context omitted.

Right now I’m experimenting with a pretty complicated model (60+ layers of multiple types), and I plan to train it on several hundred GB of data, using 8-16 node cluster (4 GPUs per node). Does Owl have a well tested and well documented autograd library with distributed GPU support (e.g. Horovod)? With a good choice of optimizers, regularizers, normalizers, etc, so I can focus on my model and not on debugging the plu…

I feel what you're saying is that regardless of how subpar a language is compared to alternatives as long as it has community built specific libraries that solve your problems you're more productive using them than anything else. Which is of course a fair point. A language by itself is probably not even in the top 3 considerations when choosing new tech. Stuff like runtime, ecosystem and the amount of available devel…

how subpar a language is compared to alternatives

In my 6 years with Python, the only dissatisfaction with the language I felt was from parallel programming. I switched to Python from C, and at the time, I missed C transparency and control over the machine, but that was compensated by the Python conciseness and convenience. Then I had to dig into C++ and I didn't like it at all. Then I played with CUDA and OpenMP, and Cilk+, but I wished all that would be natively available in a single, universal language. Then I started using Theano, then Tensorflow, and now I'm using Pytorch, and am more or less happy with it. It suits my needs well. If something else emerges with a clear advantage, I'll switch to it, but I'm not seeing it yet.

Re: Python Is Eating the World

#842

Post by _coveredInBees » Mon, Aug 12, 2019, 6:59 PM UTC

CydeWeys wrote: post #794

joshuamorton wrote: post #753

Earlier quoted context omitted.

FYI, the google style guide (or maybe the internal only version) suggests to avoid initialize-then-assign in favor of single assignment form: unclear_type_thing = ... if isinstance(unclear_type_thing, list): group_keys = unclear_type_thing else: group_keys = [unclear_type_thing] statically avoids this problem. In general, prefer immutable variables where possible. Single-assignment form is nice for a lot of reasons,…

All of these things are true, but they require a non-trivial level of experience and discipline to avoid most potential gotchas. Your average Python project on the Web isn't written to this level of quality, and when people are learning programming using Python in school they certainly aren't there yet, and are gonna hit all kinds of problems related to this stuff. But is there a way to force immutable variables in P…

I don't think that's fair to be honest. If you had simply used Pycharm with default settings you would have easily caught the first bug due to the linting. It's a fair complaint, but this specific bug is easy to catch using any modern Python IDE.

Re: Python Is Eating the World

#843

Post by 27182818284 » Mon, Aug 12, 2019, 7:00 PM UTC

_coveredInBees wrote: post #695

CydeWeys wrote: post #685

Earlier quoted context omitted.

Here's some rational "hate" for Python then. I just returned to Python for the first time in a little while to collaborate on a side project and ran into a few tricky-to-debug errors that caused a fair bit of lost time. Know what the errors were? In one case, I was iterating over the contents of what was supposed to be a list, but in some rare circumstances could instead be a string. Instead of throwing a type error,…

Your 2nd error isn't possible in Python, so I'm not sure what you did there. Regarding the first, sure, it is a bug that was annoying to catch. But, having an `Iterable` interface in Python is also really neat and useful if used responsibly. If you're programming regularly in Python, you are accustomed to the tradeoffs that come with a dynamic programming language and no static types, and you can still avoid issues l…

>Iterable` interface in Python is also really neat and useful if used responsibly.

Honestly this was a major attraction to python for me a decade plus ago as a student when I started learning--even when I used it irresponsibly. There are so many small tasks where you just kinda have to iterate over 100-1000 items that you're not worried about big-O or anything like that—you just want to iterate and work on a collection quickly for some task in the office.

Re: Python Is Eating the World

#844

Post by mpcjanssen » Mon, Aug 12, 2019, 7:03 PM UTC

willtim wrote: post #774

kolanos wrote: post #746

Earlier quoted context omitted.

Python is not weakly typed. It is strongly typed in that it forbids operations that are not well-defined. For example, adding a number to a string) rather than silently attempting to make sense of them. I agree wholeheartedly about weakly typed languages, though.

I believe that marketing Python as "strongly typed" has the potential to confuse rather than educate. Python still crashes at runtime with these errors. It has nice error messages, but it still crashes, potentially in production. If you want to create your own "types", you'll have to add your own runtime checks. It's much more sane than JavaScript, but it's not strongly typed like Haskell. Python does not automatical…

You are describing static typing. There is a well defined difference between strongly typed and statically typed.

Re: Python Is Eating the World

#845

Post by _coveredInBees » Mon, Aug 12, 2019, 7:06 PM UTC

btilly wrote: post #769

_coveredInBees wrote: post #695

Earlier quoted context omitted.

Your 2nd error isn't possible in Python, so I'm not sure what you did there. Regarding the first, sure, it is a bug that was annoying to catch. But, having an `Iterable` interface in Python is also really neat and useful if used responsibly. If you're programming regularly in Python, you are accustomed to the tradeoffs that come with a dynamic programming language and no static types, and you can still avoid issues l…

I disbelieve. And I disbelieve despite being a fan of dynamic languages. The tradeoff is that dynamic languages are faster to develop, more concise, but more expensive in maintenance exactly because of issues like this. The data that I base this opinion on is an unpublished internal report from nearly a decade ago at Google quantifying costs of projects of different size in their different languages. Which was Java,…

I can believe that. But for a lot of people, the lower initial development time/cost aspect matters a lot. If I had Google resources, sure, I'd Go with other languages perhaps, but you can still write high-quality and capable software in Python. And while the batteries included aspect of Python is not everyone's cup of tea, I personally find it quite handy to have that so I don't have to waste a ton of time evaluating different libs to do fairly standard things.

To be clear - I'm not trying to say that Python is better in any objective way. Ultimately, I think people should use the tools they have available and prefer, to build what they want.

Re: Python Is Eating the World

#846

Post by bakery2k » Mon, Aug 12, 2019, 7:08 PM UTC

pytester wrote: post #710

CydeWeys wrote: post #685

Earlier quoted context omitted.

Here's some rational "hate" for Python then. I just returned to Python for the first time in a little while to collaborate on a side project and ran into a few tricky-to-debug errors that caused a fair bit of lost time. Know what the errors were? In one case, I was iterating over the contents of what was supposed to be a list, but in some rare circumstances could instead be a string. Instead of throwing a type error,…

>In one case, I was iterating over the contents of what was supposed to be a list, but in some rare circumstances could instead be a string. Instead of throwing a type error, Python happily went along with it, and iterated over each character in the string individually. I've been using python for about 13 years professionally and I wrote up a list of "things I wish python would fix but I think probably never will" an…

> I wrote up a list of "things I wish python would fix but I think probably never will"

What else is on your list? I'd be interested to see what other parts of Python you would wish to change.

Re: Python Is Eating the World

#847

Post by Jeff_Brown » Mon, Aug 12, 2019, 7:10 PM UTC

mochomocha wrote: post #778

takeda wrote: post #728

Earlier quoted context omitted.

Python is a dynamic language, that's what dynamic languages do, you don't have a type checker but have greater flexibility, but you don't have to settle on that, you can actually use mypy and annotate types and get best out of both worlds. > And another one, I typoed a variable name as groups_keys instead of group_keys (or vice-versa, I don't remember). Instead of just throwing an error, Python happily went along wit…

mypy is a great effort, but very experimental. Try using it on any real-world large enough project and it loses most of its value as there are still a lot of unimplemented things, or because you'll depend on a third party module that hasn't support for it yet.

Case in point: Pandas, the foundation of data programming in Python, does not provide the Series or DataFrame (that's a table) types in a way that MyPy can use.

Re: Python Is Eating the World

#848

Post by triceratops » Mon, Aug 12, 2019, 7:17 PM UTC

icxa wrote: post #604

tus88 wrote: post #526

Earlier quoted context omitted.

Well in just glad this is the top comment, as Python really is taking over the world for a reason. And of all the bugs I have written in recent memory, not one came down to a lack of static typing. They were due simply to logic errors, flawed assumptions, misunderstood requirements, and good old race conditions. The static typing zealots like to think if it compiles is must be perfect, however this is a mirage. Unit…

Have you ever worked in a large engineering organization full of engineers with varying degrees of experience all trying to accomplish the same goal? I can't imagine anyone has ever tried to do engineering at scale (people wise) and did not find the value in static typing. It's why startups eventually moved off RoR once they started scaling. It's why there is such a large push to type JavaScript (have you seen the ro…

> It's why startups eventually moved off RoR once they started scaling.

I thought it was because of Ruby's poor performance characteristics.

Re: Python Is Eating the World

#849

Post by theptip » Mon, Aug 12, 2019, 7:19 PM UTC

totalperspectiv wrote: post #757

theptip wrote: post #752

Earlier quoted context omitted.

The bugs you describe should both be easy to catch with unit tests. It sounds like the problem is not that you're using Python, it's that your project lacks tests. Sure, you can typo this sort of thing; but it should be apparent within seconds when your tests go red. (And nowadays, you can also use type hints to give you a warning for this kind of thing, e.g. your IDE/mypy will complain about passing a string where t…

Serious question: If you are writting unit tests to check types, why not just use a language that has a compiler that does that for you? And if you are writing python with type hints, why not just use a language that uses the types you spend time adding to make your program faster. Python is great for sharing ideas / concepts, but under some circumstances it seems irresponsible to choose it over other viable options…

As the sibling comment said, I'm not proposing checking types in unit tests, I'm proposing checking that the behaviour is correct.

If there's a code path that passes in a bare string instead of a list, and your logic breaks, then that code path should have a failing test case. However, type hints can provide another opportunity to catch this kind of mismatch before they even get committed.

> under some circumstances it seems irresponsible to choose it over other viable options like Go (if you use Python because it's easy)

This is probably true, but I think people tend to overuse this argument (i.e. use an overly broad set of "some circumstances"). I build fintech apps with Python, for example, and don't find any of these issues to be a problem. In my experience, if you implement sound engineering practices (thorough testing at unit, integration, and system levels, code review, clear domain models, good encapsulation of concerns, etc.), then the sort of errors that bite you are not ones that a type checker would help with. I agree that the worst Python code is probably far more unsound than the worst Go code, but I don't think that's the correct comparison; you should be comparing the Python and Go code that _you_ (or your team) would write.

I think it's easy to be dogmatic about this kind of thing; in practice most people are substituting personal preference for technical suitability. Sure, there are cases where the performance or correctness characteristics of a particular language make it more suitable than another. But for most software, then whatever your team is expert in is the best choice.

Re: Python Is Eating the World

#850

Post by CydeWeys » Mon, Aug 12, 2019, 7:20 PM UTC

_coveredInBees wrote: post #842

CydeWeys wrote: post #794

Earlier quoted context omitted.

All of these things are true, but they require a non-trivial level of experience and discipline to avoid most potential gotchas. Your average Python project on the Web isn't written to this level of quality, and when people are learning programming using Python in school they certainly aren't there yet, and are gonna hit all kinds of problems related to this stuff. But is there a way to force immutable variables in P…

I don't think that's fair to be honest. If you had simply used Pycharm with default settings you would have easily caught the first bug due to the linting. It's a fair complaint, but this specific bug is easy to catch using any modern Python IDE.

I've never found the "Use this specific IDE" defense particularly valid, considering that many IDEs don't have these features and that in other languages the compiler itself protects you.

Needless to say, I was not using Pycharm for this development, nor am I likely to install an entire IDE just for a small change I'm making on a random project. It's a non-trivial burden to configure and learn an entire IDE, vs just using what I already know (which is often just emacs).