Well, I've read that it takes ten years to build great software. So golang has a while to go. Python, while not perfect has the infrastructure already built.
From Python to Go and Back Again
141–150 of 167 posts
Re: From Python to Go and Back Again
#142The claim that pypy uses less memory than Go seems...rather extraordinary. I worked with a fairly complex http api app that ran as a rather svelte wsgi framework under gunicorn, and we saw at least 10x or more increase in memory usage than cpython when we switched to pypy, once the jit was fully warmed up (memory usage seemed to hit steady state after about an hour). pypy has also historically (in my experience) been…
I don't know the details, but the memory difference was almost certainly due to the different approaches to concurrency. Python coroutines just need to save their exact stack frame while a Go goroutine will spawn a "massive" stack that likely has much more space than needed. The first case might only need tens or hundreds of bytes for a dozen local variables, while the latter case is a fixed overhead of several KB (8…
The memory used by the JIT will at some point remain constant but before that happens, it will grow as the JIT considers more and more traces to be worthy of compilation.
It can take a very long time until the memory consumed by the JIT actually remains constant. If you do continuous integration and deploy several times per day, your application might never reach that point.
Re: From Python to Go and Back Again
#143Earlier quoted context omitted.
Javascript is like Flash, it's great for the author of a page to show off, or force an ad on you, but how does it actually benefit the end user? Not at all, never has.
Really now. JS can potentially improve UI and UX, providing users with a smoother experience.
It can yes, but like anything else it often provides the opposite, even setting aside the download time.
Re: From Python to Go and Back Again
#144Earlier quoted context omitted.
It is unusual to claim that programming in a statically typed language is more error-prone than in a dynamically typed language, even if only when dealing with HTTP requests specifically. Could you elaborate? It sounds like there might be a story behind this.
Thanks for calling me out on that, I might have a hard time justifying the claim. It was a reference to the fact that, at your system boundaries the benefit of a strongly-typed language aren't only offset by the lack of flexibility, but, in the case of Go, it's weak reflection capabilities and type system. A web app has 4 (often more, rarely less) such boundary: - Getting input from users - Querying a database - Gett…
>- Getting input from users
That data is always a string (given the nature off HTTP requests). So the only issue there is converting the string to an integer when necessary. But since you should cleanse any data that arrives via HTTP request, you'd need to validate that your "integers" are actually purely numeric even in dynamically typed languages. So there's really no extra work there between dynamic and static languages.
>- Querying a database
You'd be querying an SQL database with either parametrised queries or ORMs. Both of which are data-type agnostic (ie you wouldn't be needing to convert integers into strings to embed into SQL strings).
As for No-SQL databases, there might be an issue with some and statically typed languages. But that's not an issue I've ran into with the languages and APIs against the (admittedly limited) range of no-SQL databases I've used.
>- Getting results from a database
This is where your argument is the strongest. Sometimes there can be an issue if you don't know what return values you're expecting from the database. But that's easily overcome if you actually chat to your database architects before hand. But in all honesty, I'd be disappointed in any web developer who wasn't the least bit interested in the datatypes of the records he's querying nor the structure of the database he's effectively writing a frontend for.
>- Outputting results to the user (html, json, ....)
All HTML is string, so that's a moot point. JSON, XML, etc is more a data structure problem than a data type problem. In fact I sometimes argue that JSON is statically typed since it has strings (in quotations), integers (no quotations), boolean (true / false), arrays and hashes / maps. So the real problem with exporting formats like JSON and XML is really a question of how good a language API is. Take C# for example, there's several different APIs available for encoding XML, some are appallingly bad and need about a page of boilerplate code, others are ridiculously simple. To go back on topic with Go, I've only ever worked with JSON, but outputting that in Go is very easy as Go's JSON encoder basically just takes whatever your data structure is and returns it's JSON encoded string counterpart (much like how Perl and Javascript work with JSON).
I do get the points you're making, and you're right that sometimes statically typed languages do make you jump through a few additional hoops. But most of the times these issues only arise if you're a careless programmer - in which case you're going to run into all sorts of dumb issues even with dynamically typed languages (eg if you don't validate your input data then you're going to write less secure web applications - regardless of your language of choice. That's why I sometimes look at statically typed languages as just another layer of data validation with regards to web development)
Re: From Python to Go and Back Again
#145Earlier quoted context omitted.
Didn't solve my problem. PyPy still retains multi-threading and GIL stuff if I remember correctly.
Did you try requests_futures lib? It's all about async and network speed with crawling. Not so much cpu.
I thought it would be IO bound (that's why I started with Python at first place), but since I was extracting links as well and working a bit on graph it turned out to be more CPU intensive. But well, maybe I could have written better code, better libraries, maybe multiprocessing (would have been painful though with multiprocessing). I do admit, I didn't look much into how I could improve it within Python. I just went with Go because it was quicker that way for me.
Re: From Python to Go and Back Again
#146I'll be interested in coming back to Python when it isn't a headache to deploy into production. I'm tired of an install requiring a GCC compiler on the target node. I'm also tired of having to work around the language and ecosystem to avoid dependency hell.
This was posted here, and not a bad idea:
Re: From Python to Go and Back Again
#147Earlier quoted context omitted.
I should note, we don't care about throughput for the most part. Our constraint is purely the memory use of holding open the connections. The aim is to hold as many connections as possible within 10-20% of the machines RAM, and not exceed it. As such, we need to be careful about resource usage and spikes. Goroutines feel cheap, but if you're holding 140k connections, and just 20k of them do something that spins up a…
Not exactly related to Go/PyPy, but I'm curious whether you can say something about how you handle memory and bandwidth constraints? E.g. what do you do if you want to send notifications to lots of clients but for some the connection is very slow (you would probably need to buffer the data)? Do you have hard limits of maximum buffered data until you close the connection? End to end backpressure (for which channels ar…
We're using twisted, but I believe Python 3's asyncio has a similar feature with use of non-blocking sockets, which is that you can add a hook to be triggered when too much data accumulates in user-space (can't be flushed to the kernel's tcp buffer).
In our case, when notifications buffer for a slow client, this API gets triggered and we mark the client connection as 'paused'. Until that state is cleared by more data getting to the client, notifications go to the database instead with just a flag on the client connection to check the db when the pending data was retrieved.
We do a similar thing on the receiving end to pause reading off the socket if we're already doing more work on behalf of the client at once than desired.
twisted documents this as producer/consumer: http://twisted.readthedocs.org/en/twisted-15.4.0/core/howto/...
Re: From Python to Go and Back Again
#148Earlier quoted context omitted.
Doesn't a TLS terminator proxy solve this? E.g. I usually put my application services behind HTTPS-enabled nginx and it works wonderfully.
Nope, so, the goal here is to reduce how many machines (each with their own RAM limits) are used. The task is holding open bidirectional SSL wrapped long-lived websocket connections. They're held open for hours at a time, since we need to send notifications when we get them. Every connection has a base cost of the TCP kernel send/recv buffer, which in our case we dropped a bit to 4kb each. So that's still 8kb per con…
What is your (unique) use-case about? What service do you provide to your users?
Re: From Python to Go and Back Again
#149There are people who have been working in Go for years, successfully, but who don't post comments with the same frequency and dogged determination as the middlebrow dismissers. Both Python and Go are fine. They both have their strengths and weaknesses. I personally wouldn't write a web app in Go (at least, anything beyond the most basic admin interface). I also personally wouldn't write a very large and complex Pytho…
Do you have any increased optimism about the refactorability of python given the recent addition of type annotations?
I would really like to see a type check package that you can run alongside your unit tests to verify type correctness. MyPy is nice, but I don't want to do type checks at runtime.
Re: From Python to Go and Back Again
#150Earlier quoted context omitted.
> Perfect use case for Go Go shouldn't have "use cases". One should be able to do almost everything with ease with a language built in 2008/9. And unfortunatly that is not the case. Go has excellent concurrency features, but is limited by dumb language design decisions which make it painfull to test and to write good reusable and composable libraries for. I'd love to replace my entire stack with Go but I can't. Somet…
> Something I would write in Ruby in 10 days takes 2 months in Go. Have you looked into Elixir? It has Ruby like syntax, but uses an actor model for concurrency(it runs off of the Erlang VM). For handling concurrent tasks it tends to benchmark around Go's speed, but is much nicer to do things in. While it is admittedly immature, the ecosystem still has a decent amount of packages and the tooling isn't bad.
Being a Ruby to Go to Elixir convert myself, I can only second this. Elixir is a really great language.