Live data from Hacker News

The Twisted Way

glyph.twistedmatrix.com

21–30 of 32 posts

Re: The Twisted Way

#21
post #15
post #11

Earlier quoted context omitted.

If by incomprehensible, you mean you need to read more than 10 pages of documentation to understand it, then sure.

Callbacks are bad for the same reason non-local gotos are bad. They destroy locality and make the code hard to read.

"Hard to read" is subjective; many Twisted afficionados, myself included, strongly disagree.

What impact do you feel the destruction of locality has, specifically?

Do you feel @inlineCallbacks and Corotwine answer these issues?

Re: The Twisted Way

#22
post #6
post #4

Earlier quoted context omitted.

I use Twisted and love it. What really helped me was reading the source code instead of trying to decipher the documentation. The code is really readable and things started clicking once I made that investment.

If you don't mind - what did you build with it? Was Twisted the only suitable library - what about Tornado? I'm curious because Twisted has been around for ages and seems to be in active development but I've never talked to anyone who actually uses it.

It is my understanding that Spotify is (or at least was, when I saw their presentation), pretty much a Giant Twisted App with some C++ for parts that were too slow.

Lucasfilm, justin.tv, Launchpad and TweetDeck are some of the larger names you may have heard of. A bunch of smaller startups do too.

If you've been doing XMPP you've probably used a Twisted server (it's either that or ejabberd, mostly) at some point.

If you want Tornado, why not use Cyclone? It's a port (to Twisted) of Tornado; you get to write webby parts just like tornado's and you get the benefit of all of Twisted when you need it.

FWIW, I did the Twisted Help Desk and co-hosted the Twisted tutorial two years in a row now at EuroPython, and two years in a row that tutorial has had to move to a bigger room. Asking around showed a large portion of people were there because there place of work was either using or considering using Twisted. I never cease to be surprised by the stories I hear of people using Twisted to great success for pretty nifty applications.

Re: The Twisted Way

#23
post #12
post #11

Earlier quoted context omitted.

If by incomprehensible, you mean you need to read more than 10 pages of documentation to understand it, then sure.

There are two problems with callback based frameworks. #1 In python specifically, we do not have proper closure and no anonymous blocks, this makes operating it tedious as the definition of the logic for a callback is always somewhere else from where the callback is setup. #2 In any language supporting proper closures (javascript for instance, hello node.js) excessive, nested use of callback based frameworks leads to…

In what sense does Python not have "proper closure"? Do you mean mutable closures?

Also, does @inlineCallbacks address your issue? The only fundamental difference I see (other than implementation nastiness, which I guess both have some issue with :)) is that with inlineCallbacks your yields are explicit, and with greenlets they're not.

Re: The Twisted Way

#24
post #17
post #12

Earlier quoted context omitted.

There are two problems with callback based frameworks. #1 In python specifically, we do not have proper closure and no anonymous blocks, this makes operating it tedious as the definition of the logic for a callback is always somewhere else from where the callback is setup. #2 In any language supporting proper closures (javascript for instance, hello node.js) excessive, nested use of callback based frameworks leads to…

What do you mean by greenlets? Twisted feels to me like a grand experiment that's groping towards something that we know is there, but not quite catching it yet. I guess I/O monads feel like slightly further along the path - but they're still not the true solution. What does your system look like?

Greenlets are one-shot coroutines (with no implicit scheduling). They generally work with a C extension that slices the stack.

In IO frameworks that are greenlet based, the greenlet is resumed when it has stuff waiting for it.

The obvious difference to the programmer is that a greenlet won't have yield statements in them, whereas a generator would.

Re: The Twisted Way

#25
post #9
post #3

Twisted is the least accessible of the big Python libraries that I've come across. I've come to it a couple of times when it seemed like it would suit a problem I had, only to throw up my hands in frustration and go look for a solution somewhere else. Unfortunately this article did nothing to enlighten me - it has some pretensions of being philosophical without explaining any of the practicalities. Does anyone use Tw…

I've used it, and had some success with it, but now use my own stack for network programming. I struggled with twisted several times without getting it. Then I did some non-blocking socket/select programming and thought, "I wish I had a library that.." and then realised what twisted did. There's a few reasons I'm not using it now. Main one is that I've turned against frameworks. I find it's generally easier to unders…

Your restricted environments don't allow virtualenvs?

It's interesting that inheritance vs composition comes up, since one of the most common criticisms I've heard is actually the opposite (that there's too many different objects to get things done).

However, I do understand why you might think this: protocols, which is probably the first thing you touch in Twisted, are usually implemented through inheritance. For resources, the default implementation that you subclass (twisted.web.resource.Resource) contains a bunch of behavior that you almost certainly want.

The important thing to remember is that this inheritance is only to get default behavior. If you want to implement it without inheritance, Twisted very clearly and explicitly defines the interface you have to implement:

https://twistedmatrix.com/documents/current/api/twisted.inte...

https://twistedmatrix.com/documents/current/api/twisted.web....

These are just two examples: Interfaces are all over twisted.

And, to quote from the article:

  It doesn't need to be that way, though.  When you create
  an object, it is best to create it as independently as
  possible; to test it in isolation; to discretely separate
  its every interaction with the outside world so that they
  may be carefully controlled, monitored, intercepted an
  inspected, one iteration at a time.
This has recently come up in the async-pep conversation on Python-ideas. If you have a significantly better solution for e.g LineReceiver, I'd love to steal it!

Re: The Twisted Way

#26
post #25
post #9

Earlier quoted context omitted.

I've used it, and had some success with it, but now use my own stack for network programming. I struggled with twisted several times without getting it. Then I did some non-blocking socket/select programming and thought, "I wish I had a library that.." and then realised what twisted did. There's a few reasons I'm not using it now. Main one is that I've turned against frameworks. I find it's generally easier to unders…

Your restricted environments don't allow virtualenvs? It's interesting that inheritance vs composition comes up, since one of the most common criticisms I've heard is actually the opposite (that there's too many different objects to get things done). However, I do understand why you might think this: protocols, which is probably the first thing you touch in Twisted, are usually implemented through inheritance. For re…

Thanks for feedback. #twisted ppl were very helpful to me in the early days of me learning about network programming and python.

I'm impressed with the achievement of Twisted. For example, the way it offers support for both unix-way async (select, getabyte getabyte), and the Windows/AIX alternative where you can make a system call that says "get back to me when you have a complete message that I can consume in one hit". It'll be way faster and more polished than the stuff I write too.

Part of the reason there's so many objects in it is because it does so damn much - there's support for all sorts of existing protocols in twisted. And - a powerful framework can't entirely abstract away the layering. If it does, it stops being powerful.

It might be possible to use virutalenvs. I keep meaning to look into these. I am skeptical of frameworks though. You become vulnerable to version changes, you can't take knowledge to a new platform.

Very interested to see something from the python-ideas conversation in the standard library. I'll have a serious go at whatever they end up with, and wouldn't mind if it was a stripped down twisted but with composition rather than inheritance. It may be the feature that pushes python people to v3. I'll look over the links you sent regarding this.

Re: The Twisted Way

#27
post #16

I am more interested in a post about what's wrong with twisted. Anyone care to comment?

Having used Twisted for a few projects and now using Pyramid for another, I'd say there are two big problems with Twisted that kill the experience for new developers:

1) The documentation is terrible.

Even though there's a lot of it, and even though the API documentation is pretty good, the 'narrative' and tutorial-like documentation is downright horrible. It tries to cram so many things into the same contrived examples at once, that reading it creates more confusion than giving insights. Take for example the 'evolution of a finger client', which is one of the first things you get directed to when you want to start 'learning twisted'. Instead of introducing some basic concepts that affect design decisions and then working them out in simple, individual, small examples, it starts out with a server example so trivial that it's completely un-representative of how a typical Twisted service would look like, and then, in a few pages, crams so many things into it at once that you have to go back, re-read and actually try to implement and run the example yourself to figure out what (and especially: why) the heck things are working like they are working. In one swell swoop it 'teaches' you Twisted by throwing around component-based programming, Zope interfaces, adapter classes, the factory pattern and how you can basically use it everywhere even though there's no real reason to do so, how you can expose 4 different protocols talking to the same service at once, how you can add asynchronous versions of the same services, and so on. All of it combined in a single 'Twisted application' that reads like some kind of intentially obfuscated crypto-program that somehow magically does all these things.

Compare to that the Pyramid documentation, which is absolutely fantastic and got me up to speed and writing very clean and stable applications in just a few hours. Where reading the Twisted docs the main thing going around in my head was 'wtf do I have to do here and why', while with Pyramid it everything immediately made sense. Note that I'm not saying this is a problem with Twisted itself, just with the way the documentation tries to explain how to use it.

2) Twisted doesn't advertise its specific benefits and downsides very well.

Superficially looking, many people might think Twisted is just like other web-application frameworks like Pyramid, Flask or Django, but it isn't. You can do most of everything using Twisted, but it's much lower-level than those. Just like this article mentions, Twisted is more like a toolbox to create event-based applications than it is like a web-app framework. This makes it a lot more flexible, but also more complex. While this allows all kinds of use cases that nobody would ever consider e.g. something like Flask or Django for (e.g. game servers, chat servers, etc), it also means that if your application is 'yet another web application', the other options are probably going to be a lot simpler and cleaner than using Twisted.

I think because Twisted doesn't want to choose and declare what it's main strengths and purposes are, it's attracting many developers of applications that are a much better fit for the more specialized web-app frameworks, leaving them with a really bad experience, spending too much time figuring out the framework and ending up with cryptic gibberish that doesn't really implement any actual application logic.

Don't misinterpret this as a critique of Twisted itself, I still like it a lot, exactly because it is so flexible and generic. It's just not the best tool for most web applications, but if if you are implementing some kind of event-based application that doesn't fit the philosophy of the other popular frameworks, it's extremely powerful and well-designed. I think the Twisted community should make this difference more explicit for new developers.

Re: The Twisted Way

#28
post #27
post #16

I am more interested in a post about what's wrong with twisted. Anyone care to comment?

Having used Twisted for a few projects and now using Pyramid for another, I'd say there are two big problems with Twisted that kill the experience for new developers: 1) The documentation is terrible. Even though there's a lot of it, and even though the API documentation is pretty good, the 'narrative' and tutorial-like documentation is downright horrible. It tries to cram so many things into the same contrived examp…

Your analysis of the problems with the tutorial is spot-on. It also really doesn't help that it's presented as "this is a description of what's changing on this page, and here's wall of updated code - go figure out the differences from the previous page"

Re: The Twisted Way

#29
post #6
post #4

Earlier quoted context omitted.

I use Twisted and love it. What really helped me was reading the source code instead of trying to decipher the documentation. The code is really readable and things started clicking once I made that investment.

If you don't mind - what did you build with it? Was Twisted the only suitable library - what about Tornado? I'm curious because Twisted has been around for ages and seems to be in active development but I've never talked to anyone who actually uses it.

Unfortunately, I think my employer's NDA prevents me from talking about most of what I've developed. I will say that one of the projects I used combined IRC, SMTP, and HTTP together in one server. Twisted made that incredibly easy to write, and be performant. AFAIK, Tornado is HTTP only at this point.

Re: The Twisted Way

#30
post #21
post #15

Earlier quoted context omitted.

Callbacks are bad for the same reason non-local gotos are bad. They destroy locality and make the code hard to read.

"Hard to read" is subjective; many Twisted afficionados, myself included, strongly disagree. What impact do you feel the destruction of locality has, specifically? Do you feel @inlineCallbacks and Corotwine answer these issues?

Destroying locality makes the program very difficult to read for anyone other than the person who wrote it. I've been there before and done that. I've written Z80 assembly code where functions had more than one entry point as well as exit point. Sometimes I didn't bother to keep a stack at all. Spaghetti code. I was proud of it at the time. Not so much now.

My understanding of Corotwine and greenlet is that they're basically poor man's threads. Basically, if the language doesn't give you usable threads, you can go ahead and implement your own. I've done that one too-- I once wrote an M:N threading library in userspace. Basically making a small number of kernel threads look like a large number of threads to the programmer.

The problem with rolling your own threads is that the normal tooling that comes with the language won't know how to handle what you built. There is no syntax for your add-on, and the semantics may be murky at best. Stack traces become useless, and error messages become guru meditations. Interfacing with third party libraries becomes a challenge, unless they're designed to be purely single threaded.

Additionally-- how to put this delicately? Something as fundamental as threads should not be built as an add-on to a programming language. Things built on top of a language tend to be very fragile and buggy compared to things built in to a language. The C programming language was designed before threads were even a thing in UNIX, so lacking threads was understandable. We built our own threading packages because we had to. But you shouldn't need to do that for a modern language.

Post reply on HN