Django-medusa: Rendering Django sites as static HTML

#1

Post by mtigas » Sat, Jun 30, 2012, 7:15 PM UTC

Django-medusa: Rendering Django sites as static HTML
v3.mike.tig.as

Re: Django-medusa: Rendering Django sites as static HTML

#2

Post by bifrost » Sat, Jun 30, 2012, 7:26 PM UTC

Thats kinda neat, I suspect this will be used a lot. I did something similar to this 5-6 years ago with wget but this is a lot more elegant IMHO.

Re: Django-medusa: Rendering Django sites as static HTML

#3

Post by mtigas » Sat, Jun 30, 2012, 7:57 PM UTC

bifrost wrote: post #2

Thats kinda neat, I suspect this will be used a lot. I did something similar to this 5-6 years ago with wget but this is a lot more elegant IMHO.

It’s very wget-like, due to the use of the Django HTTP test client — just slightly more elegant due to the programatic definition of what gets scraped/rendered and the addition of the "direct to S3" backend, which allows arbitrary mimetypes.

Glad you like it.

Re: Django-medusa: Rendering Django sites as static HTML

#4

Post by natrius » Sat, Jun 30, 2012, 8:43 PM UTC

"The process that actually generates the output simply uses (or abuses) Django’s internal testclient to request each URL and store the resulting data"

It warms my heart to see another example of test client abuse. We used the test client to implement a pure Python ESI processor[1]. We ran it in production for awhile, but no one should ever do that. Varnish is the answer. The code was still running on our development machines when I left.

[1] https://github.com/armstrong/armstrong.esi/

Re: Django-medusa: Rendering Django sites as static HTML

#5

Post by megaman821 » Sat, Jun 30, 2012, 9:11 PM UTC

Why all these static site generators? Just use Varnish, it is the static site generator for any and all frameworks.

Re: Django-medusa: Rendering Django sites as static HTML

#6

Post by rglullis » Sat, Jun 30, 2012, 9:22 PM UTC

megaman821 wrote: post #5

Why all these static site generators? Just use Varnish, it is the static site generator for any and all frameworks.

Do you have access to Varnish in your standard run-of-the-mill shared host service?

Re: Django-medusa: Rendering Django sites as static HTML

#7

Post by ashray » Sat, Jun 30, 2012, 9:49 PM UTC

I just don't see the point to this kind of stuff.. =/

For high traffic apps, Varnish is the answer as you don't hit the application layer.

If you think that's too complicated, try nginx-memcached - also an excellent solution.

If not that, try django's template caching with memcached - also extremely fast but will hit the application layer.

If you're in some shared hosting environment (you probably are too small still to warrant this kind of aggressive caching on static assets - but hey, efficiency never hurt anybody :P) without access to memcached, use django's cache backend with a file based cache. It's almost 100% as what this does and you don't have any additional overhead.

Beats me why people are re-inventing the wheel - or am I missing something ?

Re: Django-medusa: Rendering Django sites as static HTML

#8

Post by yuvadam » Sat, Jun 30, 2012, 9:55 PM UTC

ashray wrote: post #7

I just don't see the point to this kind of stuff.. =/ For high traffic apps, Varnish is the answer as you don't hit the application layer. If you think that's too complicated, try nginx-memcached - also an excellent solution. If not that, try django's template caching with memcached - also extremely fast but will hit the application layer. If you're in some shared hosting environment (you probably are too small still…

Yeah, deployment scenarios where you have no application layer, and can only deploy static assets (S3, GitHub pages and the likes).

Re: Django-medusa: Rendering Django sites as static HTML

#9

Post by ashray » Sat, Jun 30, 2012, 10:07 PM UTC

yuvadam wrote: post #8

ashray wrote: post #7

I just don't see the point to this kind of stuff.. =/ For high traffic apps, Varnish is the answer as you don't hit the application layer. If you think that's too complicated, try nginx-memcached - also an excellent solution. If not that, try django's template caching with memcached - also extremely fast but will hit the application layer. If you're in some shared hosting environment (you probably are too small still…

Yeah, deployment scenarios where you have no application layer, and can only deploy static assets (S3, GitHub pages and the likes).

That's interesting. So basically it's for rendering pages and then pushing them on to S3/Other-static-storage and serving them from there ?

I don't see any particular gains however since high traffic pages (static or otherwise..) served off S3 will end up costing way more than on a shared host/dedicated server.

What would be an exact use case where this would really be important and help out ? I'm just trying to understand what need it solves and under what circumstances hosting static pages on S3 is beneficial in cost/performance.

Re: Django-medusa: Rendering Django sites as static HTML

#10

Post by eragnew » Sat, Jun 30, 2012, 10:12 PM UTC

ashray wrote: post #7

I just don't see the point to this kind of stuff.. =/ For high traffic apps, Varnish is the answer as you don't hit the application layer. If you think that's too complicated, try nginx-memcached - also an excellent solution. If not that, try django's template caching with memcached - also extremely fast but will hit the application layer. If you're in some shared hosting environment (you probably are too small still…

for people like me who haven't spent the time to learn the ins and outs of django but still like hcking around in python, things like this are very enticing. so in a way, the author 'invented the wheel' for me, so i don't have to reinvent it. if that makes sense.