How to export Django data faster than ever before

#1

Post by palewire » Tue, Sep 05, 2017, 2:41 PM UTC

How to export Django data faster than ever before
californiacivicdata.org

Re: How to export Django data faster than ever before

#2

Post by Alex3917 » Tue, Sep 05, 2017, 3:09 PM UTC

This is cool. Is there a way to automatically sanitize for CSV Injection vulnerabilities? (I.e. stripping out Excel macros.)

Re: How to export Django data faster than ever before

#3

Post by palewire » Tue, Sep 05, 2017, 3:21 PM UTC

Alex3917 wrote: post #2

This is cool. Is there a way to automatically sanitize for CSV Injection vulnerabilities? (I.e. stripping out Excel macros.)

I hadn't considered that. It only accepts CSVs, not Excel files, right now. If you can provide an example of a dangerous file we can try testing against it.

Re: How to export Django data faster than ever before

#4

Post by Alex3917 » Tue, Sep 05, 2017, 3:36 PM UTC

palewire wrote: post #3

Alex3917 wrote: post #2

This is cool. Is there a way to automatically sanitize for CSV Injection vulnerabilities? (I.e. stripping out Excel macros.)

I hadn't considered that. It only accepts CSVs, not Excel files, right now. If you can provide an example of a dangerous file we can try testing against it.

I meant in terms of what it's outputting from the models, not what it's accepting as input. Since most apps accepts accept input from users via API endpoints or forms, sanitizing the output strikes me as more important than sanitizing the input, although ideally both should happen.

I don't have any good example data, but this is what I'm talking about:

https://github.com/wagtail/wagtail/issues/3466

A lot of folks don't care about this if they are making libraries for users, since it's really an issue with Excel, but if people within your company are the ones who are going to be potentially getting pwned then it makes less sense to me to just blame MS or whatever.

Re: How to export Django data faster than ever before

#5

Post by palewire » Tue, Sep 05, 2017, 4:01 PM UTC

Alex3917 wrote: post #4

palewire wrote: post #3

Earlier quoted context omitted.

I hadn't considered that. It only accepts CSVs, not Excel files, right now. If you can provide an example of a dangerous file we can try testing against it.

I meant in terms of what it's outputting from the models, not what it's accepting as input. Since most apps accepts accept input from users via API endpoints or forms, sanitizing the output strikes me as more important than sanitizing the input, although ideally both should happen. I don't have any good example data, but this is what I'm talking about: https://github.com/wagtail/wagtail/issues/3466 A lot of folks don…

Thanks for the tip. This is something I was unaware of.