How ProPublica Illinois Uses GNU Make to Load Data
propublica.org
How ProPublica Illinois Uses GNU Make to Load Data
1–10 of 69 posts
Re: How ProPublica Illinois Uses GNU Make to Load Data
#2Re: How ProPublica Illinois Uses GNU Make to Load Data
#3> The first is that we use Aria2 to handle FTP duties. Earlier versions of the script used other FTP clients that were either slow as molasses or painful to use. After some trial and error, I found Aria2 did the job better than lftp (which is fast but fussy) or good old ftp (which is both slow and fussy). I also found some incantations that took download times from roughly an hour to less than 20 minutes.
Tangential question: is it possible to use wget for ftp duties? Though may be additional FTP-specific functionality in `aria2c` of course:
https://serverfault.com/questions/25199/using-wget-to-recurs...
Re: How ProPublica Illinois Uses GNU Make to Load Data
#4> The first is that we use Aria2 to handle FTP duties. Earlier versions of the script used other FTP clients that were either slow as molasses or painful to use. After some trial and error, I found Aria2 did the job better than lftp (which is fast but fussy) or good old ftp (which is both slow and fussy). I also found some incantations that took download times from roughly an hour to less than 20 minutes. Tangential…
Have a look at the wget manual. There are lots of FTP-related options.
Re: How ProPublica Illinois Uses GNU Make to Load Data
#5> The first is that we use Aria2 to handle FTP duties. Earlier versions of the script used other FTP clients that were either slow as molasses or painful to use. After some trial and error, I found Aria2 did the job better than lftp (which is fast but fussy) or good old ftp (which is both slow and fussy). I also found some incantations that took download times from roughly an hour to less than 20 minutes. Tangential…
aria is multi connection(aria2c -x5 means five concurrent), thats the main reason for speed bump
Re: How ProPublica Illinois Uses GNU Make to Load Data
#6> The first is that we use Aria2 to handle FTP duties. Earlier versions of the script used other FTP clients that were either slow as molasses or painful to use. After some trial and error, I found Aria2 did the job better than lftp (which is fast but fussy) or good old ftp (which is both slow and fussy). I also found some incantations that took download times from roughly an hour to less than 20 minutes. Tangential…
Have a look at the wget manual. There are lots of FTP-related options.
Perhaps he meant in their specific scenario, since he linked a page that shows use of wget for FTP.
Re: How ProPublica Illinois Uses GNU Make to Load Data
#7Earlier quoted context omitted.
Have a look at the wget manual. There are lots of FTP-related options.
Perhaps he meant in their specific scenario, since he linked a page that shows use of wget for FTP.
Yes, I should have specified that I was interested in what aria2 provides for FTP in addition to what a more ubiquitous tool like wget seemingly has. u/rasz says aria2 allows multi-connections, so that seems sensible: https://news.ycombinator.com/item?id=17508858
Re: How ProPublica Illinois Uses GNU Make to Load Data
#8> The first is that we use Aria2 to handle FTP duties. Earlier versions of the script used other FTP clients that were either slow as molasses or painful to use. After some trial and error, I found Aria2 did the job better than lftp (which is fast but fussy) or good old ftp (which is both slow and fussy). I also found some incantations that took download times from roughly an hour to less than 20 minutes. Tangential…
[deleted]
Re: How ProPublica Illinois Uses GNU Make to Load Data
#9Original, https://www.propublica.org/nerds/gnu-make-illinois-campaign-...
OK, we've changed to that from https://source.opennews.org/articles/how-propublica-illinois.... Thanks!
Re: How ProPublica Illinois Uses GNU Make to Load Data
#10Since they have "A Note about Security", how about locking down that Python environment?
- Add hashes to their already-pinned requirements.txt deps: https://pip.pypa.io/en/stable/reference/pip_install/#hash-ch...
- Add a Makefile entry to run `[ -d your-environment ] || ( virtualenv your-environment && . your-environment/bin/activate && ./your-environment/bin/pip install --no-deps --require-hashes -r requirements.txt )`