What you need may be “pipeline +Unix commands” only

Post by dredmorbius » Sun, Mar 03, 2019, 12:04 PM UTC

pdimitar wrote: post #55

dredmorbius wrote: post #48

Case in point from my own recent work: I've been analysing characteristics of Google+ Communities, mostly looking for plausibly active good-faith instances. There are 8.1 million communities in total, and thanks to some friendly assistance, I'd identified slihtly more than 100,000 with both 100 or more members, and visible activity within the preceeding 31 days, as of early 2019. The task of Web scraping those 100k c…

Are you planning on open-sourcing the downloader part? I'm very interested.

    #!/bin/bash
    
    sample_file=$1
    
    comm_path='community-pages'
    base_url='https://plus.google.com/communities'
    
    i=0
    time sed -e 's,^.*/,,' $sample_file |
        while read commid;
        do
            i=$((i+1))
            echo -e "\n>>> $i  $commid &2;
    
            url="${base_url}/${commid}"
            commfile="${comm_path}/${commid}.html"
            commhead="${comm_path}/${commid}.head"
    
            echo "curl -s -o '${commfile}' -D '${commhead}' '${url}'"
    
        done

    100000056330101053659
    100000310247038604843
    100000355641542704509
    100000408644688836681
    100000537266485621548
    100000813948204546252
    100001055751908082772
    100001158162744298957
    100001173291703462139
    100001193552641351693

Post by josteink » Sun, Mar 03, 2019, 12:08 PM UTC

speedplane wrote: post #30

This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are…

Post by hashhar » Sun, Mar 03, 2019, 12:19 PM UTC

speedplane wrote: post #43

aidos wrote: post #36

Earlier quoted context omitted.

Well, the article does explicitly say that if you can do this then you don’t have “big data”. Maybe you see TB level processing a lot in your line of work, but most developers never will. Whenever I deal with anything a bit bigger, I break off the smallest section I need to deal with and work with that.

> Maybe you see TB level processing a lot in your line of work, but most developers never will. I doubt this is the case. If you're working on even a medium sized team, you'll see this level of data, whether it's internal server logs, publicly-sourced data, or a variety of other applications. Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run…

Post by tetha » Sun, Mar 03, 2019, 12:27 PM UTC

geocar wrote: post #37

gsich wrote: post #3

Earlier quoted context omitted.

If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.

One time I met a company who insisted they were sending tens of TB of data per day and would need multi-PB per year storage compressed. Took one look at the data: All json, all GUIDS and bools. If we just pre-parse it, the entire dataset for a year fits in a few 100s GB uncompressed -- literally could fit on a macbook air for most of the year. The funny thing about "big data" in my experience, is just how small it ac…

Post by sokoloff » Sun, Mar 03, 2019, 12:34 PM UTC

wodenokoto wrote: post #8

Does this add anything to the Taco Bell post linked in TFA? I suggest changing the link to: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...