Getting all your data out of Google Reader

#1

Post by kellegous » Fri, Jun 28, 2013, 2:35 PM UTC

Getting all your data out of Google Reader
blog.persistent.info

Re: Getting all your data out of Google Reader

#2

Post by sp332 » Fri, Jun 28, 2013, 2:42 PM UTC

ArchiveTeam is extracting all the data from Google Reader and uploading it to the Internet Archive. Help out by submitting your OPML file: https://news.ycombinator.com/item?id=5958119

Re: Getting all your data out of Google Reader

#3

Post by skilesare » Fri, Jun 28, 2013, 3:33 PM UTC

If this does what I think it does(And it seems to be doing it now on my machine), then this is truly, truly awesome.

Thank you. mihaip, if you are ever in Houston I will buy you a beer/ and or a steak dinner.

Re: Getting all your data out of Google Reader

#4

Post by Udo » Fri, Jun 28, 2013, 3:33 PM UTC

This is excellent, thank you for making this! I'm using it right now to make an offline archive of my Reader stuff.

My only gripe would be the tool's inability to continue after a partial run, but since I won't be using this more than once that's probably OK.

All web services should have a handy CLI extraction tool, preferably one that can be run from a CRON call. On that note, I'm very happy with gm_vault, as well.

Edit: getting a lot of XML parse errors, by the way.

Re: Getting all your data out of Google Reader

#5

Post by mihaip » Fri, Jun 28, 2013, 3:52 PM UTC

Udo wrote: post #4

This is excellent, thank you for making this! I'm using it right now to make an offline archive of my Reader stuff. My only gripe would be the tool's inability to continue after a partial run, but since I won't be using this more than once that's probably OK. All web services should have a handy CLI extraction tool, preferably one that can be run from a CRON call. On that note, I'm very happy with gm_vault, as well.…

The tool caches the API responses (in the _raw_data directory), so if you're re-running it, most of the initial requests will be served from the cache.

If the XML parse errors are listing any item IDs, feel free to email them to me (mihai at persistent dot info) and I'll see if there's any workaround from my side.

Edit: If it's "XML parse error when fetching items, retrying with high-fidelity turned off" messages that you're seeing, then those are harmless (assuming no follow-up exceptions). The retry must have succeeded.

Re: Getting all your data out of Google Reader

#6

Post by ivank » Fri, Jun 28, 2013, 4:14 PM UTC

mihaip wrote: post #5

Udo wrote: post #4

This is excellent, thank you for making this! I'm using it right now to make an offline archive of my Reader stuff. My only gripe would be the tool's inability to continue after a partial run, but since I won't be using this more than once that's probably OK. All web services should have a handy CLI extraction tool, preferably one that can be run from a CRON call. On that note, I'm very happy with gm_vault, as well.…

The tool caches the API responses (in the _raw_data directory), so if you're re-running it, most of the initial requests will be served from the cache. If the XML parse errors are listing any item IDs, feel free to email them to me (mihai at persistent dot info) and I'll see if there's any workaround from my side. Edit : If it's "XML parse error when fetching items, retrying with high-fidelity turned off" messages th…

Have you tried the JSON API? (See the requests that Google Reader itself makes.) It requires no cookies and supports getting up 1000 items per continuation.

Re: Getting all your data out of Google Reader

#7

Post by mihaip » Fri, Jun 28, 2013, 4:22 PM UTC

ivank wrote: post #6

mihaip wrote: post #5

Earlier quoted context omitted.

The tool caches the API responses (in the _raw_data directory), so if you're re-running it, most of the initial requests will be served from the cache. If the XML parse errors are listing any item IDs, feel free to email them to me (mihai at persistent dot info) and I'll see if there's any workaround from my side. Edit : If it's "XML parse error when fetching items, retrying with high-fidelity turned off" messages th…

Have you tried the JSON API? (See the requests that Google Reader itself makes.) It requires no cookies and supports getting up 1000 items per continuation.

I wrote most of Reader's JSON API in 2006-2007 :)

The tool uses the "high-fidelity" Atom output mode for getting at item bodies. That preserves namespaced XML elements and other extra data from the feed. It uses JSON for everything else, and will fall back to regular Atom output if the high fidelity mode is not well-formed (it was added in late 2010, as things were winding down, and thus never got a lot of testing).

Re: Getting all your data out of Google Reader

#8

Post by nod » Fri, Jun 28, 2013, 5:32 PM UTC

Thanks mihaip!

Worked successfully in Windows CMD for me, without using the \bin shell script:

  cd C:\mihaip-readerisdead
  set PYTHON_HOME=C:\mihaip-readerisdead
  C:\path-to-py27 reader_archive\reader_archive.py --output-directory C:\mystuff

Locked up at 251K out of 253K items for me, though. Restarting... success! Looks like it might have locked up trying to start the "Fetching comments" section on my first try.

Re: Getting all your data out of Google Reader

#9

Post by drivebyacct2 » Fri, Jun 28, 2013, 6:27 PM UTC

I guess archived RSS data for me isn't terribly important since most people seem to hide the rest of their content behind a "More" link to get those precious ad views.

Re: Getting all your data out of Google Reader

#10

Post by kcvv » Fri, Jun 28, 2013, 6:50 PM UTC

nod wrote: post #8

Thanks mihaip! Worked successfully in Windows CMD for me, without using the \bin shell script: cd C:\mihaip-readerisdead set PYTHON_HOME=C:\mihaip-readerisdead C:\path-to-py27 reader_archive\reader_archive.py --output-directory C:\mystuff Locked up at 251K out of 253K items for me, though. Restarting... success! Looks like it might have locked up trying to start the "Fetching comments" section on my first try.

I'm trying this on windows and seem to be missing base.api module. I can't seem to find this module as well - anyone have a clue where i can get this module?