Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Analyzing Your Browser History Using Python and Pandas

applecrazy.github.io

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Analyzing Your Browser History Using Python and Pandas

#12

Post by whym »

Neat. As a Firefox user, it appears that this does the trick:

    sqlite3 ~/.mozilla/firefox/$YOUR_PROFILE_ID/places.sqlite "SELECT datetime(visit_date/1000000,'unixepoch'),url FROM moz_historyvisits JOIN moz_places ON place_id=moz_places.id ORDER BY visit_date DESC" > hist.txt
(Looks like the timestamp is in UTC here.)

----

Does the original post's code perhaps count two visits to the same URL as one? Something like

    SELECT datetime(visit_time/1000000-11644473600,'unixepoch'),urls.url 
    FROM visits JOIN urls ON visits.url=urls.id
    ORDER BY visit_time DESC;
might be worth a try.

Re: Analyzing Your Browser History Using Python and Pandas

#13

Post by rahiel »

These two loops:

  raw_data = [line.split('|', 1) for line in [x.strip() for x in content]]
can be simplified to a single loop:

  raw_data = [line.strip().split('|', 1) for line in content]
Using str.replace here is also non-idiomatic:

  plt.title('Top $n Sites Visited'.replace('$n', str(topN)))
How about using str.format instead:

  plt.title('Top {n} Sites Visited'.format(n=topN))

Re: Analyzing Your Browser History Using Python and Pandas

#14

Post by cdubzzz »

spondyl wrote: post #9

I'm learning Python currently and this is the sorta stuff that's neat to see! I haven't had any reason to use pandas yet so it'll be a good excuse. It's a project that's not very long, requires data everyone has and gives a cool insight at the end. Nice!

Also learning Python and intending to learn about pandas and other data tools with this: https://github.com/jakevdp/PythonDataScienceHandbook

I haven’t made it very far yet, but so far it seems worth recommending.

Re: Analyzing Your Browser History Using Python and Pandas

#16

Post by hultner »

rahiel wrote: post #13

These two loops: raw_data = [line.split('|', 1) for line in [x.strip() for x in content]] can be simplified to a single loop: raw_data = [line.strip().split('|', 1) for line in content] Using str.replace here is also non-idiomatic: plt.title('Top $n Sites Visited'.replace('$n', str(topN))) How about using str.format instead: plt.title('Top {n} Sites Visited'.format(n=topN))

You could even use the new format strings from Python3.6.

  f"Top {topN} Sites Visited"

Re: Analyzing Your Browser History Using Python and Pandas

#17

Post by delta1 »

applecrazy wrote: post #3
ReverseCold wrote: post #2

Vivaldi does this for me in the history page itself. Pretty weird to use Python just to see a distribution of what sites you visited. Iirc stock chrome also has a way to view this.

Author of the post here. The main purpose of the post was to demonstrate collection and cleaning of data and give an overview of it through a basic visualization. In the future, I'd like to show how this data is a gold mine of information, using it to predict browsing trends, create a profile of interests, and more. Mainly to show why ad tracking/selling of user browser history is bad, but also to teach some data sci…

OT: which Hugo theme are you using? Custom?

Re: Analyzing Your Browser History Using Python and Pandas

#18

Post by david_blitz1 »

whym wrote: post #12

Neat. As a Firefox user, it appears that this does the trick: sqlite3 ~/.mozilla/firefox/$YOUR_PROFILE_ID/places.sqlite "SELECT datetime(visit_date/1000000,'unixepoch'),url FROM moz_historyvisits JOIN moz_places ON place_id=moz_places.id ORDER BY visit_date DESC" > hist.txt (Looks like the timestamp is in UTC here.) ---- Does the original post's code perhaps count two visits to the same URL as one? Something like SEL…

Thanks. As a not so savvy Firefox user this was just what I needed.

Re: Analyzing Your Browser History Using Python and Pandas

#19

Post by nerdponx »

whym wrote: post #12

Neat. As a Firefox user, it appears that this does the trick: sqlite3 ~/.mozilla/firefox/$YOUR_PROFILE_ID/places.sqlite "SELECT datetime(visit_date/1000000,'unixepoch'),url FROM moz_historyvisits JOIN moz_places ON place_id=moz_places.id ORDER BY visit_date DESC" > hist.txt (Looks like the timestamp is in UTC here.) ---- Does the original post's code perhaps count two visits to the same URL as one? Something like SEL…

Now if only Buku [0] used this to poll and automatically synchronize my bookmarks...

[0]: https://github.com/jarun/Buku

Post reply on HN