Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Analyzing Your Browser History Using Python and Pandas

applecrazy.github.io

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Analyzing Your Browser History Using Python and Pandas

#33

Post by applecrazy »

daKexx wrote: post #28
applecrazy wrote: post #27

Earlier quoted context omitted.

Not quite. I tried that but some of the entries had vertical bars in URLs (Amazon urls), making Pandas complain and refuse to import the data.

Ahh I see. Then you should maybe us the Amazon example in your post to clarify why you are using this approach. Nice post by the way.

Thanks for the feedback. Will edit my post soon.

Re: Analyzing Your Browser History Using Python and Pandas

#34

Post by 7ewis »

Just tried this, but am getting an error with the datetime. How could I fix this?

  Traceback (most recent call last):
    File "hist.py", line 14, in 
      data.datetime = pd.to_datetime(data.datetime)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py", line 373, in to_datetime
      values = _convert_listlike(arg._values, True, format)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py", line 306, in _convert_listlike
      raise e
  pandas._libs.tslib.OutOfBoundsDatetime: Out of bounds 
  nanosecond timestamp: 1601-01-01 00:00:00
The time on my Mac is correct, and datetime gets the correct time?

Re: Analyzing Your Browser History Using Python and Pandas

#35

Post by applecrazy »

7ewis wrote: post #34

Just tried this, but am getting an error with the datetime. How could I fix this? Traceback (most recent call last): File "hist.py", line 14, in data.datetime = pd.to_datetime(data.datetime) File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py", line 373, in to_datetime values = _convert_listlike(arg._values, True, format) File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes…

Aah I got this strange issue too. Search for that 1601-01-01 time stamp in the exported text file and change it to something else that isn’t before the Unix epoch

Re: Analyzing Your Browser History Using Python and Pandas

#36

Post by 7ewis »

applecrazy wrote: post #35
7ewis wrote: post #34

Just tried this, but am getting an error with the datetime. How could I fix this? Traceback (most recent call last): File "hist.py", line 14, in data.datetime = pd.to_datetime(data.datetime) File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py", line 373, in to_datetime values = _convert_listlike(arg._values, True, format) File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes…

Aah I got this strange issue too. Search for that 1601-01-01 time stamp in the exported text file and change it to something else that isn’t before the Unix epoch

That sorted it, cheers!

How long is the Chrome database kept for? I use Chrome Canary and the standard Chrome, but only had a few hundred results in each, a lot less than I was expecting. I can't remember the last time I cleared my history either.

Cool graph though anyway!

Re: Analyzing Your Browser History Using Python and Pandas

#37

Post by d0mine »

applecrazy wrote: post #35
7ewis wrote: post #34

Just tried this, but am getting an error with the datetime. How could I fix this? Traceback (most recent call last): File "hist.py", line 14, in data.datetime = pd.to_datetime(data.datetime) File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py", line 373, in to_datetime values = _convert_listlike(arg._values, True, format) File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes…

Aah I got this strange issue too. Search for that 1601-01-01 time stamp in the exported text file and change it to something else that isn’t before the Unix epoch

e.g.:

  data.loc[data.datetime == '1601-01-01 00:00:00', 'datetime'] = '1970-01-01 00:00:00'

Re: Analyzing Your Browser History Using Python and Pandas

#38

Post by xalava »

applecrazy wrote: post #35
7ewis wrote: post #34

Just tried this, but am getting an error with the datetime. How could I fix this? Traceback (most recent call last): File "hist.py", line 14, in data.datetime = pd.to_datetime(data.datetime) File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py", line 373, in to_datetime values = _convert_listlike(arg._values, True, format) File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes…

Aah I got this strange issue too. Search for that 1601-01-01 time stamp in the exported text file and change it to something else that isn’t before the Unix epoch

You can automate this by changing the query line to : sqlite3 data/ChromeHistory "SELECT datetime(last_visit_time/1000000-11644473600,'unixepoch','localtime'),url FROM urls WHERE last_visit_time > 0 ORDER BY last_visit_time desc" > data/hist.txt

Re: Analyzing Your Browser History Using Python and Pandas

#39

Post by applecrazy »

7ewis wrote: post #36
applecrazy wrote: post #35

Earlier quoted context omitted.

Aah I got this strange issue too. Search for that 1601-01-01 time stamp in the exported text file and change it to something else that isn’t before the Unix epoch

That sorted it, cheers! How long is the Chrome database kept for? I use Chrome Canary and the standard Chrome, but only had a few hundred results in each, a lot less than I was expecting. I can't remember the last time I cleared my history either. Cool graph though anyway!

I’m not sure how long the history is stored, but I had over 2k entries for six months of history.

And, by the way, I’m not done with this series. Next week I’ll try to predict browsing patterns using this dataset.

Re: Analyzing Your Browser History Using Python and Pandas

#40

Post by schemathings »

I have a far more mundane use case - you have friends over, play songs off of youtube all night, after the fact, would like to publish a playlist on youtube, but it's not simple using browser history. I always figured I'd have to write a plugin and find clusters of short youtube sessions and go from there, but as a backend guy, I might play around with this stack instead.
Post reply on HN