Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Analyzing Your Browser History Using Python and Pandas

applecrazy.github.io

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Analyzing Your Browser History Using Python and Pandas

#22

Post by applecrazy »

delta1 wrote: post #17
applecrazy wrote: post #3

Earlier quoted context omitted.

Author of the post here. The main purpose of the post was to demonstrate collection and cleaning of data and give an overview of it through a basic visualization. In the future, I'd like to show how this data is a gold mine of information, using it to predict browsing trends, create a profile of interests, and more. Mainly to show why ad tracking/selling of user browser history is bad, but also to teach some data sci…

OT: which Hugo theme are you using? Custom?

I'm using the Minos Hugo theme.

Re: Analyzing Your Browser History Using Python and Pandas

#23

Post by leblancfg »

There's a video (now pay-walled) from Cameron Davidson Pilon of Bayesian Methods for Hackers fame, where he takes this approach to build a basic Markov Chain model to predict patterns in his browsing.

https://dataorigami.net/collections/all/products/create-mark...

Re: Analyzing Your Browser History Using Python and Pandas

#24

Post by applecrazy »

rahiel wrote: post #13

These two loops: raw_data = [line.split('|', 1) for line in [x.strip() for x in content]] can be simplified to a single loop: raw_data = [line.strip().split('|', 1) for line in content] Using str.replace here is also non-idiomatic: plt.title('Top $n Sites Visited'.replace('$n', str(topN))) How about using str.format instead: plt.title('Top {n} Sites Visited'.format(n=topN))

thanks! The code in my post is mainly more of a POC than anything else, but in my next post I’ll try to be more Pythonic.

Re: Analyzing Your Browser History Using Python and Pandas

#26

Post by applecrazy »

drej wrote: post #8

It seems a bit odd to export the data in the command line to then work with it in pandas, given that it's a stock sqlite3 database. I'd just use sqlite3.connect and pd.read_sql_query directly. (I know, the db will be locked when Chrome is running, but you can always shutil.copy it to a temp location.)

Hmm something to keep in mind for next week’s post.

Re: Analyzing Your Browser History Using Python and Pandas

#27

Post by applecrazy »

daKexx wrote: post #20

I don't think that the pulled data is *extremely messy(. You can also read `hist.txt` with: pd.read_csv('hist.txt', sep='|', header=None)

Not quite. I tried that but some of the entries had vertical bars in URLs (Amazon urls), making Pandas complain and refuse to import the data.

Re: Analyzing Your Browser History Using Python and Pandas

#28

Post by daKexx »

applecrazy wrote: post #27
daKexx wrote: post #20

I don't think that the pulled data is *extremely messy(. You can also read `hist.txt` with: pd.read_csv('hist.txt', sep='|', header=None)

Not quite. I tried that but some of the entries had vertical bars in URLs (Amazon urls), making Pandas complain and refuse to import the data.

Ahh I see. Then you should maybe us the Amazon example in your post to clarify why you are using this approach. Nice post by the way.

Re: Analyzing Your Browser History Using Python and Pandas

#29

Post by applecrazy »

spondyl wrote: post #9

I'm learning Python currently and this is the sorta stuff that's neat to see! I haven't had any reason to use pandas yet so it'll be a good excuse. It's a project that's not very long, requires data everyone has and gives a cool insight at the end. Nice!

Thanks for the kind words! Be sure to stay tuned next week, where we’ll be using the same data to predict browsing patterns
Post reply on HN