Full MongoDB database dump of the Blippex search engine

#1

Post by karli » Tue, Jul 23, 2013, 2:21 PM UTC

Full MongoDB database dump of the Blippex search engine
blippex.github.io

Re: Full MongoDB database dump of the Blippex search engine

#2

Post by rgiar » Tue, Jul 23, 2013, 3:18 PM UTC

so this is just when a given site was crawled?

  "_id": "b919f02c8f053c41e8ee86311ca9b0f6,
  "url": "https://www.example.com/",
  "host": "www.example.com",
  "root": "example.com",
  "time_spent": [
    {
      "sec": 45,
      "seen_at": ISODate("2013-06-23T00: 41: 44.0Z")
    },
    {
      "sec": 5,
      "seen_at": ISODate("2013-07-01T14: 41: 44.0Z")
    }

Re: Full MongoDB database dump of the Blippex search engine

#3

Post by mgamache » Tue, Jul 23, 2013, 3:21 PM UTC

So it's a bunch of internet URLs without content or content metadata?

Re: Full MongoDB database dump of the Blippex search engine

#4

Post by karli » Tue, Jul 23, 2013, 3:23 PM UTC

rgiar wrote: post #2

so this is just when a given site was crawled? "_id": "b919f02c8f053c41e8ee86311ca9b0f6, "url": "https://www.example.com/", "host": "www.example.com", "root": "example.com", "time_spent": [ { "sec": 45, "seen_at": ISODate("2013-06-23T00: 41: 44.0Z") }, { "sec": 5, "seen_at": ISODate("2013-07-01T14: 41: 44.0Z") }

Hi,

yes, as it is said in the blogpost, the only thing missing is the full text of the page for indexing & searching in it, we don't dare to release it because of copyright issues (he, you distribute the full text of my page!).

With this data you could for example built a new alexa and find out what was the most visited page last week :)

Re: Full MongoDB database dump of the Blippex search engine

#5

Post by itsmeduncan » Tue, Jul 23, 2013, 3:26 PM UTC

This will be fun as a list of places to try out 0-day exploits on.

Re: Full MongoDB database dump of the Blippex search engine

#6

Post by enigmo » Tue, Jul 23, 2013, 3:30 PM UTC

karli wrote: post #4

rgiar wrote: post #2

so this is just when a given site was crawled? "_id": "b919f02c8f053c41e8ee86311ca9b0f6, "url": "https://www.example.com/", "host": "www.example.com", "root": "example.com", "time_spent": [ { "sec": 45, "seen_at": ISODate("2013-06-23T00: 41: 44.0Z") }, { "sec": 5, "seen_at": ISODate("2013-07-01T14: 41: 44.0Z") }

Hi, yes, as it is said in the blogpost, the only thing missing is the full text of the page for indexing & searching in it, we don't dare to release it because of copyright issues (he, you distribute the full text of my page!). With this data you could for example built a new alexa and find out what was the most visited page last week :)

How would we figure out which page was visited the most last week? Are these crawl logs or access logs?

Re: Full MongoDB database dump of the Blippex search engine

#7

Post by bkanber » Tue, Jul 23, 2013, 3:32 PM UTC

mgamache wrote: post #3

So it's a bunch of internet URLs without content or content metadata?

Seems that way. I mean, that's still be valuable and interesting for other reasons, but let's not call it a "dump of a search engine". There's nothing in there that's actually searchable!

Still, nice gesture by Blippex. Somebody will find something interesting to do with this, even if they just use it educationally.

Re: Full MongoDB database dump of the Blippex search engine

#8

Post by mgamache » Tue, Jul 23, 2013, 3:35 PM UTC

enigmo wrote: post #6

karli wrote: post #4

Earlier quoted context omitted.

Hi, yes, as it is said in the blogpost, the only thing missing is the full text of the page for indexing & searching in it, we don't dare to release it because of copyright issues (he, you distribute the full text of my page!). With this data you could for example built a new alexa and find out what was the most visited page last week :)

How would we figure out which page was visited the most last week? Are these crawl logs or access logs?

It would be crazy cool to get a real-time feed of browsed URLs (not this dump format). Kind of like the mythical Twitter fire-hose.

Re: Full MongoDB database dump of the Blippex search engine

#9

Post by haldujai » Tue, Jul 23, 2013, 3:39 PM UTC

enigmo wrote: post #6

karli wrote: post #4

Earlier quoted context omitted.

Hi, yes, as it is said in the blogpost, the only thing missing is the full text of the page for indexing & searching in it, we don't dare to release it because of copyright issues (he, you distribute the full text of my page!). With this data you could for example built a new alexa and find out what was the most visited page last week :)

How would we figure out which page was visited the most last week? Are these crawl logs or access logs?

From blippex.org:

'Blippex is a search engine by the people, for the people. Individuals that have our browser extension installed tell us how long they stayed on a webpage.'

I haven't had a chance to download the dump yet but I'm assuming that the time points are time spent on the website by users.

Re: Full MongoDB database dump of the Blippex search engine

#10

Post by geraldbaeck » Tue, Jul 23, 2013, 3:46 PM UTC

mgamache wrote: post #8

enigmo wrote: post #6

Earlier quoted context omitted.

How would we figure out which page was visited the most last week? Are these crawl logs or access logs?

It would be crazy cool to get a real-time feed of browsed URLs (not this dump format). Kind of like the mythical Twitter fire-hose.

Nice idea I think we should do that.

Gerald, CTO blippex