Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A database of everything (over 55M keys)

outpan.com

Post reply on HN
Print view View original

11–20 of 43 posts

Re: Show HN: A database of everything (over 55M keys)

#11

Post by outpan »

outpan wrote: post #10
GuiA wrote: post #8

Signup page is blanking out for me. Really curious to try it. It's a really neat idea. Has this been done before? I've never seen anything like it. The full value of this would likely come from interesting, productive, insightful visualizations of the underlying graph that is being built. Questions that come to my mind: - What if you write bots that scrape Wikipedia, Twitter, etc. and output entries from semantic ana…

I think graph overlap is what actually determines what data is "accurate". There are currently bots writing to the database by people who are not connected and I have yet to see the overlap (since the key space is too large for the number of bots right now). I'm excited to see how that plays out. I will send you an email :)

As for your question about real world applications: Outpan was only a product database up until last week. It is used in over a hundred apps, some with more than a million users.

I expect this to work on a larger key space as well. It is interesting to see how the expansion works out in terms of usage patterns.

Re: Show HN: A database of everything (over 55M keys)

#12

Post by exogen »

GuiA wrote: post #8

Signup page is blanking out for me. Really curious to try it. It's a really neat idea. Has this been done before? I've never seen anything like it. The full value of this would likely come from interesting, productive, insightful visualizations of the underlying graph that is being built. Questions that come to my mind: - What if you write bots that scrape Wikipedia, Twitter, etc. and output entries from semantic ana…

> Has this been done before?

Depends what you mean by "this"! RDF [1] and most of the technology surrounding it and the "Semantic Web" are based on (subject, predicate, object) triples almost exactly like this, where each element is often a URI, and objects are often strings just like they are here.

It even has taken this idea to the next level where the statements expressed by such a triple can themselves be given an "anonymous" ID, which can then be used as a subject or object – meaning you can make meta statements about the statement itself, all while still using this simple system of triples.

There are even entire languages built around querying graphs of such triples: https://www.w3.org/TR/sparql11-query/

DBpedia [1] is one such project that attempts to encode data from Wikipedia in triples like this; their About page says that the 2014 version of the database had 3 billion triples, so that number is probably much higher now. Here's a preview if you want to see what these triples look like:

• Homepages of things: http://downloads.dbpedia.org/preview.php?file=2015-10_sl_cor...

• Genders of things: http://downloads.dbpedia.org/preview.php?file=2015-10_sl_cor...

etc. You'll notice that RDF predicates are all namespaced by URIs; that way you can unambiguously know in what sense "homepage" and "gender" are used (consider more ambiguous properties like "length"). That means there can be other uses of "homepage", "gender", "length" etc. that mean different things, and those will be namespaced by a different URI.

Anyway, this Outpan project is obviously a more loose and freeform version of that – but only slightly; RDF is not very strict at all, it's just that people have thought a lot about how to successfully model the entire world's information, and so real-world RDF ontologies end up looking somewhat complicated. I'm not sure if a freeform version like this has been widely attempted before.

[1] https://en.wikipedia.org/wiki/Resource_Description_Framework [2] http://wiki.dbpedia.org/

Re: Show HN: A database of everything (over 55M keys)

#15

Post by exogen »

exogen wrote: post #12
GuiA wrote: post #8

Signup page is blanking out for me. Really curious to try it. It's a really neat idea. Has this been done before? I've never seen anything like it. The full value of this would likely come from interesting, productive, insightful visualizations of the underlying graph that is being built. Questions that come to my mind: - What if you write bots that scrape Wikipedia, Twitter, etc. and output entries from semantic ana…

> Has this been done before? Depends what you mean by "this"! RDF [1] and most of the technology surrounding it and the "Semantic Web" are based on (subject, predicate, object) triples almost exactly like this, where each element is often a URI, and objects are often strings just like they are here. It even has taken this idea to the next level where the statements expressed by such a triple can themselves be given a…

To borrow a subject matter that's currently popular on the Outpan homepage, here are the first 500 facts DBpedia knows about Donald Trump:

    SELECT DISTINCT ?property ?value WHERE {
         ?property ?value
    } LIMIT 500
Results: http://dbpedia.org/sparql?default-graph-uri=http%3A%2F%2Fdbp... (although, note, not every dataset they have is loaded into their SPARQL endpoint)

As you can see there's a lot of metadata type properties, but scroll down and you can see his birthdate, children, alma mater, etc.

This page is just a prettified version of that data: http://dbpedia.org/page/Donald_Trump

Re: Show HN: A database of everything (over 55M keys)

#16

Post by outpan »

exogen wrote: post #15
exogen wrote: post #12

Earlier quoted context omitted.

> Has this been done before? Depends what you mean by "this"! RDF [1] and most of the technology surrounding it and the "Semantic Web" are based on (subject, predicate, object) triples almost exactly like this, where each element is often a URI, and objects are often strings just like they are here. It even has taken this idea to the next level where the statements expressed by such a triple can themselves be given a…

To borrow a subject matter that's currently popular on the Outpan homepage, here are the first 500 facts DBpedia knows about Donald Trump: SELECT DISTINCT ?property ?value WHERE { ?property ?value } LIMIT 500 Results: http://dbpedia.org/sparql?default-graph-uri=http%3A%2F%2Fdbp... (although, note, not every dataset they have is loaded into their SPARQL endpoint) As you can see there's a lot of metadata type propertie…

This is great! thanks. I will look into adding the dbpedia.org data.

Re: Show HN: A database of everything (over 55M keys)

#17

Post by matrix_decoder »

exogen wrote: post #12
GuiA wrote: post #8

Signup page is blanking out for me. Really curious to try it. It's a really neat idea. Has this been done before? I've never seen anything like it. The full value of this would likely come from interesting, productive, insightful visualizations of the underlying graph that is being built. Questions that come to my mind: - What if you write bots that scrape Wikipedia, Twitter, etc. and output entries from semantic ana…

> Has this been done before? Depends what you mean by "this"! RDF [1] and most of the technology surrounding it and the "Semantic Web" are based on (subject, predicate, object) triples almost exactly like this, where each element is often a URI, and objects are often strings just like they are here. It even has taken this idea to the next level where the statements expressed by such a triple can themselves be given a…

This is exactly what I thought when I saw this project. Having worked with DBPedia data and the triples format; I was just wondering how Outpan just stole the idea of Triples and packaged it as a new idea

Re: Show HN: A database of everything (over 55M keys)

#18

Post by exogen »

outpan wrote: post #16
exogen wrote: post #15

Earlier quoted context omitted.

To borrow a subject matter that's currently popular on the Outpan homepage, here are the first 500 facts DBpedia knows about Donald Trump: SELECT DISTINCT ?property ?value WHERE { ?property ?value } LIMIT 500 Results: http://dbpedia.org/sparql?default-graph-uri=http%3A%2F%2Fdbp... (although, note, not every dataset they have is loaded into their SPARQL endpoint) As you can see there's a lot of metadata type propertie…

This is great! thanks. I will look into adding the dbpedia.org data.

Check out https://www.wikidata.org/ for another similar project with additional data! Their keys tend to be more opaque [1], but otherwise it's a very similar approach.

[1] e.g. the key for "Earth" is Q2: https://www.wikidata.org/wiki/Q2

Re: Show HN: A database of everything (over 55M keys)

#19

Post by outpan »

exogen wrote: post #15
exogen wrote: post #12

Earlier quoted context omitted.

> Has this been done before? Depends what you mean by "this"! RDF [1] and most of the technology surrounding it and the "Semantic Web" are based on (subject, predicate, object) triples almost exactly like this, where each element is often a URI, and objects are often strings just like they are here. It even has taken this idea to the next level where the statements expressed by such a triple can themselves be given a…

To borrow a subject matter that's currently popular on the Outpan homepage, here are the first 500 facts DBpedia knows about Donald Trump: SELECT DISTINCT ?property ?value WHERE { ?property ?value } LIMIT 500 Results: http://dbpedia.org/sparql?default-graph-uri=http%3A%2F%2Fdbp... (although, note, not every dataset they have is loaded into their SPARQL endpoint) As you can see there's a lot of metadata type propertie…

Would you be open to having a little chat via email? hi@outpan.com

Re: Show HN: A database of everything (over 55M keys)

#20

Post by nenadg »

outpan wrote: post #14
erikb wrote: post #13

Is 55 million keys a lot for tracking "everything"? My expectation was that "everything" would need more than 55 trillion keys.

"a journey of a thousand miles begins with a single step"

"a journey of a everything begins with a 55M keys"
Post reply on HN