Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: TidalWaves API – live, tokenized news metadata from around the world

tidalwaves.io

Post reply on HN
Print view View original

11–20 of 47 posts

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#11

Post by chris_f »

prohobo wrote: post #6
awestroke wrote: post #5

Earlier quoted context omitted.

If they have the exact same title, perhaps you can skip having a deep neural net and just do a string comparison? Might be sligthtly faster ;) Finding duplicate stories that have different texts and titles sounds like something you should use AI for

I'm definitely thinking of ways to improve it for long-term stories. You're right, fuzzy string matching would do wonders for that, thanks :)

I've used trigram matching of article title strings with some success for comparing similarity of news article. If the strings are greater than .6 they are probably about the same topic.

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#12

Post by wombatmobile »

prohobo wrote: post #10
wombatmobile wrote: post #9

What are the 1000+ news sources? Is there a list of publications? e.g. New York Times, The Australian, Bangkok Post... ? I understand your data comes from GDELT. I'm new to that and all I could glean from visiting the GDELT site at first go was that they have access to a lot of historical data sources - but I couldn't see a list of 1000+ current news publications. Is there a list somewhere? Are all the sources typogr…

They're all online written news sources, but I believe GDELT is also getting into TV news so that might be coming in the future. There is no official list currently, but you could query either GDELT or TidalWaves to see which are available in the data sets. I could add a page to the marketing website which displays all of them, instead of the top sources for the latest news - would that be something useful for people…

Thanks for replying. I'm not sure what is useful yet - first of all I am trying to understand the scope of the data sources and the methodology for selections displayed in TidalWaves.

I would like to see the list of publications if that is possible.

I am wondering what "trending stories" means in TidalWaves? For example, if the NYT at any point contains 500 stories, and is refreshed say 3 times every 24 hours, with say 100 stories aged off and 100 new stories added each addition, what is a trending story in the NYT? Does the front page headline count more than a page 27 minor traffic report?

How is the top 5 trending stories for USA derived?

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#13

Post by arusahni »

As someone who has built event pipelines from GDELT data, I'd advise folks to not start (a GDELT project) with high expectations. Not only are there plenty of duplicates, but tagging quality is all over the place, especially for non-Western media.

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#14

Post by artembugara »

Nice project. I actually did pretty much the same a long time ago: an API around GDELT data.

Your map looks fantastic, btw.

The problem with GDELT is that they don't have that much coverage. I discovered it when we began to build our own News API [1].

I failed to sell this data over an API. I think you should see much more interest for an application. API like that is a bit complex. Almost all potential clients who really needed GDELT already parsing this data from Big Query, Redshift, or GDELT's dump files.

Anyway, check our solution, probably we could collaborate somehow. Map is cool! Feel free to reach me over artem [at] newscatcherapi [dot] com

[1] https://newscatcherapi.com

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#15

Post by artembugara »

arusahni wrote: post #13

As someone who has built event pipelines from GDELT data, I'd advise folks to not start (a GDELT project) with high expectations. Not only are there plenty of duplicates, but tagging quality is all over the place, especially for non-Western media.

I worked with GELT data for a long time. Same feedback.

What are you using instead of GDELT now?

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#18

Post by prohobo »

wombatmobile wrote: post #12
prohobo wrote: post #10

Earlier quoted context omitted.

They're all online written news sources, but I believe GDELT is also getting into TV news so that might be coming in the future. There is no official list currently, but you could query either GDELT or TidalWaves to see which are available in the data sets. I could add a page to the marketing website which displays all of them, instead of the top sources for the latest news - would that be something useful for people…

Thanks for replying. I'm not sure what is useful yet - first of all I am trying to understand the scope of the data sources and the methodology for selections displayed in TidalWaves. I would like to see the list of publications if that is possible. I am wondering what "trending stories" means in TidalWaves? For example, if the NYT at any point contains 500 stories, and is refreshed say 3 times every 24 hours, with s…

I just did a quick query grouping all sources and number of articles per source in the database (currently about 4 days worth of data): https://pastebin.com/HyktFpML

As you can see, there are already almost 8000 sources. I'm honestly not sure how GDELT scrapes its data though, but this is what they do for over 5 years now and it's backed by Google Cloud processing, so I'm sure it's ever expanding and very thorough.

The kinds of articles which are discovered revolve around world events, however. So you won't see random blog posts, or traffic reports. Instead you'll see geopolitics, big tech news, and cultural discourse.

Trending stories are found for the entire set of articles in a batch, with no separation for source, category, or location. If the top stories all happen to be in the USA in a given moment, then that's all you'll find. If NYT happens to publish an article that fits an overall story being talked about in the rest of the batch, it'll get added to that story.

Trending popularity is decided by how many individual sources talk about the given topic. This is good enough to identify the Zeitgeist of a moment, but it likely won't catch nuances in evolving stories.

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#19

Post by chris_f »

arusahni wrote: post #13

As someone who has built event pipelines from GDELT data, I'd advise folks to not start (a GDELT project) with high expectations. Not only are there plenty of duplicates, but tagging quality is all over the place, especially for non-Western media.

The one nice thing about GDELT is it translates non-english language news so that you can use an english search term and see articles in different language. It's not perfect (i.e. machine translation), but I am not aware of any other service that provides similar functionality.

More info: https://blog.gdeltproject.org/gdelt-translingual-translating...

(unrelated: scrolling on that page is horrible)

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#20

Post by arusahni »

artembugara wrote: post #15
arusahni wrote: post #13

As someone who has built event pipelines from GDELT data, I'd advise folks to not start (a GDELT project) with high expectations. Not only are there plenty of duplicates, but tagging quality is all over the place, especially for non-Western media.

I worked with GELT data for a long time. Same feedback. What are you using instead of GDELT now?

That was with a previous org, so thankfully not something I have to worry about now.

Our domain was pretty well defined, and we weren't building a real-time system, so we developed an adjudication tool and hired a few part time employees to evaluate and clean up our output.

Post reply on HN