Show HN: TidalWaves API – live, tokenized news metadata from around the world
1–10 of 47 posts
Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world
#2I spent 3 months building a data pipeline for parsing and transforming raw data from the GDELT project, as well as applying a machine learning model to find breaking news stories, which turned out to be a pretty nifty tool for creating visualizations and data mining applications.
So, I built an API around it and released it on Rapid API today.
I'm hoping my pricing isn't too high.. It's currently set so that a Tier 1 sub can create the marketing site I linked, and Tier 2 subs can access historical data as well. But, I also hope to create a steady income so I can work on other projects in the future.
Hope you guys like it, and please send your comments and crits.
Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world
#3There are duplicates on the list of 4 breaking stories. They have the exact same title, too.
Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world
#4There are duplicates on the list of 4 breaking stories. They have the exact same title, too.
Yep, currently the breaking stories are identified in the 15 minute intervals, so all the duplicate stories showing are from different times. If they have the same title that means that I can train the model to combine them.
They also could be press releases, considering the same title was found for multiple articles over the past 30-45 minutes. The data pipeline currently removes duplicate titles, but only inside single batches. So, it could be a long sustained press release.. like propaganda.
As of writing it shows three different stories though:
- 12:15 Will there be a coronavirus relief bill with $1,200 checks?
- 12:30 Nigeria protesters break curfew amid gunfire, chaos in Lagos
- 12:30 Google antitrust case to turn on how search engine grew dominant
Maybe what you're seeing is a bug in the website client, I'll investigate it.
Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world
#5There are duplicates on the list of 4 breaking stories. They have the exact same title, too.
Yep, currently the breaking stories are identified in the 15 minute intervals, so all the duplicate stories showing are from different times. If they have the same title that means that I can train the model to combine them. They also could be press releases, considering the same title was found for multiple articles over the past 30-45 minutes. The data pipeline currently removes duplicate titles, but only inside si…
If they have the exact same title, perhaps you can skip having a deep neural net and just do a string comparison? Might be sligthtly faster ;)
Finding duplicate stories that have different texts and titles sounds like something you should use AI for
Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world
#6Earlier quoted context omitted.
Yep, currently the breaking stories are identified in the 15 minute intervals, so all the duplicate stories showing are from different times. If they have the same title that means that I can train the model to combine them. They also could be press releases, considering the same title was found for multiple articles over the past 30-45 minutes. The data pipeline currently removes duplicate titles, but only inside si…
If they have the exact same title, perhaps you can skip having a deep neural net and just do a string comparison? Might be sligthtly faster ;) Finding duplicate stories that have different texts and titles sounds like something you should use AI for
I'm definitely thinking of ways to improve it for long-term stories. You're right, fuzzy string matching would do wonders for that, thanks :)
Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world
#7There are duplicates on the list of 4 breaking stories. They have the exact same title, too.
Yep, currently the breaking stories are identified in the 15 minute intervals, so all the duplicate stories showing are from different times. If they have the same title that means that I can train the model to combine them. They also could be press releases, considering the same title was found for multiple articles over the past 30-45 minutes. The data pipeline currently removes duplicate titles, but only inside si…
There are news about Pakistan and Hungary in Poland.
Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world
#8Earlier quoted context omitted.
Yep, currently the breaking stories are identified in the 15 minute intervals, so all the duplicate stories showing are from different times. If they have the same title that means that I can train the model to combine them. They also could be press releases, considering the same title was found for multiple articles over the past 30-45 minutes. The data pipeline currently removes duplicate titles, but only inside si…
There are news about Pakistan and Hungary in Poland.
Most news articles mention multiple locations. It's probable that those pieces talked about both Pakistan/Hungary and Poland, so will be found in both countries.
The locations are ordered by salience (proximity to start of the article, number of times mentioned), but not completely filtered, so on this specific visualization you will see a lot of matched locations which you wouldn't consider part of the article, but were mentioned in passing.
Trying to understand the content of an article is still really, really difficult, and error prone.
Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world
#9What are the 1000+ news sources? Is there a list of publications? e.g. New York Times, The Australian, Bangkok Post... ?
I understand your data comes from GDELT. I'm new to that and all I could glean from visiting the GDELT site at first go was that they have access to a lot of historical data sources - but I couldn't see a list of 1000+ current news publications. Is there a list somewhere? Are all the sources typographical, or might some be radio/tv? Thank you.
Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world
#10What are the 1000+ news sources? Is there a list of publications? e.g. New York Times, The Australian, Bangkok Post... ? I understand your data comes from GDELT. I'm new to that and all I could glean from visiting the GDELT site at first go was that they have access to a lot of historical data sources - but I couldn't see a list of 1000+ current news publications. Is there a list somewhere? Are all the sources typogr…
They're all online written news sources, but I believe GDELT is also getting into TV news so that might be coming in the future.
There is no official list currently, but you could query either GDELT or TidalWaves to see which are available in the data sets.
I could add a page to the marketing website which displays all of them, instead of the top sources for the latest news - would that be something useful for people?