Show HN: FluidDATA API – Create your own audio search engine with FluidDATA • Hacker News Board

Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#1

Post by mathatoms » Tue, Apr 10, 2018, 1:14 PM UTC

Show HN: FluidDATA API – Create your own audio search engine with FluidDATA
github.com

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#2

Post by bmelton » Tue, Apr 10, 2018, 1:33 PM UTC

Oh hi!

Despite the green user, I've actually worked with these guys and seen this project up close and in person, and the amount of data they're harvesting here is pretty overwhelming.

There are a few other places doing transcription nowadays, but they're just doing that, and this is a bit richer an API for getting more tailoring done against your source data.

Either way, if you're looking for a way to add audio transcription to your podcast or vlog, this is a cool service. If you're looking to make that audio searchable with the fewest number of steps, this is probably the coolest service around.

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#3

Post by Uffizi » Tue, Apr 10, 2018, 1:37 PM UTC

I like the ability to skip though the audio stream to your search term locations within it, pretty cool. Nice work!

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#4

Post by dandancanfly » Tue, Apr 10, 2018, 1:58 PM UTC

Uffizi wrote: post #3

I like the ability to skip though the audio stream to your search term locations within it, pretty cool. Nice work!

Yeah! This looks like a promising way to quickly search many podcasts/audio streams at once. I wish I had this tool for my college research

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#5

Post by chatmasta » Tue, Apr 10, 2018, 2:55 PM UTC

Cool project, and a mammoth undertaking in terms of scraping and data processing. Would you be able to share any details on what your ingestion infrastructure looks like?

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#6

Post by Nadya » Tue, Apr 10, 2018, 2:57 PM UTC

This is actually really cool! I'm guessing it is English only though? I don't see examples of any other languages and due to the complexity of word->audio matching I imagine other languages aren't supported.

I think a better title would be "Audio search engine for Podcasts using FluidDATA". It had briefly gotten my hopes up that I'd be able to make a search engine for my music, based just on the title.

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#7

Post by mathatoms » Tue, Apr 10, 2018, 3:11 PM UTC

Nadya wrote: post #6

This is actually really cool! I'm guessing it is English only though? I don't see examples of any other languages and due to the complexity of word->audio matching I imagine other languages aren't supported. I think a better title would be "Audio search engine for Podcasts using FluidDATA". It had briefly gotten my hopes up that I'd be able to make a search engine for my music, based just on the title.

Thanks. Its English only for now. We could support other languages if there is a large enough demand.

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#8

Post by mathatoms » Tue, Apr 10, 2018, 3:24 PM UTC

chatmasta wrote: post #5

Cool project, and a mammoth undertaking in terms of scraping and data processing. Would you be able to share any details on what your ingestion infrastructure looks like?

We were planning on writing up a blog post to go over what our backend looks like. But essentially we have written a crawler to discover audio on the internet and a distributed processing framework to download, extract metadata, and transcribe the audio.

We've iterated through a few storage solutions and have settled on using GlusterFS+zfs running on Storinators. So far we have about 350TB of data indexed in our collection.

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#9

Post by dandancanfly » Tue, Apr 10, 2018, 3:28 PM UTC

mathatoms wrote: post #8

chatmasta wrote: post #5

Cool project, and a mammoth undertaking in terms of scraping and data processing. Would you be able to share any details on what your ingestion infrastructure looks like?

We were planning on writing up a blog post to go over what our backend looks like. But essentially we have written a crawler to discover audio on the internet and a distributed processing framework to download, extract metadata, and transcribe the audio. We've iterated through a few storage solutions and have settled on using GlusterFS+zfs running on Storinators. So far we have about 350TB of data indexed in our coll…

That's pretty neat. After you download the audio and process it, do you delete the data, or store it for safe keeping? 350TB is a healthy chunk of data.

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#10

Post by chatmasta » Tue, Apr 10, 2018, 3:30 PM UTC

mathatoms wrote: post #8

chatmasta wrote: post #5

Cool project, and a mammoth undertaking in terms of scraping and data processing. Would you be able to share any details on what your ingestion infrastructure looks like?

We were planning on writing up a blog post to go over what our backend looks like. But essentially we have written a crawler to discover audio on the internet and a distributed processing framework to download, extract metadata, and transcribe the audio. We've iterated through a few storage solutions and have settled on using GlusterFS+zfs running on Storinators. So far we have about 350TB of data indexed in our coll…

Wow! Sounds awesome, I would love to read a blog post on that.

Are you co-locating the hardware? What is bandwidth pricing like?