Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

github.com

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#11

Post by mathatoms »

dandancanfly wrote: post #9
mathatoms wrote: post #8

Earlier quoted context omitted.

We were planning on writing up a blog post to go over what our backend looks like. But essentially we have written a crawler to discover audio on the internet and a distributed processing framework to download, extract metadata, and transcribe the audio. We've iterated through a few storage solutions and have settled on using GlusterFS+zfs running on Storinators. So far we have about 350TB of data indexed in our coll…

That's pretty neat. After you download the audio and process it, do you delete the data, or store it for safe keeping? 350TB is a healthy chunk of data.

We have enough storage to hold on to the data. We keep the data around so we can retranscribe files as we update our language models.

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#12

Post by nibbleshift »

dandancanfly wrote: post #4
Uffizi wrote: post #3

I like the ability to skip though the audio stream to your search term locations within it, pretty cool. Nice work!

Yeah! This looks like a promising way to quickly search many podcasts/audio streams at once. I wish I had this tool for my college research

If you haven't stumbled across it yet, you can check out the FluidDATA web search that let's you search millions of podcasts by phrase or mention here https://fluiddata.com/

You can register here https://accounts.bitplatter.com/?next=https://fluiddata.com/ to get 100 free searches per month.

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#13

Post by mathatoms »

chatmasta wrote: post #10
mathatoms wrote: post #8

Earlier quoted context omitted.

We were planning on writing up a blog post to go over what our backend looks like. But essentially we have written a crawler to discover audio on the internet and a distributed processing framework to download, extract metadata, and transcribe the audio. We've iterated through a few storage solutions and have settled on using GlusterFS+zfs running on Storinators. So far we have about 350TB of data indexed in our coll…

Wow! Sounds awesome, I would love to read a blog post on that. Are you co-locating the hardware? What is bandwidth pricing like?

Thanks. Our blog is located at https://blog.bitplatter.com.

We are co-locating some of our infrastructure. The backend that does the data processing is running in a rack on our own hardware. The user facing portions are hosted in GCE.

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#14

Post by Uffizi »

nibbleshift wrote: post #12
dandancanfly wrote: post #4

Earlier quoted context omitted.

Yeah! This looks like a promising way to quickly search many podcasts/audio streams at once. I wish I had this tool for my college research

If you haven't stumbled across it yet, you can check out the FluidDATA web search that let's you search millions of podcasts by phrase or mention here https://fluiddata.com/ You can register here https://accounts.bitplatter.com/?next=https://fluiddata.com/ to get 100 free searches per month.

Thanks, I just messed around with it a bit and enjoyed the discovery. You all seem to have a ton of content across the web processed, it's very interesting.

Re: Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

#16

Post by dandancanfly »

shirman wrote: post #15

Btw, what is the best way to search trought the audio files? For example if I know for what sound am I searching for

Are you looking for a specific sound, or is it a word in the English language? These cats can help if it's a word in the English language: I'm not sure if they can search for specific sounds, although I'm sure it's possible down the road.
Post reply on HN