Great job with Marginalia. Do you plan to open source your data as well or only the code?
If I can solve the logistics, maybe. I don't have the needed bandwidth and off-site storage at this point.
Marginalia Goes Open Source
71–75 of 75 posts
Re: Marginalia Goes Open Source
#72Earlier quoted context omitted.
What sort of sizes are we talking about? I'm thinking if it would be possible to "crowdfund" the storage-costs for a requester-pays s3 bucket for it.
I'm probably producing around 250-500 Gb data/month at this point.
If it's something that compresses really well (eg text data in a database), then live compression filesystems (eg ZFS, likely others) could potentially help make that workable.
Re: Marginalia Goes Open Source
#73Earlier quoted context omitted.
What sort of sizes are we talking about? I'm thinking if it would be possible to "crowdfund" the storage-costs for a requester-pays s3 bucket for it.
I'm probably producing around 250-500 Gb data/month at this point.
He might know some folks.
Re: Marginalia Goes Open Source
#74Earlier quoted context omitted.
I'm probably producing around 250-500 Gb data/month at this point.
Any idea how compressible that is? If it's something that compresses really well (eg text data in a database), then live compression filesystems (eg ZFS, likely others) could potentially help make that workable.
Re: Marginalia Goes Open Source
#75> I'm currently looking for hosting for a large term frequency data file that is necessary for several of the search engine's core functions. I really don't have the bandwidth to serve it myself. It's only a couple of hundred megabytes so it'll probably be solvable somehow. I have a bit of (rented) space available. Most of it on Uberspace. What are the expected traffic requirements? Contact me if you like. Will updat…