Live data from Hacker News

Marginalia Goes Open Source

memex.marginalia.nu

71–75 of 75 posts

Re: Marginalia Goes Open Source

#71

Great job with Marginalia. Do you plan to open source your data as well or only the code?

If I can solve the logistics, maybe. I don't have the needed bandwidth and off-site storage at this point.

Idea: Host datasets as one or more torrents. Thoughts?

Re: Marginalia Goes Open Source

#72

Earlier quoted context omitted.

What sort of sizes are we talking about? I'm thinking if it would be possible to "crowdfund" the storage-costs for a requester-pays s3 bucket for it.

I'm probably producing around 250-500 Gb data/month at this point.

Any idea how compressible that is?

If it's something that compresses really well (eg text data in a database), then live compression filesystems (eg ZFS, likely others) could potentially help make that workable.

Re: Marginalia Goes Open Source

#73

Earlier quoted context omitted.

What sort of sizes are we talking about? I'm thinking if it would be possible to "crowdfund" the storage-costs for a requester-pays s3 bucket for it.

I'm probably producing around 250-500 Gb data/month at this point.

Reach out to Jason Scott -- textfiles.com -- and see if he knows anyone who would be interested.

He might know some folks.

Re: Marginalia Goes Open Source

#74

Earlier quoted context omitted.

I'm probably producing around 250-500 Gb data/month at this point.

Any idea how compressible that is? If it's something that compresses really well (eg text data in a database), then live compression filesystems (eg ZFS, likely others) could potentially help make that workable.

The data is either already compressed or dense binary soup, so no luck.

Re: Marginalia Goes Open Source

#75

> I'm currently looking for hosting for a large term frequency data file that is necessary for several of the search engine's core functions. I really don't have the bandwidth to serve it myself. It's only a couple of hundred megabytes so it'll probably be solvable somehow. I have a bit of (rented) space available. Most of it on Uberspace. What are the expected traffic requirements? Contact me if you like. Will updat…

GitHub Releases can host files up to 2 GB, for free.
Post reply on HN