Live data from Hacker News

Ask HN: Best document storage solution in 2023

news.ycombinator.com

1–10 of 15 posts

Ask HN: Best document storage solution in 2023

#1
We have a application which requires a document store. This need to hold up to 10 million documents and be accessible via APIs to retrieve documents to display in our application.

We are considering everything from Dropbox-type solutions to blob storage in GCP.

What kind of document storage solutions are people using in 2023 to meet this use case?

Re: Ask HN: Best document storage solution in 2023

#2
It's too much of a generic question to be answered right. Do you need global availability? Do you need high speed downloads? Are you worried about bandwidth costs? etc.

We use S3 + Cloudfront for documents that we want to be quickly accessed by our customers. We use SFTP for our internal docs when we don't care that much about availability and speed.

Re: Ask HN: Best document storage solution in 2023

#3
post #2

It's too much of a generic question to be answered right. Do you need global availability? Do you need high speed downloads? Are you worried about bandwidth costs? etc. We use S3 + Cloudfront for documents that we want to be quickly accessed by our customers. We use SFTP for our internal docs when we don't care that much about availability and speed.

No, not global. Each office only really need access to their own documents, there are over 2,000 offices though. It does need to be high speed and yes cost is a factor.

Re: Ask HN: Best document storage solution in 2023

#5
post #2

It's too much of a generic question to be answered right. Do you need global availability? Do you need high speed downloads? Are you worried about bandwidth costs? etc. We use S3 + Cloudfront for documents that we want to be quickly accessed by our customers. We use SFTP for our internal docs when we don't care that much about availability and speed.

No, not global. Each office only really need access to their own documents, there are over 2,000 offices though. It does need to be high speed and yes cost is a factor.

If you are willing to try out some beta software, give https://didgets.com/ a look. You can load in hundreds of millions of documents or other kinds of files and put meta-data tags on each of them. Queries for individual documents or sets of them execute with sub-second speeds. The software is in beta and available for free download.

Currently working on the indexing feature to index the contents of the files too.

Re: Ask HN: Best document storage solution in 2023

#8
post #2

It's too much of a generic question to be answered right. Do you need global availability? Do you need high speed downloads? Are you worried about bandwidth costs? etc. We use S3 + Cloudfront for documents that we want to be quickly accessed by our customers. We use SFTP for our internal docs when we don't care that much about availability and speed.

What SFTP client do you use in Linux?

The command line isn’t very convenient.

Re: Ask HN: Best document storage solution in 2023

#9
One possibility is to use our open-core document management API build to deploy in your AWS account: https://github.com/formkiq/formkiq-core

The files are stored in S3, with customizable metadata storage in DynamoDB. As the system is designed to run on AWS Serverless and Managed Services, the majority of the cost will come from S3 storage fees.

Re: Ask HN: Best document storage solution in 2023

#10
post #8
post #2

It's too much of a generic question to be answered right. Do you need global availability? Do you need high speed downloads? Are you worried about bandwidth costs? etc. We use S3 + Cloudfront for documents that we want to be quickly accessed by our customers. We use SFTP for our internal docs when we don't care that much about availability and speed.

What SFTP client do you use in Linux? The command line isn’t very convenient.

We don't, we're accessing it programmatically from php, it has built in support. I'm not sure how it works under the hood.
Post reply on HN