Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Common problems with large file uploads

blog.filepicker.io

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Common problems with large file uploads

#21

Post by repsilat »

meanguy wrote: post #18
tagx wrote: post #2

I'd really like to be able to use Dropbox as a magic upload handler for any file I upload on my local HD, not just those in my Dropbox folder. They handle the logic of getting all my files into the cloud. Why can't I point a website to my Dropbox and say here, this is handling the file upload?

Dropbox only supports files up to 150MB via their API. I inquired about bumping the limit on a per-app basis; no response.

Doesn't matter too much if you're just using Dropbox as a storage backend for a more complex service. If your box is full of encrypted 2MB segments backing a pseudo-filesystem you can upload files as big as you like and not have to worry about broken connections.

The best part is that you can use other services with similar APIs for pseudo-RAID or just a union-mount. Sharing is more difficult than vanilla Dropbox, of course, but I'm working on it...

Re: Common problems with large file uploads

#22

Post by Ralith »

ChrisNorstrom wrote: post #20

8gb+ files? I found a way but you have to use a JAVA FTP Applet. I tested these two here: http://jupload.sourceforge.net/ and http://www.jfileupload.com/ Dragged and dropped an 8gb+ file and left it on for 5 hours. Worked perfectly. No time outs, no errors, and I'm on a shared hosting account at 1and1. My problem with them is that it wasn't possible to hide the FTP username and password, they were always in javascrip…

What exactly did you expect them to do about it? For a client-side tool to establish a plain FTP connection, it needs to possess authentication credentials.

Re: Common problems with large file uploads

#23

Post by rajbot »

liyanchang wrote: post #13
rajbot wrote: post #10

I've been dealing with browser-based large file uploads, which means dealing with lots of browser-specific issues. Fortunately, things are getting better, especially for the webkit-based browsers. Firefox still has some issues, and I check https://bugzilla.mozilla.org/show_bug.cgi?id=678648 pretty regularly. Just today this bug, which was filed in 2003, changed from Status = NEW to Status = ASSIGNED. Today is a good…

First, I'm impressed that someone was uploading 2gb files back in 2003... Agreed. Good to see that firefox is going to be able to do more than 2gb soon.

That title is a bit misleading.. On some platforms you can already use Firefox to do >2GB uploads, but there is still a 4GB limit..

If anyone wants to help beta-test a HTML5 uploader that calls archive.org's S3-like endpoint under the hood (no IE or Opera support yet, though Opera 12 is now working..): http://archive.org/upload/

Re: Common problems with large file uploads

#24

Post by mbreese »

ChrisNorstrom wrote: post #20

8gb+ files? I found a way but you have to use a JAVA FTP Applet. I tested these two here: http://jupload.sourceforge.net/ and http://www.jfileupload.com/ Dragged and dropped an 8gb+ file and left it on for 5 hours. Worked perfectly. No time outs, no errors, and I'm on a shared hosting account at 1and1. My problem with them is that it wasn't possible to hide the FTP username and password, they were always in javascrip…

You could always just hard-code the username/password into the applet and recompile. That shouldn't be too hard...

Or, if you control the FTP server, you could dynamically add and remove random virtual users/passwords to the FTP server (hopefully virtual users). Then when the client javascript gets the username/password, it could only be used once.

Re: Common problems with large file uploads

#25

Post by zimbatm »

Given the title I was expecting the article to provide a solution.

From personal experience, the bigger the file, the more likely you will experience a connection cut in the middle of the upload. That is why the most important thing it to support resumable uploads.

At the moment there is no clear consensus on how to handle that. Amazon S3 has one protocol, Google uses two revisions of a different protocol, one on YouTube[2], another on Google Cloud Storage[3]. Both work by first creating a session that you refer to when uploading the chunks. There is also the Nginx upload module[4] that delegates the session ID to the client for some reason.

And there is no browser client available to my knowledge.

That's all I know folks

[1]: http://docs.amazonwebservices.com/AmazonS3/latest/API/mpUplo... [2]: https://developers.google.com/youtube/2.0/developers_guide_p... [3]: https://developers.google.com/storage/docs/developer-guide [4]: http://www.grid.net.ru/nginx/resumable_uploads.en.html

Re: Common problems with large file uploads

#26

Post by cstavish »

mbreese wrote: post #24
ChrisNorstrom wrote: post #20

8gb+ files? I found a way but you have to use a JAVA FTP Applet. I tested these two here: http://jupload.sourceforge.net/ and http://www.jfileupload.com/ Dragged and dropped an 8gb+ file and left it on for 5 hours. Worked perfectly. No time outs, no errors, and I'm on a shared hosting account at 1and1. My problem with them is that it wasn't possible to hide the FTP username and password, they were always in javascrip…

You could always just hard-code the username/password into the applet and recompile. That shouldn't be too hard... Or, if you control the FTP server, you could dynamically add and remove random virtual users/passwords to the FTP server (hopefully virtual users). Then when the client javascript gets the username/password, it could only be used once.

One could scan the .class file for string literals with relative ease. Obfuscation would be an improvement, but still not completely secure.

Re: Common problems with large file uploads

#27

Post by gue5t »

cstavish wrote: post #26
mbreese wrote: post #24

Earlier quoted context omitted.

You could always just hard-code the username/password into the applet and recompile. That shouldn't be too hard... Or, if you control the FTP server, you could dynamically add and remove random virtual users/passwords to the FTP server (hopefully virtual users). Then when the client javascript gets the username/password, it could only be used once.

One could scan the .class file for string literals with relative ease. Obfuscation would be an improvement, but still not completely secure.

It would hardly be an improvement. Wireshark would be a first step for most reverse engineers when there's network authentication involved.

Re: Common problems with large file uploads

#29

Post by jasomill »

As plenty of file transfer protocols, clients, and servers support resumable transfers (FTP, SFTP, rsync, proprietary browser-based tools, etc., or even basic HTTP if you arrange for the file to be pulled rather than pushed and your "client's server" has byte-range support), perhaps this should be titled "why you shouldn't use a single HTTP POST request from a browser to upload a large file". The general reason seems to be "because this is not a use case this feature is commonly designed for and tested against."

Re: Common problems with large file uploads

#30

Post by mbreese »

cstavish wrote: post #26
mbreese wrote: post #24

Earlier quoted context omitted.

You could always just hard-code the username/password into the applet and recompile. That shouldn't be too hard... Or, if you control the FTP server, you could dynamically add and remove random virtual users/passwords to the FTP server (hopefully virtual users). Then when the client javascript gets the username/password, it could only be used once.

One could scan the .class file for string literals with relative ease. Obfuscation would be an improvement, but still not completely secure.

Well, we are talking about FTP, so the string is going to travel over the line in plaintext anyway.
Post reply on HN