111–120 of 158 posts
Somehow people don't use self-signed certificates all over the web but for sftp it's "fine" apparently.
An alternative to doing file based SFTP is to just treat SFTP like an API. A company I work for implemented an SFTP service where every operation simply translates to some SQL DB lookup. And a file download kicks off a larger SQL query and generates the report on the fly, streaming the result straight through to the SFTP client. Works great! SFTP can be an API just like HTTP. Under the hood the protocol is reasonably…
What if you open an SFTP handle, and then write 5 bytes halfway through a 20 GB file and close the handle? How do you translate that?
An alternative to doing file based SFTP is to just treat SFTP like an API. A company I work for implemented an SFTP service where every operation simply translates to some SQL DB lookup. And a file download kicks off a larger SQL query and generates the report on the fly, streaming the result straight through to the SFTP client. Works great! SFTP can be an API just like HTTP. Under the hood the protocol is reasonably…
What if you open an SFTP handle, and then write 5 bytes halfway through a 20 GB file and close the handle? How do you translate that?
There are a handful of services out there that do this, I know because I've needed it on multiple occasions. It's nice that Amazon is providing it in house now, but it just reminds of me the last time I went to re:invent and walked through the vendor area and thought about how many of these companies are four dev cycles away from Amazon producing a baked in competitor. Hard to make a B To B Amazon tool these days.
That’s quite literally AWS’s playbook. Fill the vendor hall with tech, learn it, see what sticks and then crush it. This is why I’m bullish on cloud agnostic tech. These practices don’t typically fair well in the enterprise space. This is why companies like MSFT are interesting to me. They partner and rarely kill. Amazon is the complete opposite.
Earlier quoted context omitted.
That’s quite literally AWS’s playbook. Fill the vendor hall with tech, learn it, see what sticks and then crush it. This is why I’m bullish on cloud agnostic tech. These practices don’t typically fair well in the enterprise space. This is why companies like MSFT are interesting to me. They partner and rarely kill. Amazon is the complete opposite.
> They partner and rarely kill. How the times have changed.
There are a handful of services out there that do this, I know because I've needed it on multiple occasions. It's nice that Amazon is providing it in house now, but it just reminds of me the last time I went to re:invent and walked through the vendor area and thought about how many of these companies are four dev cycles away from Amazon producing a baked in competitor. Hard to make a B To B Amazon tool these days.
There was a flaw in our product that had one of our customers pushing to use Amazon's offering. Turns out Amazon's service cost about 4x what it cost us to do it ourselves. It wasn't obvious at the time given Amazon's purposefully obtuse pricing.
Eventually we fixed the bug, brought our customer back in line with the rest, saved some money, and have continued providing this service much cheaper than Amazon. I think it works for us because of a few reasons:
1. The service we offer is challenging enough that most dev teams won't want to do it themselves, they'll outsource it
2. Amazon has little incentive to charge less (see #1), little competition
3. We're small enough that we can still provide that face-to-face level of service and hand holding that's nearly impossible to get from a larger org
Amazon/Microsoft/Google may come into a particular market, but it doesn't automatically imply that they can (or even will) do it better/cheaper/faster.
This would mean that rsync now works with S3 as it has an sftp target... correct?
You should try the mirror sub-system of lftp [1]. It can replicate rsync behavior on a chroot sftp server. No idea if that works on Amazon, but I use it all the time on my own chroot sftp servers. [1] - https://tinyvpn.org/sftp/#lftp
For regular tasks you could also look at “rclone” which is like rsync in many ways but can upload to s3, backblaze b2, sftp and any more directly. Without remote support.
Earlier quoted context omitted.
Can you elaborate? I mean a plain CentOS server running SFTP, S3FS seems about as set and forget as it gets. And each? Surely chrooting users would let you consolidate all of those servers into one (or one cluster for HA I suppose).
> Can you elaborate? I mean a plain CentOS server running SFTP, S3FS seems about as set and forget as it gets. Think about the operational costs: someone needs to manage keys, logging, security updates, when S3FS coughs a lung and hangs you need to catch that problem and remount it to restore service, etc. This service reuses the existing authentication systems so you don't need to spend time configuring and managing…