71–80 of 158 posts
Earlier quoted context omitted.
I think the point being that when a filesystem the mv is atomic and just updates an inode but on S3 those operations can take place on thousand of different machines.
SFTP is not a filesystem, though; to get a rename to be atomic, you must pass SSH_FXF_RENAME_ATOMIC, and the server can return SSH_FX_OP_UNSUPPORTED.
Enterprises will love this. There are so many legacy app flows kicked off via sftp/scp file drops. Being able to hook into those via lambda events on the associated S3 bucket will create a whole ecosystem of enterprise spaghetti for years to come.
The only downside is whitelisting but not on the SFTP server-side. Many enterprises restrict egress SFTP (usually for security reasons) so you need to provide IPs and they can’t frequently change because it can take enterprise network admins quite some time to deal with all of the bureaucracy and change control. That said, I wouldn’t be surprised if modern networking gear can handle CNAMEs but there’s no guarantee th…
Anyone know if this can be used directly from CloudFormation?
A tonne of our (enterprise-ey) customers had such trouble trying to integrate into our S3 flow that we started launching VPS for each that abstracted it away into simple SFTP upload/download, which they were used to. Although this is much more expensive than Lightsail, the man hours saved will make it worthwhile.
And each? Surely chrooting users would let you consolidate all of those servers into one (or one cluster for HA I suppose).
We use a number of AWS LightSail servers for SFTP today, which mostly sit idle. Will definitely adopt this!
The AWS pricing page for this service says it costs about $225/month for a lightly used instance. I implemented the same kind of thing on AWS using a nano-sized instance for about $10/month. The instance is managed with an Ansible Role for automated SFTP server management. I connected it with an off-the-shelf AWS Lamda function which listens for S3 PUT events and copies files to the SFTP server as needed. My solution…
> SFTP (Secure File Transfer Protocol) It's SSH File Transfer Protocol. When you say Secure File Transfer Protocol many people think about FTP over SSL if you don't emphasize it's about SSH.
A tonne of our (enterprise-ey) customers had such trouble trying to integrate into our S3 flow that we started launching VPS for each that abstracted it away into simple SFTP upload/download, which they were used to. Although this is much more expensive than Lightsail, the man hours saved will make it worthwhile.
Can you elaborate? I mean a plain CentOS server running SFTP, S3FS seems about as set and forget as it gets. And each? Surely chrooting users would let you consolidate all of those servers into one (or one cluster for HA I suppose).
Think about the operational costs: someone needs to manage keys, logging, security updates, when S3FS coughs a lung and hangs you need to catch that problem and remount it to restore service, etc. This service reuses the existing authentication systems so you don't need to spend time configuring and managing integration with your customers’ LDAP/AD infrastructure, etc. If you deal with anything which hits PCI, HIPAA, etc. you need to be able to certify that your custom design meets those requirements as well.
That's not to say you can't do it yourself but for many places there's a fairly significant amount of work where the cost of doing it yourself is greater than 5+ years of managed service costs.
Earlier quoted context omitted.
That’s quite literally AWS’s playbook. Fill the vendor hall with tech, learn it, see what sticks and then crush it. This is why I’m bullish on cloud agnostic tech. These practices don’t typically fair well in the enterprise space. This is why companies like MSFT are interesting to me. They partner and rarely kill. Amazon is the complete opposite.
> They partner and rarely kill. How the times have changed.
This would mean that rsync now works with S3 as it has an sftp target... correct?