41–50 of 73 posts
What missing from description is network setup. Is it ec2 classic, VPC? Is ec2 getting to s3 through IG? Hopefully not through NAT. There is also VPC endpoint to s3. Which all may have different performance profiles especially with multiple instances.
Has the author (if they are reading here) considered using Joyent's Manta to take the processing to the data instead?
Has the author (if they are reading here) considered using Joyent's Manta to take the processing to the data instead?
The bigger point in the article is that these exact "take processing to the data" architectures operate exceedingly well on S3, GCS, Azure.
And, as a biased observer, these architectures operate on GCS the best due to great performance measured in the article, quick VM standup times, low VM prices, and per-minute billing.
Has the author (if they are reading here) considered using Joyent's Manta to take the processing to the data instead?
There are plenty of architectures that do exactly this. EMR-on-S3, Google Dataproc on GCS, Snowflake-on-S3, BigQuery-on-GCS, etc etc. The bigger point in the article is that these exact "take processing to the data" architectures operate exceedingly well on S3, GCS, Azure. And, as a biased observer, these architectures operate on GCS the best due to great performance measured in the article, quick VM standup times, l…
EMR-on-S3 is the "copy the data to the processing nodes" variety.
Has the author (if they are reading here) considered using Joyent's Manta to take the processing to the data instead?
Spinning up a cluster of VMs and use 10 seconds and they charge you min. 1 hour seems expensive to me.
Has the author (if they are reading here) considered using Joyent's Manta to take the processing to the data instead?
There are plenty of architectures that do exactly this. EMR-on-S3, Google Dataproc on GCS, Snowflake-on-S3, BigQuery-on-GCS, etc etc. The bigger point in the article is that these exact "take processing to the data" architectures operate exceedingly well on S3, GCS, Azure. And, as a biased observer, these architectures operate on GCS the best due to great performance measured in the article, quick VM standup times, l…
Earlier quoted context omitted.
There are plenty of architectures that do exactly this. EMR-on-S3, Google Dataproc on GCS, Snowflake-on-S3, BigQuery-on-GCS, etc etc. The bigger point in the article is that these exact "take processing to the data" architectures operate exceedingly well on S3, GCS, Azure. And, as a biased observer, these architectures operate on GCS the best due to great performance measured in the article, quick VM standup times, l…
I'm still trying to parse the docs and Manta source code to see what it actually does, but it seems unique if the data storage nodes are also the data processing nodes and no data transfer happens from some storage service before the job begins. The other key factor is having neither startup time nor the cost of a perpetually running cluster. Per my comment below [1], we have used Lambda with S3 to get something like…
What missing from description is network setup. Is it ec2 classic, VPC? Is ec2 getting to s3 through IG? Hopefully not through NAT. There is also VPC endpoint to s3. Which all may have different performance profiles especially with multiple instances.
Network was VPC. The EC2 instance had an IG attached, yes, but I'm not sure if you're asking if an internal vs. external URL for S3 was used? Are you saying there's a better endpoint than s3- .amazonaws.com for S3 requests from EC2?
It's a private connection to AWS services including S3. You'd use the same URL as it's a routing basically. No idea if VPC endpoints would be better than IG though. P.S. Just tested and I get about half of the latency on VPC endpoint.
Earlier quoted context omitted.
Any comments on how it worked out with Lambda?
Reluctant to say much because the benchmarks weren't formal. However... The throughput correlated directly with how much RAM we allocated to the Lambda function (which presumably means we were sharing the VM with fewer other jobs). 512 MB RAM, 19.5 MB/s 768 MB RAM, 29.8 MB/s 1024 MB RAM, 38.4 MB/s 1536 MB RAM, 43.7 MB/s Note that this also used the node.js AWS SDK, which is slower to download files than some other AP…
Has the author (if they are reading here) considered using Joyent's Manta to take the processing to the data instead?