Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Corral – A Serverless MapReduce Framework

github.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Show HN: Corral – A Serverless MapReduce Framework

#31

Post by Annatar »

scarface74 wrote: post #30
Annatar wrote: post #27

Earlier quoted context omitted.

Netflix has an army of professional system and kernel engineers who optimize and engineer the OSes they run on; they just use AWS for virtual machine and datacenter capacity.

They use a lot of AWS services - not just virtual machines - First they use Ubuntu Linux - SES (Email) - ElasticSearch (AWS has their own managed version) - SQS (queueing system) - S3 (storage) -auto scaling Most of the optimization's they do are standard things at scale where they are tuning by measuring performance. https://youtu.be/89fYOo1V2pA and the slide deck from the presentation. https://www.slideshare.net/br…

So if anything, they are not running “serverless”, far from it: they really dig their hands into the OS and system engineering. Just recently they did kernel engineering on FreeBSD to push the networking as close to 100 Gbps per interface as possible. You picked a bad example to make your point, a point which I do not, from experience of doing this for several decades, recognize as valid anyway.

By the by, I’m that guy who runs a private datacenter in the basement, from designing one’s own rack mountable servers to crimping the network cables and running the fiber. Infrastructure is not something one should entrust to others, because these others cannot be trusted, as has been proven by AWS outages time and time again. And yes, I system engineer my own operating system as well. Well I did, it’s all been running automatically for years now.

Re: Show HN: Corral – A Serverless MapReduce Framework

#32

Post by scarface74 »

Annatar wrote: post #31
scarface74 wrote: post #30

Earlier quoted context omitted.

They use a lot of AWS services - not just virtual machines - First they use Ubuntu Linux - SES (Email) - ElasticSearch (AWS has their own managed version) - SQS (queueing system) - S3 (storage) -auto scaling Most of the optimization's they do are standard things at scale where they are tuning by measuring performance. https://youtu.be/89fYOo1V2pA and the slide deck from the presentation. https://www.slideshare.net/br…

So if anything, they are not running “serverless”, far from it: they really dig their hands into the OS and system engineering. Just recently they did kernel engineering on FreeBSD to push the networking as close to 100 Gbps per interface as possible. You picked a bad example to make your point, a point which I do not, from experience of doing this for several decades, recognize as valid anyway. By the by, I’m that g…

You were making a point about people should run their own data centers. Netflix is not running their own data centers. Can you run your own data center in 90 areas around the world? Can you run your own globally distributed CDN? Your own globally distributed DNS? Are you saying that you never had any downtime? Can you stand up a replica of your object store, your caching and your database in both Europe and Asia so it can be closer to your outsourced developers and foreign customers? Does your data center in your basement allow me to set up subnets that span regions (not AWS “regions”, geographic areas - “Availability Zones”) just in case your basement floods?

What if I don't want a replica of data, what if I want to duplicate my entire infrastructure in Asia so the developers there can have a clone of our infrastructure - databases, storage, VMs, load balancers, multiple availability zones, etc. How long would it take you to do that? I could do it with a JSON script and run a command.

How fast can you provision a half dozen load balancers and five or ten dedicated computers - not VMs “dedicated hosts”? I can do it by creating a JSON file and running one command from my terminal.

And why should I trust you to set up a more reliable, redundant, network than AWS? Again Netflix didn’t trust themselves to create more reliable infrastructure and decided to trust a competitor to do the “undifferentiated heavy lifting” and the guy who lead the transition is now a VP at AWS, so I think he knows something about infrastructure.

Re: Show HN: Corral – A Serverless MapReduce Framework

#33

Post by Annatar »

scarface74 wrote: post #32
Annatar wrote: post #31

Earlier quoted context omitted.

So if anything, they are not running “serverless”, far from it: they really dig their hands into the OS and system engineering. Just recently they did kernel engineering on FreeBSD to push the networking as close to 100 Gbps per interface as possible. You picked a bad example to make your point, a point which I do not, from experience of doing this for several decades, recognize as valid anyway. By the by, I’m that g…

You were making a point about people should run their own data centers. Netflix is not running their own data centers. Can you run your own data center in 90 areas around the world? Can you run your own globally distributed CDN? Your own globally distributed DNS? Are you saying that you never had any downtime? Can you stand up a replica of your object store, your caching and your database in both Europe and Asia so i…

You were making a point about people should run their own data centers.

I'm making the point that one can never write high quality software if one does not master system administration and then system engineering first. It's impossible to write high quality software without understanding the substrate on which software is built. That's my message.

Re: Show HN: Corral – A Serverless MapReduce Framework

#34

Post by scarface74 »

Annatar wrote: post #33
scarface74 wrote: post #32

Earlier quoted context omitted.

You were making a point about people should run their own data centers. Netflix is not running their own data centers. Can you run your own data center in 90 areas around the world? Can you run your own globally distributed CDN? Your own globally distributed DNS? Are you saying that you never had any downtime? Can you stand up a replica of your object store, your caching and your database in both Europe and Asia so i…

You were making a point about people should run their own data centers. I'm making the point that one can never write high quality software if one does not master system administration and then system engineering first. It's impossible to write high quality software without understanding the substrate on which software is built. That's my message.

It could easily be said that one cannot write quality software if they don't understand how compilers work, how to write assembly language, how processors are fabbed, etc.

How far down the rabbit hole do you want to go?

It's not a matter of "not understanding" how it works, it's a matter of focusing on your core competency and even if you know how to do something, it's about where you can add value and where is it best to outsource.

AWS has a dozens of services across dozens of areas across the globe to handle infrastructure. Why waste time doing the "undifferentiated heavy lifting" that you can't do as well? No matter how good you think you are, you can't efficiently setup infrastructure as fast and as reliable as AWS.

How quickly could you setup duplicate data centers on opposite sides of the continent for disaster recovery and/or to reduce latency? I can do it by running a CloudFormation template.

Re: Show HN: Corral – A Serverless MapReduce Framework

#35

Post by dang »

Annatar wrote: post #33
scarface74 wrote: post #32

Earlier quoted context omitted.

You were making a point about people should run their own data centers. Netflix is not running their own data centers. Can you run your own data center in 90 areas around the world? Can you run your own globally distributed CDN? Your own globally distributed DNS? Are you saying that you never had any downtime? Can you stand up a replica of your object store, your caching and your database in both Europe and Asia so i…

You were making a point about people should run their own data centers. I'm making the point that one can never write high quality software if one does not master system administration and then system engineering first. It's impossible to write high quality software without understanding the substrate on which software is built. That's my message.

Can you please not do technical flamewars on HN? The value that should drive discussions here is intellectual curiosity (https://news.ycombinator.com/newsguidelines.html), not nitpicking or irritable fault-finding.

Re: Show HN: Corral – A Serverless MapReduce Framework

#36

Post by Annatar »

scarface74 wrote: post #34
Annatar wrote: post #33

Earlier quoted context omitted.

You were making a point about people should run their own data centers. I'm making the point that one can never write high quality software if one does not master system administration and then system engineering first. It's impossible to write high quality software without understanding the substrate on which software is built. That's my message.

It could easily be said that one cannot write quality software if they don't understand how compilers work, how to write assembly language, how processors are fabbed, etc. How far down the rabbit hole do you want to go? It's not a matter of "not understanding" how it works, it's a matter of focusing on your core competency and even if you know how to do something, it's about where you can add value and where is it be…

How far down the rabbit hole do you want to go?

All the way down to the hardware.

Re: Show HN: Corral – A Serverless MapReduce Framework

#37

Post by Annatar »

dang wrote: post #35
Annatar wrote: post #33

Earlier quoted context omitted.

You were making a point about people should run their own data centers. I'm making the point that one can never write high quality software if one does not master system administration and then system engineering first. It's impossible to write high quality software without understanding the substrate on which software is built. That's my message.

Can you please not do technical flamewars on HN? The value that should drive discussions here is intellectual curiosity ( https://news.ycombinator.com/newsguidelines.html ), not nitpicking or irritable fault-finding.

Considering that the topic of "serverless" is itself highly controversial, consider that mentioning it here is akin to starting a flamewar. Consider the other side as well, not just one side.

My intent wasn't and isn't a flamewar. The topic is not one of "intellectual curiosity" either.

Re: Show HN: Corral – A Serverless MapReduce Framework

#38

Post by dang »

Annatar wrote: post #37
dang wrote: post #35

Earlier quoted context omitted.

Can you please not do technical flamewars on HN? The value that should drive discussions here is intellectual curiosity ( https://news.ycombinator.com/newsguidelines.html ), not nitpicking or irritable fault-finding.

Considering that the topic of "serverless" is itself highly controversial, consider that mentioning it here is akin to starting a flamewar. Consider the other side as well, not just one side. My intent wasn't and isn't a flamewar. The topic is not one of "intellectual curiosity" either.

The OP certainly clears the bar for being intellectually interesting in HN's sense.

It's true that 'serverless' is a bit of a trigger word in technical discussions and people disagree about what it means, etc., but there are degrees of these things, and you stepped into several further degrees of flamewar. Please don't do that!

Re: Show HN: Corral – A Serverless MapReduce Framework

#39

Post by [deleted] »

Annatar wrote: post #36
scarface74 wrote: post #34

Earlier quoted context omitted.

It could easily be said that one cannot write quality software if they don't understand how compilers work, how to write assembly language, how processors are fabbed, etc. How far down the rabbit hole do you want to go? It's not a matter of "not understanding" how it works, it's a matter of focusing on your core competency and even if you know how to do something, it's about where you can add value and where is it be…

How far down the rabbit hole do you want to go? All the way down to the hardware.

[deleted]
Post reply on HN