Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean VPC

blog.digitalocean.com

Post reply on HN
Print view View original

1–10 of 169 posts

Re: DigitalOcean VPC

#2

Post by jrockway »

Do they talk at all about what they're using to provide the VPC overlay? I have a DO k8s cluster and it uses Cilium for the CNI, which turns out to be quite useful, so I guess I'm wondering if they're also using Cilium for this.

(Over in AWS land, they wrote a CNI for their own VPC networking. It turns out to have many strange limitations. For example, you can only run 17 pods on a certain type of node, because that node is only allowed to have 19 VPC addresses. I was quite surprised when pods stopped scheduling even though CPU and memory were available. Turns out internal IP addresses are a resource, too. DigitalOcean has the advantage of starting fresh, so might be able to use something open source that can be played with in a dev environment and extended with open source projects.)

Re: DigitalOcean VPC

#3

Post by nerdbaggy »

I find some of the limits weird https://www.digitalocean.com/docs/networking/vpc/

- VPC network ranges cannot overlap with the ranges of other networks in the same account. (Edit: Does this mean each VPC in the account has to have a non overlapping subnet?)

-Resources do not currently support multiple private network interfaces and cannot be placed in multiple VPC networks.

- Not being able to change the VPC connected to stuff without taking a snapshot

Re: DigitalOcean VPC

#4

Post by pqdbr »

When are you going to have a datacenter in Brazil? We don't mind if we have to pay more than your listed prices for other locations. We know Brazil is more expensive. Just do it already.

Re: DigitalOcean VPC

#6

Post by andoriyu »

jrockway wrote: post #2

Do they talk at all about what they're using to provide the VPC overlay? I have a DO k8s cluster and it uses Cilium for the CNI, which turns out to be quite useful, so I guess I'm wondering if they're also using Cilium for this. (Over in AWS land, they wrote a CNI for their own VPC networking. It turns out to have many strange limitations. For example, you can only run 17 pods on a certain type of node, because that…

> Turns out internal IP addresses are a resource, too.

That's not what is happening in AWS. IP address are resources (duh), but that's no the issue. With their CNI plugin each pod gets its own Elastic Network Interface. ENIs aren't just virtio's virtual network, it could be ENA (100Gbps) or Intel VF (10Gbs). It's a hardware limitation of amazon virtualization stack starting with previous generation instances.

> I was quite surprised when pods stopped scheduling even though CPU and memory were available.

This is well documented here: https://github.com/aws/amazon-vpc-cni-k8s

Re: DigitalOcean VPC

#7

Post by t3rabytes »

nerdbaggy wrote: post #3

I find some of the limits weird https://www.digitalocean.com/docs/networking/vpc/ - VPC network ranges cannot overlap with the ranges of other networks in the same account. (Edit: Does this mean each VPC in the account has to have a non overlapping subnet?) -Resources do not currently support multiple private network interfaces and cannot be placed in multiple VPC networks. - Not being able to change the VPC connecte…

Pretty standard? Taking AWS for example:

- You can do this, but it's highly discouraged since it means no VPC peering if you ever need that.

- Can't do this at all with network interfaces, it all is via VPC peering.

- Can't change the VPC after an instance has been created, you have to take a snapshot and relaunch it.

Re: DigitalOcean VPC

#8

Post by nerdbaggy »

t3rabytes wrote: post #7
nerdbaggy wrote: post #3

I find some of the limits weird https://www.digitalocean.com/docs/networking/vpc/ - VPC network ranges cannot overlap with the ranges of other networks in the same account. (Edit: Does this mean each VPC in the account has to have a non overlapping subnet?) -Resources do not currently support multiple private network interfaces and cannot be placed in multiple VPC networks. - Not being able to change the VPC connecte…

Pretty standard? Taking AWS for example: - You can do this, but it's highly discouraged since it means no VPC peering if you ever need that. - Can't do this at all with network interfaces, it all is via VPC peering. - Can't change the VPC after an instance has been created, you have to take a snapshot and relaunch it.

Interesting, didn’t know that about AWS. In more familiar with the Google cloud version of VPC. Seems the DO implementation is more like the AWS version

Re: DigitalOcean VPC

#9

Post by dilyevsky »

jrockway wrote: post #2

Do they talk at all about what they're using to provide the VPC overlay? I have a DO k8s cluster and it uses Cilium for the CNI, which turns out to be quite useful, so I guess I'm wondering if they're also using Cilium for this. (Over in AWS land, they wrote a CNI for their own VPC networking. It turns out to have many strange limitations. For example, you can only run 17 pods on a certain type of node, because that…

Better way of doing natively addressable pods is assign whole subnets (like /25) as secondary interface and distribute that to pods via cni. I think gke pod network works that way. Not sure why eks decided 17 pods is ok lol

Re: DigitalOcean VPC

#10

Post by SteveNuts »

JakeAl wrote: post #5

They must be great, my servers are constantly receiving hack attempts from Digital Ocean IPs.

Does this mean that previously to this change, without a software firewall running you'd be vulnerable to attacks on the private network from other customers? (I've never used DO).
Post reply on HN