Live data from Hacker News

How we use HashiCorp Nomad

blog.cloudflare.com

51–60 of 166 posts

Re: How we use HashiCorp Nomad

#51
post #47
post #13

Off topic but... I've always wondered how Cloudflare can not charge for the bandwidth. Even the free plan is super generous (CDN, SSL, etc). How are they making a profit when AFAIK all other CDNs charge you for the bandwidth (and I assume they have to pay for to their providers)?

> How are they making a profit They aren't. In Q1 they made a loss of $33M on $91M in revenue.

Oh wow.

In fact it seems they have been loosing more and more money over the years.

https://finance.yahoo.com/quote/NET/financials/

What are they counting on happening here? Obviously free customers won't switch to paid plans.

Re: How we use HashiCorp Nomad

#52
post #35

Earlier quoted context omitted.

One reason is "settlement-free peering".

So what you're saying is that Cloudflare is so big they can reach these sort of deals with connectivity providers where they don't pay for bandwidth themselves?

Short answer:

Well, to a certain extent. I didn't mean to imply that their monthly spend for bandwidth is zero -- I'm sure they aren't anywhere close to peering 100% of their traffic, they aren't in the DFZ, and, of course, they've gotta pay somebody (or, more correctly, several somebodies) to connect all of those datacenters together!

---

Long answer:

About six years ago, they described their connectivity in "The Relative Cost of Bandwidth Around the World" [0]:

> "In CloudFlare's case, unlike Netflix, at this time, all our peering is currently "settlement free," meaning we don't pay for it. Therefore, the more we peer the less we pay for bandwidth."

> "Currently, we peer around 45% of our total traffic globally (depending on the time of day), across nearly 3,000 different peering sessions."

Remember, that was about six years ago. I wouldn't be surprised if both their peers and peering sessions have increased by an order of magnitude since that article was published -- just think of all the datacenters that they're in today that weren't back then, especially outside of North America!

Additionally, they've got an "open" policy when it comes to peering, as well as a presence damn near everywhere [1,2]. Since they're "mostly outbound", the eyeball networks will come to them, wanting to peer.

Running an anycast network and being "everywhere" also has some other benefits. They perform large-scale "traffic engineering" -- deciding which prefixes they advertise where, when, and to who, and the freedom to change that on the fly -- so they've got tremendous control over where traffic comes in to and, perhaps more importantly, exits from their network (bandwidth is ~15x more expensive in Africa, Australia and South America than North America, example).

So, yes, CloudFlare is still paying for transit but, at their level, it's relatively "dirt cheap". Plus, in addition to the increases mentioned above, bandwidth is likely an order of magnitude cheaper -- at least -- than it was six years ago!

---

EDIT:

Two years later, in August 2016, CloudFlare published an update [3] to the article linked above. A few highlights:

> "Since August 2014, we have tripled the number of our data centers from 28 to 86, with more to come."

> "CloudFlare has an “open peering” policy, and participates at nearly 150 internet exchanges, more than any other company."

> "... of the traffic that we are currently able to serve locally in Africa, we manage to peer about 90% ..."

> ".... we can peer 100% of our traffic in the Middle East ..."

> "Today, however, there are six expensive networks that are more than an order of magnitude more expensive than other bandwidth providers around the globe ... these six networks represent less than 6% of the traffic but nearly 50% of our bandwidth costs."

---

[0]: https://blog.cloudflare.com/the-relative-cost-of-bandwidth-a...

[1]: https://www.peeringdb.com/asn/13335

[2]: https://bgp.he.net/AS13335#_ix

[3]: https://blog.cloudflare.com/bandwidth-costs-around-the-world...

Re: How we use HashiCorp Nomad

#53
post #35

Earlier quoted context omitted.

So what you're saying is that Cloudflare is so big they can reach these sort of deals with connectivity providers where they don't pay for bandwidth themselves?

Short answer: Well, to a certain extent. I didn't mean to imply that their monthly spend for bandwidth is zero -- I'm sure they aren't anywhere close to peering 100% of their traffic, they aren't in the DFZ, and, of course, they've gotta pay somebody (or, more correctly, several somebodies) to connect all of those datacenters together! --- Long answer: About six years ago, they described their connectivity in "The Re…

Thanks for the info!

Re: How we use HashiCorp Nomad

#54
post #26

Earlier quoted context omitted.

Indeed. Helm offers great features but it suffers from the kubernetes unnecessary complexity and by using golang templates in YAML. When I started with kubernetes I converted my small Docker compose files to kubernetes files. Later I rewrote everything in helm charts. Now it's almost more YAML and golang templates lines than business logic lines in my applications. I'm considering to go back to Docker compose files.…

Highly recommend trying Jsonnet (via https://github.com/bitnami/kubecfg and https://github.com/bitnami-labs/kube-libsonnet ) as an alternative. It makes writing Kubernetes manifests much more expressive and integrates better with Git/VCS based workflows. Another language like Dhall or CUE might also work, but I'm not aware of a kubecfg equivalent for them. Jsonnet in general is a pretty damn good configuration langua…

If you like those, I'd take a look at Grafana's Tanka [0]. It also uses jsonnet but has some additional features such as showing a diff of your changes before you apply, easy vendored libraries using jsonnet-bundler, and the concept of "environments" which prevents you from accidentally applying changes to the wrong namespace/cluster.

[0] https://github.com/grafana/tanka

Re: How we use HashiCorp Nomad

#55
post #33

Earlier quoted context omitted.

See the second answer for a statement from cf https://webmasters.stackexchange.com/questions/88659/how-can...

I'm familiar with those terms but still. There are people with free accounts moving GBs every month through their network and I imagine those free users must account for a very large percentage of their traffic.

I’d imagine at this point they are heavily peered in most markets, driven by said free users, so there isn’t a significant opex hit bandwidth -wise. Space/power opex plus network/compute hardware capex probably dominates their spend.

Re: How we use HashiCorp Nomad

#56

Earlier quoted context omitted.

Helm charts are declarative way of deploying app(s) and their accompanying resources.

Maybe I was doing it wrong but every guide was verb based - “helm install X”. My declarative ideal ended up being a text file full of helm install commands and that wasn’t what I wanted.

Quick-start guides takes the easiest path to get something running, which is `helm install` in the Helm world.

If you want to have complete control of what you're pushing to the API, use Helm as an starting point instead, run `helm template` and save the YAML output to some file, publish it using `kubectl` or some other rollout tool. I recommend using `kapp` [1] for rollouts.

[1] https://get-kapp.io/

Re: How we use HashiCorp Nomad

#57

The biggest hurdle with adopting nomad is the kubernetes ecosystem and related network effects. Things like operators only run on Kubernetes, and they're driving an entirely new paradigm of infrastructure and application management. HashiCorp is doing their best to keep up while supporting standard kube interfaces like CSI/CNI/CRI, but I don't know how they can possibly stay relevant with Kubernetes momentum. In my o…

This is what Nomad is though... It works without their other products and also natively integrates into them. Deploying Vault, Consul, and Nomad gives you a very nice experience.

Also with Consul 1.8 and Nomad 0.11 you’ll get Consul Connect with Ingress gateways which solves some of those problems you mentioned.

Re: How we use HashiCorp Nomad

#58
post #13

Off topic but... I've always wondered how Cloudflare can not charge for the bandwidth. Even the free plan is super generous (CDN, SSL, etc). How are they making a profit when AFAIK all other CDNs charge you for the bandwidth (and I assume they have to pay for to their providers)?

CloudFlare does not charge for bandwidth? Their paid plans used to start somewhere around $2000.

Colocation providers charge hundreds of dollars to have an (allegedly) unmetered 100 Mbps network link.

AWS charges network usage per GB. It's the same flat number if you use half the capacity half the time or so.

What do all providers have in common? They charge enough to cover the costs of providing the service and make a profit.

Re: How we use HashiCorp Nomad

#59
post #12

Earlier quoted context omitted.

Helm charts are declarative way of deploying app(s) and their accompanying resources.

Helm, however, is objectively terrible with its yaml-based templating language and zero practical modularity.

Sometimes I even wish they could embed a JavaScript interpreter... After all, YAML is almost equivalent to JSON, which the perfect templating language for JSON is -- JavaScript tbh.

Or people have to keep inventing half baked things.

Re: How we use HashiCorp Nomad

#60
post #12

Earlier quoted context omitted.

Helm, however, is objectively terrible with its yaml-based templating language and zero practical modularity.

Indeed. Helm offers great features but it suffers from the kubernetes unnecessary complexity and by using golang templates in YAML. When I started with kubernetes I converted my small Docker compose files to kubernetes files. Later I rewrote everything in helm charts. Now it's almost more YAML and golang templates lines than business logic lines in my applications. I'm considering to go back to Docker compose files.…

We just moved to the kubernetes ecosystem - specifically for integration with spot instances of AWS (and cut our costs by 60%).

But I hate the UX of kubernetes. Compose was beautiful.

I have hope - since compose files have become an independent specification. https://www.docker.com/blog/announcing-the-compose-specifica...

Kubernetes distro built around the compose file specification would be a unicorn.

Post reply on HN