Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building Netflix's Distributed Tracing Infrastructure

netflixtechblog.com

Post reply on HN
Print view View original

1–10 of 19 posts

Re: Building Netflix's Distributed Tracing Infrastructure

#3

Post by user5994461 »

mnsmns wrote: post #2

Curious to know if any TSDBs (Timeseries databases) evaluated before finalizing on Cassandra to store the traces.

I don't think there's any free TSDB available. The project is maybe 3 years old.

InfluxDB is more recent and quite limited. Things like sharding is not supported and they stated it would never be supported except in a paid edition when they make one.

Prometheus is more recent. Similar story with scaling. They changed storage formats and rewrote once or twice in the past few years, it's moving really fast. It's more of a standalone product for server metrics (node exporter + prometheus + grafana), wouldn't recommend to use as a general purpose database.

Re: Building Netflix's Distributed Tracing Infrastructure

#6

Post by 02020202 »

user5994461 wrote: post #3
mnsmns wrote: post #2

Curious to know if any TSDBs (Timeseries databases) evaluated before finalizing on Cassandra to store the traces.

I don't think there's any free TSDB available. The project is maybe 3 years old. InfluxDB is more recent and quite limited. Things like sharding is not supported and they stated it would never be supported except in a paid edition when they make one. Prometheus is more recent. Similar story with scaling. They changed storage formats and rewrote once or twice in the past few years, it's moving really fast. It's more o…

s you're saying that big tech does not want to PAY for using big tech? :D

Re: Building Netflix's Distributed Tracing Infrastructure

#7

Post by myvoiceismypass »

davydog187 wrote: post #4

I find it interesting that they do not mention https://opentelemetry.io/ at all

Too new or not mature enough at the time: “When we started building Edgar four years ago, there were very few open-source distributed tracing systems that satisfied our needs.”

Re: Building Netflix's Distributed Tracing Infrastructure

#9

Post by jrockway »

davydog187 wrote: post #4

I find it interesting that they do not mention https://opentelemetry.io/ at all

OpenTelemetry is just an API for instrumenting your software. For actually collecting and viewing your traces, you'll need an actual implementation -- Jaeger and Zipkin are some open source options.

If you develop an open source project and one of your goals is "I sure wish my users could use Datadog instead of being forced to use an open source thing!" then OpenTelemetry is a project you should keep an eye on. The goal is to make the instrumentation agnostic to the underlying provider.

Re: Building Netflix's Distributed Tracing Infrastructure

#10

Post by jrockway »

mnsmns wrote: post #2

Curious to know if any TSDBs (Timeseries databases) evaluated before finalizing on Cassandra to store the traces.

I don't think time is a particularly strong keying methodology for distributed tracing. You want to be able to quickly select all spans that have a certain trace ID. The time that the span was ingested is relatively unimportant to the querying mechanics, and you don't do any operations like "combine samples into a 1h average after 5d".
Post reply on HN