171–178 of 178 posts
Earlier quoted context omitted.
Ah, I meant use your hosting provider's load balancer + use their VPS-es. I think Digital Ocean allows this, AWS and GCP as well. That way you're outsourcing the problem of "but what if the load balancer itself falls down?".
I usually want to load-balance between different hosting providers (so if AWS goes down, I still have DO running). From what I've seen, most of them only provide load-balancing within their local networks.
Earlier quoted context omitted.
I usually want to load-balance between different hosting providers (so if AWS goes down, I still have DO running). From what I've seen, most of them only provide load-balancing within their local networks.
Hmmmm... Cloudflare then?
Also, I'm starting to dislike having everything go through Cloudflare (all my DNS). I don't dislike the company, I just don't like one entity controlling the entire internet. I don't want even more vendor-lock-in tbh.
I'm gravitating towards the failover solution, to have a VPS do health-checks on my services, if one goes down, use the Cloudflare API to update the DNS entries to the backup servers.
Earlier quoted context omitted.
Hmmmm... Cloudflare then?
That's a good idea, I was considering it, looked at their pricing, but was too expensive. Also, I'm starting to dislike having everything go through Cloudflare (all my DNS). I don't dislike the company, I just don't like one entity controlling the entire internet. I don't want even more vendor-lock-in tbh. I'm gravitating towards the failover solution, to have a VPS do health-checks on my services, if one goes down,…
I sympathize with your reservations, it's just that at this stage of my life I am OK with having some loose ends in my infrastructure. To me CF is a good tradeoff (also I am pretty sure their free tier is quite generous? but maybe I am thinking different kinds of services) for the moment.
But in light of the recent French court order to main providers (CF included) to poison DNS in order to fight the anti-piracy circumventing users... maybe we should just move the entire control plane to Tor, or use the regular internet but doubly/triply encrypted, or use stuff like Freenet / NNCP etc.
I'll arrive at your conundrum soon enough though, I too want complete independence but I am slightly scared because the current internet infrastructure is constantly under attack to make it more malleable to censorship. Whatever people might think of piracy or any other scapegoat ("think of the children" seems to always be the adage) censorship is still not okay.
Earlier quoted context omitted.
I use sentry selfhosted and started with Otel under gcloud and grafana and everything worked. Perfectly even, with low effort in dotnet. However with sentry it’s still a pain and the visualization in sentry is kinda weird, since it goes beyond tracing. And since sentry itself has no otel endpoint it is also really hard to do things like tail sampling.
Can you give more context on what you mean around sampling. What does an OTLP endpoint have to do with it?
we do use it quite extensivly besides the overhead, because we get lots of insights wenn we have a slow running endpoint.
Earlier quoted context omitted.
OTel is very easy to add.. I've added it to several Go projects. For some frameworks like .NET you can do it automatically. The harder/more annoying part is setting up a viewer/collector like Jaeger. I've done that too but just in memory and it fills up quick.
For my small scale projects, Openobserve.ai has been super helpful. It ships as a single binary and (in non h/a setup) saves traces/logs/metrics to disk. I just set it up as a systems service and start sending telemetry via localhost. Code at https://github.com/bbkane/shovel_ansible/
Earlier quoted context omitted.
Well, telemetry is defined as logs, metrics, traces... So it kinda makes sense that OTEL supports the major aspects of telemetry.
Says who? Sentry has many other types of telemetry and we’ve existed long before OTel. Who are these all knowing humans who say this is what telemetry is? Are they also going to build every collector for every kind of past current telemetry? The whole idea that some marketing bs has translated to technology fact is why we’re in this mess.
Earlier quoted context omitted.
Whether it is affecting 5% or 10% of your customers, if it is erroring at that rate you are going to want to find the root cause ASAP. Traces let you do that, whereas the precise number does nothing. I am a big supporter of metrics but I don't see this as the use case at all.
(not your OP) This is true, but I find that metrics are useful whether something is going wrong or not (metrics that show 100% success are useful in determining baselines and what "normal" is), whereas collecting traces _when nothing is going wrong_ is not useful -- it's just taking up space and ingress, and thus costing me money. My typical approach in the past has been to use metrics to determine when something is…
Not all problems result in error traces to analyse.
Example, you release buggy client that doesn't call "POST /order/finalize" when it should.
There are no error traces, there are just missing HTTP requests. Metrics reveal that calls to "POST /order/finalize" for iOS apps are down 50% WoW.
Earlier quoted context omitted.
With only sampled traces though it’s very hard to understand the impact of the problem. There are some bad traces but is it affecting 5%, 10% or 90% of your customers. Metrics shine there.
Strange example, you'd think you want to fix this as quickly as humanly possible, no? Also we don't sample traces, it's a fire hose of data aimed at the OTel collector. We do archive them / move them to colder and cheaper storage after a little time though, and we found that a viable money-saving strategy and a good balance overall.
Example, you release buggy client that doesn't call "POST /order/finalize" when it should.
There are no error traces, there are just missing HTTP requests. Metrics reveal that calls to "POST /order/finalize" for iOS apps are down 50% WoW.