Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monitoring my Minecraft server with OpenTelemetry and Prometheus

dash0.com

Post reply on HN
Print view View original

41–50 of 64 posts

Re: Monitoring my Minecraft server with OpenTelemetry and Prometheus

#41

Post by mmanciop »

dewey wrote: post #29
strogonoff wrote: post #28

Earlier quoted context omitted.

I agree, and I am as guilty of procrastination. However, the author is not really procrastinating—he gets paid for this. Me, I do in fact procrastinate on setting up a Minecraft server infra in the cloud. Maybe that’s precisely why the solution to this problem strikes me as inadequate: > So, the Minecraft server should work reliably and, if it goes down, I should know well before they do How are metrics helpful? Ther…

You are somehow very negative about this piece and are not understanding that your definition of fun is not universal. As you said "There is so much fun that could be had in setting up an actually resilient system instead.", maybe the author has more fun setting up alerts and metrics instead of a resilient system like you do? The truth is that in most real-world scenarios getting alerts, metrics is much more importan…

> As you said "There is so much fun that could be had in setting up an actually resilient system instead.", maybe the author has more fun setting up alerts and metrics instead of a resilient system like you do?

Adding the backup for the world files, already having Systemd bringing back a crashing server, makes the setup rather resilient. Sure, there's infinite more things that can go wrong, but with swiftly decreasing likelihood.

> The truth is that in most real-world scenarios getting alerts, metrics is much more important than building a fully resilient system (Expensive, maybe overengieering for early stage etc.).

This, very much this.

> However, the author is not really procrastinating—he gets paid for this. As the first sentence in the blog post says "One of the secret pleasures of life is to be paid for things you would do for free.", which I can very much understand as I often work or play with things I could use at work in my free time.

Yes :-)

Re: Monitoring my Minecraft server with OpenTelemetry and Prometheus

#42

Post by mmanciop »

treyd wrote: post #33
strogonoff wrote: post #27

Earlier quoted context omitted.

From my understanding, Paper and the like are good for Minecraft servers focused around specific mini-games (rather than freedorm building), and are the only sensible choice for servers with many people (or not that many people, but really underpowered hardware). However, they may be a problem if players are sensitive to possible non-vanilla behaviour (as you mentioned, and it’s not limited to cheaty duping). Thankfu…

It's kinda nuts. The upstream mojang server binary starts to groan if you have >4-5 players on the same server doing stuff. They've really been dropping the ball on optimization in recent years. Paper is good enough for anyone but very technical players pushing to the limits of redstone tick timing logic, entity behavior, chunk loading mechanics, etc. These don't matter even for advanced players doing normal things.

I actually had to splurge got 2 VCPUs on Digital Ocean to avoid "skipping ticks" and it does sound pretty nuts to me. We play max 3 players. I would expect the server with such a load to be able to run on a slightly tuned up toaster.

Re: Monitoring my Minecraft server with OpenTelemetry and Prometheus

#43

Post by mmanciop »

ajmurmann wrote: post #13
strogonoff wrote: post #10

Earlier quoted context omitted.

Well, as far as I’m concerned, if they are toy apps, why stress? If they are going to go in production at some point, then sure; but this certainly is not happening with a family game server.

Family game server going down can be very stressful, especially if you have kids. Also, I've had phone tech support sessions with family that were more stressful than calls with large banks who were worried about losing very large amounts of money in case of an outage. Different stressful, but nonetheless...

Indeed.

My personal definition of nanosecond is the time passing between the Minecraft server having a hiccup, and the first scream piercing the air.

The printer not printing is DEFCON 5 material.

Re: Monitoring my Minecraft server with OpenTelemetry and Prometheus

#44

Post by mmanciop »

strogonoff wrote: post #24
jauntywundrkind wrote: post #20

Earlier quoted context omitted.

I know I sound like a freak to you, but you sound like a deranged freak to me too. Who would opt for ignorance? Who would opt not to have data? Who would opt not to see more? Its insanity to me to resist enrichment so. Limiting yourself to only naive senses is a wild proposition to me. The scientific mindset compels us to see further: it is a wild privilege to see more, to build and use tools that expand how we can s…

I don’t think you’re deranged. I do think this is a post about using telemetry 1) in (to me) excessive ways that defeat the point of the thing being measured 2) published on the website of a company that sells said telemetry solution. Furthermore, to me useless or excessive data is very much a reality (if you do not agree that it is a possibility and a thing that happens, we have clearly no way of understanding each…

To be fair, the setup of the article works with most modern observability solutions, in same cases just by replacing the endpoint and authentication token. Turning telemetry processing into a sort of utility is one of the great things that OpenTelemetry did. Now, among vendors, we compete on delivering insights on the telemetry, as opposed to just collecting it. If you are interested, I wrote about it a while back [1].

About excessive telemetry, that depends on what you want to achieve. Using facilities in the OpenTelemetry Collector like [2], you can easily drop all telemetry you have no use for. At the cost of tooting my own horn, we actually provide super easy ways of doing the same dropping at no charge whatsoever to the end user in Dash0 [3].

[1] https://thenewstack.io/is-otel-the-last-observability-agent-...

[2] https://github.com/open-telemetry/opentelemetry-collector-co...

[3] https://www.dash0.com/changelog/spam-filters

Re: Monitoring my Minecraft server with OpenTelemetry and Prometheus

#45

Post by mmanciop »

strogonoff wrote: post #14
ajmurmann wrote: post #13

Earlier quoted context omitted.

Family game server going down can be very stressful, especially if you have kids. Also, I've had phone tech support sessions with family that were more stressful than calls with large banks who were worried about losing very large amounts of money in case of an outage. Different stressful, but nonetheless...

> Family game server going down can be very stressful, especially if you have kids. Telemetry does not address this, though. Shoving it into a container and assigning it a simple “restart if down” rule does. Minecraft is a flaky beast, if you run snapshots and/or mods. Metrics or not, often “start again” is all you need. Furthermore, this is a game that adds new gameplay features multiple times per month. If you do n…

> Telemetry does not address this, though. Shoving it into a container and assigning it a simple “restart if down” rule does.

A Systemd unit as shown in [1] does it too without using containers and with fewer moving parts of using containers. I use containers every day at $work. I have been using containers since before Docker was a thing. In this case, it's entirely overkill: Systemd units use the important things like cgroups already.

For the upgrade: depends. You do need a container image regardless, and I have not seen official ones. Upgrading servers in Minecraft requires upgrading clients to match, and my kids prefer to play, more than upgrade. (Unless a biome is released. Then it must be immediately available to them.) But then again, I just need to download the binary with a cURL call. And if the configurations change, Docker won't help me there one bit anyhow.

[1] https://github.com/dash0hq/minecraft-server/blob/main/drople...

Re: Monitoring my Minecraft server with OpenTelemetry and Prometheus

#46

Post by Yasuraka »

mmanciop wrote: post #36
Yasuraka wrote: post #4

Earlier quoted context omitted.

Why are you doing this?

Because I enjoy observability and monitoring a LOT, and because my kids nag me to hell and back when our home IT infrastructure is having a bad day.

I was asking some LLM spammer

Re: Monitoring my Minecraft server with OpenTelemetry and Prometheus

#47

Post by skrtskrt »

mmanciop wrote: post #42
treyd wrote: post #33

Earlier quoted context omitted.

It's kinda nuts. The upstream mojang server binary starts to groan if you have >4-5 players on the same server doing stuff. They've really been dropping the ball on optimization in recent years. Paper is good enough for anyone but very technical players pushing to the limits of redstone tick timing logic, entity behavior, chunk loading mechanics, etc. These don't matter even for advanced players doing normal things.

I actually had to splurge got 2 VCPUs on Digital Ocean to avoid "skipping ticks" and it does sound pretty nuts to me. We play max 3 players. I would expect the server with such a load to be able to run on a slightly tuned up toaster.

to be fair with the power on most people's laptops and phones now I think we tend to lose track of just how little "1 CPU" is if you're not just running like, a small web app.

Re: Monitoring my Minecraft server with OpenTelemetry and Prometheus

#48

Post by ehnto »

doabell wrote: post #21

> I am a man of simple tastes, and running the “vanilla” Minecraft server as a Systemd unit on a Linux VM in the cloud Minecraft is famously under-optimized and needy in terms of CPU frequency. If running a vanilla (no server mods) version, then using something optimized, like PaperMC is a better idea for datacenter VMs. (Until you need to dupe sand or something.) The other route is installing a bunch of optimization…

People love to bother about Java MC performance, but I ran a modded Tekkit sever for like 10 years on a base Digital Ocean VM. Shoutout to Digital Ocean for having no impactful changes for 10 years too. They give me a VM, I run the thing, life is good.

Re: Monitoring my Minecraft server with OpenTelemetry and Prometheus

#49

Post by strogonoff »

mmanciop wrote: post #42
treyd wrote: post #33

Earlier quoted context omitted.

It's kinda nuts. The upstream mojang server binary starts to groan if you have >4-5 players on the same server doing stuff. They've really been dropping the ball on optimization in recent years. Paper is good enough for anyone but very technical players pushing to the limits of redstone tick timing logic, entity behavior, chunk loading mechanics, etc. These don't matter even for advanced players doing normal things.

I actually had to splurge got 2 VCPUs on Digital Ocean to avoid "skipping ticks" and it does sound pretty nuts to me. We play max 3 players. I would expect the server with such a load to be able to run on a slightly tuned up toaster.

It is not cheap for the cloud. Had to use some beefy variety of EC2 medium instance for 4 players or so, with a simple dash for starting it up and terminating, I think using spot instance pricing. Otherwise it cost a pretty penny. At that point I did not use any performance mods, though.

Re: Monitoring my Minecraft server with OpenTelemetry and Prometheus

#50

Post by strogonoff »

dewey wrote: post #29
strogonoff wrote: post #28

Earlier quoted context omitted.

I agree, and I am as guilty of procrastination. However, the author is not really procrastinating—he gets paid for this. Me, I do in fact procrastinate on setting up a Minecraft server infra in the cloud. Maybe that’s precisely why the solution to this problem strikes me as inadequate: > So, the Minecraft server should work reliably and, if it goes down, I should know well before they do How are metrics helpful? Ther…

You are somehow very negative about this piece and are not understanding that your definition of fun is not universal. As you said "There is so much fun that could be had in setting up an actually resilient system instead.", maybe the author has more fun setting up alerts and metrics instead of a resilient system like you do? The truth is that in most real-world scenarios getting alerts, metrics is much more importan…

> The truth is that in most real-world scenarios getting alerts, metrics is much more important than building a fully resilient system (Expensive, maybe overengieering for early stage etc.).

Funny, because I have the opposite opinion. Build for failure first; if it’s critical/production then also monitor, but if an earthquake takes down an EC2 zone and you have no ability to spin it up exactly the way it was then the avalanche of alerts and metrics falling off a cliff[0] isn’t exactly going to help you (or your mental well-being).

Generally speaking, if you build for failure first, then monitoring becomes much more useful and actionable; and simultaneously it becomes much less important for a hobby project.

[0] That assuming you gather them from a different zone that wasn’t affected by the same downtime in the first place; speaking of, how are you monitoring your monitors? and so on.

Post reply on HN