11–16 of 16 posts
logrotate, systemd, `journalctl -u server-name | grep whatever`
Victoriametrics for metrics (7x ram reduction vs prom), Loki for logs but evaluating Victorialogs. Then grafana for ui. Cannot recommend vm enough, it’s fantastic.
If you hit some issues with VictoriaLogs or if you have ideas on how VictoriaLogs usability could be improved, then please file issues at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/ . We appreciate users' input and always try making our products easier to use.
logrotate, systemd, `journalctl -u server-name | grep whatever`
What if the machine is unavailable? It is better to store logs from multiple hosts into a centralized database, so the logs could be investigated even if the original host is no longer available for any reason.
If the server is unavailable, then my entire product is missing because the entire product is on one server including the database.
I use that stack for this self hosted dashboard that I wrote. I can spin up a new web server or docker container in seconds. The web servers have proxies automatically provided and serve WebSockets on the same port as http. The dashboard tool also provides remote terminal and file system access. Everything displays in a web browser.
I run PiHole for DNS in my stack. I have toyed with the idea of writing my own PiHole alternative but taken no such action yet.
I also run PhotoPrism, Jellyfin, Meali, and some other stuff.
Here's our stack:
Logs: We self-host a simple stack using Fluentd to collect logs, which are then shipped to Elasticsearch for storage and analysis. It's powerful, gives us full control over our data, and is more cost-effective at our scale than a managed logging service.
Metrics & Monitoring: For this, we decided not to reinvent the wheel. We use Datadog. The out-of-the-box dashboards, alerting, and deep integration with our cloud provider (AWS/GCP) save our small team hundreds of hours. The cost is justified by the engineering time we save.
Traces: We're currently using Datadog's APM for tracing as well, as it's tightly integrated. However, we're actively exploring moving to OpenTelemetry for more vendor neutrality in the future.
It's working out well. The key has been to be honest about our most valuable resource: engineering time. We self-host where we have a clear need for control (logs), and we pay for a service where the platform provides undeniable value and speed (metrics/monitoring).
Interesting thread. We're building Markhub, a B2B collaboration SaaS, and we have a pragmatic, hybrid approach to our observability stack. Our philosophy is: use open source where it gives us control and flexibility, and use managed SaaS where it saves us time and engineering overhead. Here's our stack: Logs: We self-host a simple stack using Fluentd to collect logs, which are then shipped to Elasticsearch for storag…
The DD tradeoff is a capricious one. Hats off to running a tight ship since your DD bill would let you know if you let complexity get the better of your stack (;