Earlier quoted context omitted.
No benefit using go over C#, IMO, and I am also baffled by the switch
I just used go as an example, any compiled language would be better.
I highly suspect it wouldn't be better in brainfuck...
Show HN: A Better Log Service
21–30 of 109 posts
Re: Show HN: A Better Log Service
#22My current log solution that is based on Clickhouse I’m tinkering with in free time in Victorialogs. https://docs.victoriametrics.com/victorialogs/
Re: Show HN: A Better Log Service
#23My current log solution that is based on Clickhouse I’m tinkering with in free time in Victorialogs. https://docs.victoriametrics.com/victorialogs/
[deleted]
Re: Show HN: A Better Log Service
#24This is nice. At work, we use Datadog for logging, and I have previously used CloudWatch, Splunk, and Honeycomb. Among these, only Honeycomb makes implementing canonical log lines [1] easier. I want arbitrarily wide, structured logs [2] without paying exorbitant costs for cardinality. Our Datadog costs are outrageous, and it seems like no one cares at this point. Pydantic Logfire is also doing some good work in Pytho…
[deleted]
Re: Show HN: A Better Log Service
#25Don't tell me why, but I've developed an instinct that recognizes solutions that use Clickhouse under the hood :)
Tell us more!
It uses both, MySQL for the metadata and ClickHouse for the logs. The selfhost page explains a bit more about the architecture.
edit: the connection to ClickHouse uses the MySQL driver, this is actually a very nice CH feature, you can connect to CH using the regular mysql or postgresql client tools. The PHP MySQL PDO driver works seamlessly. One catch, using advanced features like CH query timeouts requires a CTE function, check the model/txtlogrowdb.php file if you're interested.
Re: Show HN: A Better Log Service
#26This looks very interesting! My suggestion for the self-hosting is to create docker images and use docker-compose. The self-hosting currently is a bit of effort to setup. I also wonder if PHP is a good language for this. For the UI, yea that's fine and makes sense. But for the log processor that's going to need to handle a high throughput which PHP just isn't good at. For the same resources, you can have Go doing tho…
Thanks for the tip, I will check if inserting rows with Go is any faster. For reference, inserting a log takes three steps, first the log data is stored in a Redis Stream (memory), a number of logs are taken from the stream and saved to disk and finally inserted in batches in ClickHouse. I've created it so you can take the ClickHouse server offline without losing any data (it will be inserted later).
For reference, moving about 4k logs from memory to disk takes less than 0.1 second. This is a real log from one of the webservers:
Start new cron loop: 2024-12-18 08:11:16.397...stored 3818 rows in /var/www/txtlog/txtlog/tmp/txtlog.rows.2024-12-18_081116397_ES2gnY3fVc (0.0652 seconds).
Storing this data in ClickHouse takes a bit more than 0.1 second:
Start new cron loop: 2024-12-18 08:11:17.124...parsing file /var/www/txtlog/txtlog/tmp/txtlog.rows.2024-12-18_081116397_ES2gnY3fVc
* Inserting 3818 row(s) on database server 1...0.137 seconds (approx. 3021.15 KB).
* Removed /var/www/txtlog/txtlog/tmp/txtlog.rows.2024-12-18_081116397_ES2gnY3fVc
As for Docker, I'm too much of a Docker noob but I appreciate the suggestion.
Re: Show HN: A Better Log Service
#27This looks very interesting! My suggestion for the self-hosting is to create docker images and use docker-compose. The self-hosting currently is a bit of effort to setup. I also wonder if PHP is a good language for this. For the UI, yea that's fine and makes sense. But for the log processor that's going to need to handle a high throughput which PHP just isn't good at. For the same resources, you can have Go doing tho…
On the other side some people (me) are happy to have an actual self hosting setup and not being forced to use a docker setup with unknown overhead.
Re: Show HN: A Better Log Service
#28We need glorified (rip)grep instead of ELKS and friends, which have huge learning curve. I welcome this effort.
I'm pretty satisfied with Loki. It just ingests the logs and offers a powerful query language to extract data at query time (e.g. parse JSON, run regexes, plot). It can store data in a local folder or S3-compatible storage. I also gave up on configuring ELK in the past...
Re: Show HN: A Better Log Service
#29Earlier quoted context omitted.
It uses Clickhouse, though, which should be xtremelly fast for this.
Yes. But PHP still needs to process it before it goes to Clickhouse. PHP is the bottleneck.
If that "bottleneck" is thousands of requests per second then it doesn't really matter for smaller deployments does it? (Which seems to be the target audience and not FAANG)
I'm not a big fan when folks call out languages as bottlenecks when they have no proof on the actual overhead and how much faster it would be in another language.
Re: Show HN: A Better Log Service
#30We need glorified (rip)grep instead of ELKS and friends, which have huge learning curve. I welcome this effort.
I'm pretty satisfied with Loki. It just ingests the logs and offers a powerful query language to extract data at query time (e.g. parse JSON, run regexes, plot). It can store data in a local folder or S3-compatible storage. I also gave up on configuring ELK in the past...
yep, loki is basically "distributed grep"