31–40 of 109 posts
We need glorified (rip)grep instead of ELKS and friends, which have huge learning curve. I welcome this effort.
I'm pretty satisfied with Loki. It just ingests the logs and offers a powerful query language to extract data at query time (e.g. parse JSON, run regexes, plot). It can store data in a local folder or S3-compatible storage. I also gave up on configuring ELK in the past...
I've heard good things about Axiom[0], especially for high scale needs. 0: https://axiom.co/
Don't tell me why, but I've developed an instinct that recognizes solutions that use Clickhouse under the hood :)
We need glorified (rip)grep instead of ELKS and friends, which have huge learning curve. I welcome this effort.
Very nice. A lot of the complexity you described is why I've settled on using CloudWatch logs for anything I have on AWS. I don't need a fancy UI, just a powerful querying language for investigation and debugging. With that said, it would be nice to see at least some mechanism for building aggregates queries (for example, 4* results in the last 24 hours by user) but if it's ClickHouse underneath, I assume that's easy…
We need glorified (rip)grep instead of ELKS and friends, which have huge learning curve. I welcome this effort.
Pretty unrelated, but i like how it displays large amount of potentially diverse JSON events. Would need some better filtering and sorting, hiding of keys etc. Products which do this well are Elastic and Splunk, but are too heavy for my taste.
Earlier quoted context omitted.
Yes. But PHP still needs to process it before it goes to Clickhouse. PHP is the bottleneck.
If that "bottleneck" is thousands of requests per second then it doesn't really matter for smaller deployments does it? (Which seems to be the target audience and not FAANG) I'm not a big fan when folks call out languages as bottlenecks when they have no proof on the actual overhead and how much faster it would be in another language.
Most PHP deployments barely reach a hundred per server.
And this is an open source project is should be designed to handle basic production workloads which it could but it'll cost you a bunch more than if you used the correct languages.
> I'm not a big fan when folks call out languages as bottlenecks when they have no proof on the actual overhead and how much faster it would be in another language.
Honestly, I thought it was so obvious that an interpreted language is not good for high throughput endpoints that it didn't need to be proven. I also thought it was obvious that a logging system is going to handle lots and lots of data.
It could be easily proven by doing a bunch of work but obviously there is no point in me proving it.
This looks very interesting! My suggestion for the self-hosting is to create docker images and use docker-compose. The self-hosting currently is a bit of effort to setup. I also wonder if PHP is a good language for this. For the UI, yea that's fine and makes sense. But for the log processor that's going to need to handle a high throughput which PHP just isn't good at. For the same resources, you can have Go doing tho…
Go and friends may make for more efficient resource utilization, but it will be marginal in the grand scheme of things unless there are plans to do massively different things.
As it is this code is very simple. I haven't used PHP in 15 years and I was able to trace through this from front-end to back-end in less than 3 minutes.
To me it look like a really great level of complexity for the problem it solves.
Keep it up, OP.