Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Telescope – an open-source web-based log viewer for logs in ClickHouse

github.com

Post reply on HN
Print view View original

71–75 of 75 posts

Re: Show HN: Telescope – an open-source web-based log viewer for logs in ClickHouse

#71

Post by piterrro »

corytheboyd wrote: post #46
piterrro wrote: post #25

There's also Logdy ( https://github.com/logdyhq/logdy-core ) that can work with raw files and comes with a UI as well in a single precompiled binary so no need for installs and setups. If you're looking for a simple solution for browsing log files with a web UI, this might be it! (I'm the author)

Heyo I’ve noticed Lodgy come up a few times on HN now, and was curious if you explored making it a proper desktop application instead of a two-part UI and CLI application. Did you rule that out for some reason?

I'm not ruling that out, however there was no user feedback that's the use case honestly. So far users love that they can just drop a binary on a remote server and sping up a web UI. Similar with the local env. The nature of Logdy is that it's primarily designed to work in the CLI. What would be the use case for a desktop app?

Re: Show HN: Telescope – an open-source web-based log viewer for logs in ClickHouse

#73

Post by nh2 »

sleepybrett wrote: post #70
nh2 wrote: post #66

Earlier quoted context omitted.

Loki works fine for what it does; the problem is what it lacks. It doesn't do full-text search indices. So if you just search for some word across all your logs (to find eg when a rare error happened), it is very slow (it runs the equivalent of grep, at 500 MB/s on my machine). If you have a couple TB, it takes half an hour! As you say, even plain grep is usually faster for such plain linear search. I want full-text…

see to me, having at one point been responsible for maintaining an ES instance for logs (and exporters and all the other bits) I feel like the prices you pay in engineering hours and hardware costs to maintain all those indexes while keeping ES from absolutely melting down is way too high. I think grep is amazing but yes if you unleash it on 'all the logs' without narrowing yourself down to a time frame first or some…

It should not take engineering time to have a database compute full-text indices. In sensible systems, you do "CREATE INDEX" and done.

To search multiple TBs of logs, you need a single 40 $/month server containing an 8 TB SSD running sensible software/index algorithm.

I agree that ElasticSearch is bloated and needs undue engineering time. But it doesn't need to be that way.

For example Quickwit finds things subsecond.

It's a huge improvement when queries go from 10 minutes linear search to instant.

(Its index is still not perfect for me because it doesn't support fully simple exact prefix/infix search, but otherwise it does the job fast with few resources.)

> Full fuzzy text serch is probably overkill

Yes, I think most people don't need fuzzy search for log search. They just need indexed grep.

> I think grep is amazing but yes if you unleash it on 'all the logs' without narrowing yourself down to a time frame first or some other taxonomy is going to be slow. This seems like a skill issue, frankly.

Right, grep is not the tool for the job. It's neglecting all sensible algorithms that solve this problem. It's like saying "I don't use binary search, only linear search", and spend human effort to pre-select the range so that it's fast enough.

When you're searching for the rare bugs, you also can't just limit the the time frame.

Re: Show HN: Telescope – an open-source web-based log viewer for logs in ClickHouse

#74

Post by sleepybrett »

nh2 wrote: post #73
sleepybrett wrote: post #70

Earlier quoted context omitted.

see to me, having at one point been responsible for maintaining an ES instance for logs (and exporters and all the other bits) I feel like the prices you pay in engineering hours and hardware costs to maintain all those indexes while keeping ES from absolutely melting down is way too high. I think grep is amazing but yes if you unleash it on 'all the logs' without narrowing yourself down to a time frame first or some…

It should not take engineering time to have a database compute full-text indices. In sensible systems, you do "CREATE INDEX" and done. To search multiple TBs of logs, you need a single 40 $/month server containing an 8 TB SSD running sensible software/index algorithm. I agree that ElasticSearch is bloated and needs undue engineering time. But it doesn't need to be that way. For example Quickwit finds things subsecond…

I think we might be working at different scales a 40/month server with 8tb of disk would be a puddle on the floor in my current circumstances.

The problem is that many if not most applications have their own log structure so just saying 'index it' doesn't cut it at all.

Re: Show HN: Telescope – an open-source web-based log viewer for logs in ClickHouse

#75

Post by nh2 »

sleepybrett wrote: post #74
nh2 wrote: post #73

Earlier quoted context omitted.

It should not take engineering time to have a database compute full-text indices. In sensible systems, you do "CREATE INDEX" and done. To search multiple TBs of logs, you need a single 40 $/month server containing an 8 TB SSD running sensible software/index algorithm. I agree that ElasticSearch is bloated and needs undue engineering time. But it doesn't need to be that way. For example Quickwit finds things subsecond…

I think we might be working at different scales a 40/month server with 8tb of disk would be a puddle on the floor in my current circumstances. The problem is that many if not most applications have their own log structure so just saying 'index it' doesn't cut it at all.

I'm not sure I understand.

I was talking about what it takes to search through a couple TB of logs. I said that with grep and Loki it's slow due to the linear search, and that indexing systems make it much faster (from many minutes to subsecond).

That's independent of whether you have more than just a couple TB of logs. If you have more, you just get more servers. You'll still get the subsecond results that I find so beneficial.

Post reply on HN