Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beyond Elk: Lightweight and Scalable Cloud-Native Log Monitoring

greptime.com

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Beyond Elk: Lightweight and Scalable Cloud-Native Log Monitoring

#21

Post by atombender »

killme2008 wrote: post #14
atombender wrote: post #13

Earlier quoted context omitted.

Thanks, that seems promising. So much of the documentation is schema-oriented, I didn't see that it supported dynamic schemas. I find it interesting that Greptime is completely time-oriented. I don't think you can create tables without a time PK? The last time I needed log storage, I ended up picking ClickHouse, because it has no such restrictions on primary keys. We use non-time-based tables all the time, as well as…

Yes, GreptimeDB requires a time index column for optimized storage and querying. It's not a constraint of a primary key, but just an independent table constraint. Could you elaborate on why you find this inconvenient? I assumed logs, for example, would naturally include a timestamp.

It's less convenient because it makes the database less general-purpose. The moment you need to go beyond time-based data, you have to reach for other tools.

ClickHouse is such a wonderful database precisely it's so incredibly flexible. While most data I interact with is time-based, I also store lots of non-time-based data there to complement the time-based tables. The rich feature set of table engines, materialized views, and dictionaries means you have a lot of different tools to pick from to design your solution. For example, to optimize ETL lookup, I use a lot of dictionaries, which are not time-based.

As an example, let's say I'm ingesting logs into Greptime and some log lines have a customer_id. I would like the final table, or least a view, to be cross-referenced with the customer so that it can include the customer's name. I suppose one would have to continually ingest customer data into a Greptime table with today's date, and then join on today's date?

Re: Beyond Elk: Lightweight and Scalable Cloud-Native Log Monitoring

#22

Post by atombender »

Reading the web site, I just noticed the open-source version does not have "Log query endpoints".

Does that mean you have to use SQL (or the visual SQL builder) to query logs, and you don't get access to a log query language the way Kibana gives you KQL and Lucene syntax?

If so, I think it's a little disingenuous to write an article comparing the ELK stack, which is open source and comes with a perfectly usable query UI, to Greptime's equivalent, which is not.

Re: Beyond Elk: Lightweight and Scalable Cloud-Native Log Monitoring

#23

Post by killme2008 »

atombender wrote: post #21
killme2008 wrote: post #14

Earlier quoted context omitted.

Yes, GreptimeDB requires a time index column for optimized storage and querying. It's not a constraint of a primary key, but just an independent table constraint. Could you elaborate on why you find this inconvenient? I assumed logs, for example, would naturally include a timestamp.

It's less convenient because it makes the database less general-purpose. The moment you need to go beyond time-based data, you have to reach for other tools. ClickHouse is such a wonderful database precisely it's so incredibly flexible. While most data I interact with is time-based, I also store lots of non-time-based data there to complement the time-based tables. The rich feature set of table engines, materialized…

Fair point. Joining time-series data with business data is often necessary. While GreptimeDB currently supports external tables for Parquet and CSV files, we plan to expand this support to include datasources like MySQL and PG in the future.

Re: Beyond Elk: Lightweight and Scalable Cloud-Native Log Monitoring

#24

Post by killme2008 »

atombender wrote: post #22

Reading the web site, I just noticed the open-source version does not have "Log query endpoints". Does that mean you have to use SQL (or the visual SQL builder) to query logs, and you don't get access to a log query language the way Kibana gives you KQL and Lucene syntax? If so, I think it's a little disingenuous to write an article comparing the ELK stack, which is open source and comes with a perfectly usable query…

In fact, we have an open-source query language, but it's still in experimental, so we don't present it on the website. The description of the enterprise feature is not precise. Sorry for the inconvenience.

GreptimeDB also open-sources the log view UI if you read the article.

I agree with you that ETL is so powerful, and GreptimeDB is so young, we still have lots of work to do. Thank you.

Re: Beyond Elk: Lightweight and Scalable Cloud-Native Log Monitoring

#25

Post by killme2008 »

reconnecting wrote: post #20
killme2008 wrote: post #11

Earlier quoted context omitted.

Thanks for pointing it out! The footer has been updated.

Thank you for your prompt attention to this matter. Until next year, then.

We'll find a way to fix it forever :D

Re: Beyond Elk: Lightweight and Scalable Cloud-Native Log Monitoring

#26

Post by killme2008 »

up2isomorphism wrote: post #18

This space is so crowded, I think any new startup is very unlikely to survive, unless it solves its own business case first.

Yes, so many startups are trying to solve the log issue in the current stack.

In my personal observation, the vast majority of startups are still focused on the product layer and use ClickHouse directly for storage. However, ClickHouse’s tightly coupled storage and compute architecture makes it difficult to scale, and this becomes a real problem as workloads grow. GreptimeDB, on the other hand, is more focused on being an all-in-one observability database. Our log UI, however, still has quite a gap compared to products like Kibana.

This space is very crowded. I think it’s unlikely that any new startup will succeed here unless it can first solve its own business use case exceptionally well.

Would love to hear your thoughts.

Re: Beyond Elk: Lightweight and Scalable Cloud-Native Log Monitoring

#27

Post by atombender »

killme2008 wrote: post #24
atombender wrote: post #22

Reading the web site, I just noticed the open-source version does not have "Log query endpoints". Does that mean you have to use SQL (or the visual SQL builder) to query logs, and you don't get access to a log query language the way Kibana gives you KQL and Lucene syntax? If so, I think it's a little disingenuous to write an article comparing the ELK stack, which is open source and comes with a perfectly usable query…

In fact, we have an open-source query language, but it's still in experimental, so we don't present it on the website. The description of the enterprise feature is not precise. Sorry for the inconvenience. GreptimeDB also open-sources the log view UI if you read the article. I agree with you that ETL is so powerful, and GreptimeDB is so young, we still have lots of work to do. Thank you.

Thanks, sounds interesting. It's actually not at all clear from the article that the UI, as presented, is open source. I'm looking for an ELK replacement (in an enterprise setting), so it sounds like Greptime is something I might be able to use.

Re: Beyond Elk: Lightweight and Scalable Cloud-Native Log Monitoring

#28

Post by killme2008 »

atombender wrote: post #27
killme2008 wrote: post #24

Earlier quoted context omitted.

In fact, we have an open-source query language, but it's still in experimental, so we don't present it on the website. The description of the enterprise feature is not precise. Sorry for the inconvenience. GreptimeDB also open-sources the log view UI if you read the article. I agree with you that ETL is so powerful, and GreptimeDB is so young, we still have lots of work to do. Thank you.

Thanks, sounds interesting. It's actually not at all clear from the article that the UI, as presented, is open source. I'm looking for an ELK replacement (in an enterprise setting), so it sounds like Greptime is something I might be able to use.

Thanks for your feedback. We fixed the descriptions of log query endpoints. Hope it's more clear. Glad you're considering giving it a try and looking forward to your feedback.
Post reply on HN