Supabase Logs: open-source logging server

BigQuery reduces storage costs as even before their recent pricing change, the cost per GB [0] is on par with and slightly lower than s3 storage costs [0], which we can use as a estimated market price for data storage. BigQuery makes money off the querying costs, which we take steps to optimize and minimize, with table partitions, caching, and ui design, both client side and server side. These all help to reduce querying costs. Table partitioning also helps to cut per-GB storage costs by half by switching to long-term logical storage after 90 days.

Of course, using blog storage might possibly result in comparable cost, however, relying on blob storage would likely increase the querying costs (in terms of GET requests) as well as complexity to query across multiple stored objects/buckets, as opposed to relying on BQ to handle the querying.

In the long term, we would likely continue using BQ for our platform infrastructure, unless GCP changes their pricing in a way that adversely affects us. When it comes to self-hosting, it would of course depend on how much complexity one would like to take on, and out-sourcing the storage and querying management is a better option in most cases.

We would not rule out such features, but we consider them nice-to-have features and are very far down the priority list. At the moment we're mostly focused on improving integration with the Supabase products and platform. It is actually possible to log stack traces, and is supported out-of-the-box for certain integrations such as the Logflare Logger Backend [2].

Postgres without any extensions is not optimized for columnar storage, and would not be an optimal experience for such large scale data storage and querying. It is also not advisable to use the same production datastore for both your application and your observability, it is better to keep them separate to avoid coupling. If one really wants to use the same Postgres server for everything, there are extensions that allow for Postgres to work as a columnar store, such as citus[3] and timescaledb[4], and we have not ruled out supporting these extensions as Logflare backends.

[0]: https://cloud.google.com/bigquery/pricing#storage

[1]: https://aws.amazon.com/s3/pricing/

[2]: https://github.com/Logflare/logflare_logger_backend

[3]: https://github.com/citusdata/citus

[4]: https://github.com/timescale/timescaledb

edit: Ah looks like Chase answered as well :)

I have not seen anything open source that really separates compute and storage like that in one package. You can setup Clickhouse to use an object store but there's a bunch of nuance there. People are starting to hack together something similar using DuckDB on Lambda, basically as a query engine on top of S3. I wonder if you can use DuckDb to get a SQL interface going?

Going to have to take a close look for sure. SQL is definitely a blocker. With Supabase we give people the ability to query their logs with full SQL. And with Logflare Endpoints you create an API endpoint with a SQL query.

Supabase Logs / Logflare seems primarily interested in creating graphs from logs rather than using logs for diagnostic purposes.

I've been looking for a log solution that's good for the use case of high retention but low volume.

I have a few small apps that generate a few MB of logs per month, so basically nothing. But I still want to have all my logs searchable in one place.

Most logging solutions set retention based on time rather than data size. So regardless of how much you're logging, they throw away your logs within somewhere between 7-30 days unless you're on an insane Enterprise plan.

I was excited about LogFlare because it supports unlimited retention, but I ran into too many issues and had to cancel my subscription:

* To search your logs, you need to write a SQL-like query in LogFlare's DSL. You can't just put in a route (e.g. /api/auth) like you can with other log analytics.

* Search only shows the matching lines. Usually, what I want to see is the log line in context. For example, if I search "catastrophic error" I want to see the log lines leading up to that, not just that specific line.

* Search is limited to a maximum of 100 results. If you want to see more results, you need to rewrite your query rather than just scroll up or hit a "load more" button.

* When you do adjust the query to a larger time window, the query will fail because it can't generate a graph unless you also adjust the group_by in your query to match the new time window's limits. This is an annoying obstacle if you don't care about graphing the results and are just trying to diagnose an issue in your logs.

I found support lacking as well. I emailed support to ask if I was misunderstanding how to use Logflare or if it was just designed for a different use case. I was on a paid plan, but I still had to wait 3 business days for a response. When the response came, they just said that it was designed for me but didn't address any of the issues I brought up.

I do like that Logflare/Supabase let you bring your own BigQuery. That's nice for customers like me who want low-volume, high retention. I hope they continue iterating because it has potential.

In the meantime, I've found LogTail to be a pretty good alternative, but they're limited to 30 days of retention even on the highest tier plan.

Also does anybody know what they're doing behind the scenes with the database? I know their storage uses s3, functions (I think) use Deno, this uses BigQuery. Is their db on RDS/Aurora? If so how do they claim max DB size of 1024 TB while Aurora is 128 TB?