Supabase Logs: open-source logging server
supabase.com
Supabase Logs: open-source logging server
1–10 of 68 posts
Re: Supabase Logs: open-source logging server
#2> Logflare was available under a BSL license prior to joining Supabase. We’ve since changed the license to Apache 2.0, aligning it with our open source philosophy.
I wish more companies were like this!
(note, the vector.dev link is broken)
Re: Supabase Logs: open-source logging server
#3https://github.com/Logflare/logflare/tree/staging
> Simply provide your BigQuery credentials and we stream logs into your BigQuery table while automatically managing the schema
I didn't know BigQuery was capable of accepting streaming log data - in my mental model of the world it was the kind of database that you update using the occasional batch job, not from a streaming source of data.
Looks like that's the tabledata.insertAll method which has been around for quite a few years - though it's now called the "legacy streaming API" on https://cloud.google.com/bigquery/docs/streaming-data-into-b... which suggests using the more recent Storage Write API instead: https://cloud.google.com/bigquery/docs/write-api
Re: Supabase Logs: open-source logging server
#4https://github.com/Logflare/logflare/tree/staging > Simply provide your BigQuery credentials and we stream logs into your BigQuery table while automatically managing the schema I didn't know BigQuery was capable of accepting streaming log data - in my mental model of the world it was the kind of database that you update using the occasional batch job, not from a streaming source of data. Looks like that's the tableda…
It's pretty awesome & very cost effective! When I was at Twilio we streamed logstash/fluentd (forget which) -> kinesis -> BigQuery and it worked great - certainly better than the days we were trying to manage ES ourselves
Re: Supabase Logs: open-source logging server
#5I weep a bit every time I see a new Supabase capability knowing I'm stuck on Firebase.
Re: Supabase Logs: open-source logging server
#6It would be awesome if this could use Quickwit as a backend which is a new promising alternative to Elasticsearch, I’ve been using it internally and it’s much more lightweight and easier to run.
Re: Supabase Logs: open-source logging server
#7I weep a bit every time I see a new Supabase capability knowing I'm stuck on Firebase.
What's stopping you from coming over to Supabase?
Re: Supabase Logs: open-source logging server
#8https://github.com/Logflare/logflare/tree/staging > Simply provide your BigQuery credentials and we stream logs into your BigQuery table while automatically managing the schema I didn't know BigQuery was capable of accepting streaming log data - in my mental model of the world it was the kind of database that you update using the occasional batch job, not from a streaming source of data. Looks like that's the tableda…
It's pretty awesome & very cost effective! When I was at Twilio we streamed logstash/fluentd (forget which) -> kinesis -> BigQuery and it worked great - certainly better than the days we were trying to manage ES ourselves
Were people using it as an actual ES replacement for logs? fluentd -> GCP log sink -> BQ can get awful if you need to work with it on daily basis
Re: Supabase Logs: open-source logging server
#9I weep a bit every time I see a new Supabase capability knowing I'm stuck on Firebase.
Firebase uses Google Cloud Logging. Taking a quick look at the blog post here, Google Cloud Logging already seems to support everything it describes.
Is there something in it that makes it a better solution in some way than what Google is already providing? (Note that Supabase Logs appears to rely on Google BigQuery so you'll be running on Google either way.)
Re: Supabase Logs: open-source logging server
#10hey hn, supabase ceo here
this one is a long-time coming and it's a continuation of our acquisition of Logflare[0]. Since the acquisition we've be open-source-ing the server, which you can find here: https://github.com/Logflare/logflare
Logflare handles about 1.5 billion log-events everyday on supabase. It's built with Elixir and has no problems with that workload.
This is really just the start of the Logflare updates. All logs are currently ingested into BigQuery, and we are adding support for Clickhouse and other backends (focusing primarily on open source backend). Over time Logflare will function very much like an open source Sentry alternative, where you can ingest data from various sources.
The team will be around if you have any questions about the technical implementation
[0] acquision: https://supabase.com/blog/supabase-acquires-logflare