Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Supabase Logs: open-source logging server

supabase.com

Post reply on HN
Print view View original

31–40 of 68 posts

Re: Supabase Logs: open-source logging server

#31

Post by kiwicopple »

doublesocket wrote: post #28
kiwicopple wrote: post #22

Earlier quoted context omitted.

The tools written in elixir are this Logging server, our Realtime server, and a Postgres Pooler which we launched on HN this weekend [0] The decision is centered more around the BEAM [1], the virtual machine which powers Elixir/Erlang. It was designed for high-availability systems and it's very efficient at managing large numbers of processes and threads. In the context of these tools, it's useful for handling open c…

What's the contingency plan if Steve gets hit by a bus? (very much hoping he doesn't)

We have steve locked in a safe-room.

But we also sponsor other members of the community to work on PostgREST[0]. We don't promote the paid sponsorship much. Instead, we find people who are already doing the work and then offer them sponsorship to continue doing it.

[0] sponsorship: https://opencollective.com/supabase-postgrest

Re: Supabase Logs: open-source logging server

#32

Post by kiwicopple »

gregwebs wrote: post #30
doublesocket wrote: post #28

Earlier quoted context omitted.

What's the contingency plan if Steve gets hit by a bus? (very much hoping he doesn't)

There’s more than 1 developer out there that knows Haskell and SQL. You just run into issues if you want those developers to sit in your office next to you.

> sit in your office next to you

you comment is probably metaphorical, but it's worth pointing out how we have de-risked this: supabase is fully distributed. We look for the right person no matter where they live. We're currently 65 people in more than 20 countries

Re: Supabase Logs: open-source logging server

#33

Post by mosselman »

kiwicopple wrote: post #12
sidcool wrote: post #11

Is self hosting possible and feasible?

Yes, the docs are here: https://supabase.com/docs/reference/self-hosting-analytics/i... One caveat: the only backend-store supported right now BigQuery. We will be releasing support for Clickhouse and Postgres in the coming months, we just couldn't fit it into this Launch Week

Is there somewhere I can subscribe for an update on this specifically?

Re: Supabase Logs: open-source logging server

#34

Post by kiwicopple »

mosselman wrote: post #33
kiwicopple wrote: post #12

Earlier quoted context omitted.

Yes, the docs are here: https://supabase.com/docs/reference/self-hosting-analytics/i... One caveat: the only backend-store supported right now BigQuery. We will be releasing support for Clickhouse and Postgres in the coming months, we just couldn't fit it into this Launch Week

Is there somewhere I can subscribe for an update on this specifically?

It's easiest to "watch releases" on either the Logflare repo[0] or the Supabase repo[1].

If you're signed up to Supabase we send 1 email per month with all of our releases.

[0] Logflare: https://github.com/Logflare/logflare

[1] Supabase: https://github.com/supabase/supabase

Re: Supabase Logs: open-source logging server

#35

Post by nextaccountic »

lyziinc wrote: post #23
swyx wrote: post #14

Earlier quoted context omitted.

thank you thank you. the longer i spend as a dev the more i have infinite appreciation for better logging. this is asking a lot but could we get a breakdown as to what you think are relevant decision factors for choice of logging server? (eg vs logstash/kibana, splunk and i guess people DIY with clickhouse but we can effectively ignore that)

Hi I’m one of the logflare devs and I work on observability at Supabase. Great question. To directly address some of the tools you mentioned: - Logstash is the transport and transformation portion (along with Filbert) in the elastic stack, and it performs the same functions as vector. It is out of scope for Logflare, which focuses on acting as a centralised server to point all your logging pipelines to. - Kibana is t…

> - Reliability. The last thing that you would want is for your application to take high load and go down, but you’re unable to debug it because the high traffic led to high log load and subsequently took down your o11y server. Logflare is built on the BEAM and can handle high loads without breaking a sweat. We’ve handled over 10x average load for ingestion spikes and Logflare just chugs along.

Can logflare scale out into multiple containers / vms / machines? Is Supabase currently deploying something like autoscaling with kubernetes or something?

Re: Supabase Logs: open-source logging server

#36

Post by ignoramous »

lyziinc wrote: post #26

I'm one of the Logflare devs @ supabase that made this a reality, feel free to AMA about the technical details.

Hi,

You mentioned above that BigQuery reduces cost. I am surprised by that assertion, tbh. Can you point out ways in which Logflare uses it that makes it so (for ex, is it tiered-storage with a BQ front-end)?

How does Logflare's approach contrast with other entrants like axiom.co/99 who are leveraging blob stores (Cloudflare R2) for storage and serverless for querying for lower costs?

Multiple pluggable storage/query backends (like Clickhouse) is all good, but is there a default that Logflare is going to recommend / settle on?

Are there plans to go beyond just APM with Logflare (like metrics and traces, for instance)?

I guess, at some level, this product signals a move away from Postgres-for-everything stance?

Thanks.

Re: Supabase Logs: open-source logging server

#38

Post by lyziinc »

nextaccountic wrote: post #35
lyziinc wrote: post #23

Earlier quoted context omitted.

Hi I’m one of the logflare devs and I work on observability at Supabase. Great question. To directly address some of the tools you mentioned: - Logstash is the transport and transformation portion (along with Filbert) in the elastic stack, and it performs the same functions as vector. It is out of scope for Logflare, which focuses on acting as a centralised server to point all your logging pipelines to. - Kibana is t…

> - Reliability. The last thing that you would want is for your application to take high load and go down, but you’re unable to debug it because the high traffic led to high log load and subsequently took down your o11y server. Logflare is built on the BEAM and can handle high loads without breaking a sweat. We’ve handled over 10x average load for ingestion spikes and Logflare just chugs along. Can logflare scale out…

For the Logflare infra, we manage a cluster of around 6 VMs. Not a very complex setup, and no need for k8s as we have a monolith architecture. We also don't scale horizontally as much as the cross-node chatter increases with each additional node.

The supabase infra is much more complex, however.

Re: Supabase Logs: open-source logging server

#39

Post by chasers »

harpratap wrote: post #8
martin_ wrote: post #4

Earlier quoted context omitted.

It's pretty awesome & very cost effective! When I was at Twilio we streamed logstash/fluentd (forget which) -> kinesis -> BigQuery and it worked great - certainly better than the days we were trying to manage ES ourselves

Were people using it as an actual ES replacement for logs? fluentd -> GCP log sink -> BQ can get awful if you need to work with it on daily basis

Curious how was it awful for you? Really have not regretted any part of using BigQuery.
Post reply on HN