21–30 of 68 posts
It would be awesome if this could use Quickwit as a backend which is a new promising alternative to Elasticsearch, I’ve been using it internally and it’s much more lightweight and easier to run.
We would need to support SQL though.
Just putting the link here: http://github.com/quickwit-oss/quickwit
Would love to have your feedback on ease of use/perf/whatever here/on twitter/discord (https://discord.quickwit.io/).
hey hn, supabase ceo here this one is a long-time coming and it's a continuation of our acquisition of Logflare[0]. Since the acquisition we've be open-source-ing the server, which you can find here: https://github.com/Logflare/logflare Logflare handles about 1.5 billion log-events everyday on supabase. It's built with Elixir and has no problems with that workload. This is really just the start of the Logflare update…
Seems that a lot of Supabase's project ls written in Elixir, which is not a very popular choice. What is the reason behind choosing Elixir?
The decision is centered more around the BEAM [1], the virtual machine which powers Elixir/Erlang. It was designed for high-availability systems and it's very efficient at managing large numbers of processes and threads.
In the context of these tools, it's useful for handling open connections and spiky workloads.
edit: Also worth mentioning - we don't care too much about the language. We aim to use the right tool for the job, or preferably an existing open source tool. A good example of this is PostgREST which is written in Haskell. The only person on our team who knows Haskell is Steve (the maintainer). We don't mind because it's the right tool for the job, it's open source, and it has an amazing community.
[0] Pooler: https://news.ycombinator.com/item?id=35501718
[1] BEAM: https://blog.lelonek.me/elixir-on-erlang-vm-demystified-3205...
hey hn, supabase ceo here this one is a long-time coming and it's a continuation of our acquisition of Logflare[0]. Since the acquisition we've be open-source-ing the server, which you can find here: https://github.com/Logflare/logflare Logflare handles about 1.5 billion log-events everyday on supabase. It's built with Elixir and has no problems with that workload. This is really just the start of the Logflare update…
thank you thank you. the longer i spend as a dev the more i have infinite appreciation for better logging. this is asking a lot but could we get a breakdown as to what you think are relevant decision factors for choice of logging server? (eg vs logstash/kibana, splunk and i guess people DIY with clickhouse but we can effectively ignore that)
Great question. To directly address some of the tools you mentioned:
- Logstash is the transport and transformation portion (along with Filbert) in the elastic stack, and it performs the same functions as vector. It is out of scope for Logflare, which focuses on acting as a centralised server to point all your logging pipelines to.
- Kibana is the visualisation layer of the elastic stack. For Supabase, this functionality is taken over by the Supabase Studio, and the reporting capabilities will eventually converge to compete to match APM services like sentry etc.
- Splunk’s core is not open source, and is very much geared to large contract enterprise customers. Their main product is also much more geared towards visualisation as opposed to bare log analysis.
When it comes to a logging server/service, you’d consider the following factors:
- Cost. Logging is quite expensive, and the way that Logflare leverages BQ (and in the future, other OLAP engines) cuts down storage costs greatly
- Reliability. The last thing that you would want is for your application to take high load and go down, but you’re unable to debug it because the high traffic led to high log load and subsequently took down your o11y server. Logflare is built on the BEAM and can handle high loads without breaking a sweat. We’ve handled over 10x average load for ingestion spikes and Logflare just chugs along.
- Querying capabilities. Storing logs isn’t enough, you need to effectively debug and aggregate your logs for insights. This incurs both querying costs and additional complexity in the sense that your storage mechanism must be able to handle such complex queries without breaking the bank. Logflare performs optimisations for these, performing table partitioning and caching to make sure costs are kept low. This allows Supabase to expose all logging data to users and perform joins and filters within the Logs Explorer to their hearts’ content.
Exists a pino transport?
Exists a pino transport?
Earlier quoted context omitted.
thank you thank you. the longer i spend as a dev the more i have infinite appreciation for better logging. this is asking a lot but could we get a breakdown as to what you think are relevant decision factors for choice of logging server? (eg vs logstash/kibana, splunk and i guess people DIY with clickhouse but we can effectively ignore that)
Hi I’m one of the logflare devs and I work on observability at Supabase. Great question. To directly address some of the tools you mentioned: - Logstash is the transport and transformation portion (along with Filbert) in the elastic stack, and it performs the same functions as vector. It is out of scope for Logflare, which focuses on acting as a centralised server to point all your logging pipelines to. - Kibana is t…
Earlier quoted context omitted.
Seems that a lot of Supabase's project ls written in Elixir, which is not a very popular choice. What is the reason behind choosing Elixir?
The tools written in elixir are this Logging server, our Realtime server, and a Postgres Pooler which we launched on HN this weekend [0] The decision is centered more around the BEAM [1], the virtual machine which powers Elixir/Erlang. It was designed for high-availability systems and it's very efficient at managing large numbers of processes and threads. In the context of these tools, it's useful for handling open c…
(very much hoping he doesn't)
It would be awesome if this could use Quickwit as a backend which is a new promising alternative to Elasticsearch, I’ve been using it internally and it’s much more lightweight and easier to run.
Cool :) We would need to support SQL though. Just putting the link here: http://github.com/quickwit-oss/quickwit Would love to have your feedback on ease of use/perf/whatever here/on twitter/discord ( https://discord.quickwit.io/ ).
Earlier quoted context omitted.
The tools written in elixir are this Logging server, our Realtime server, and a Postgres Pooler which we launched on HN this weekend [0] The decision is centered more around the BEAM [1], the virtual machine which powers Elixir/Erlang. It was designed for high-availability systems and it's very efficient at managing large numbers of processes and threads. In the context of these tools, it's useful for handling open c…
What's the contingency plan if Steve gets hit by a bus? (very much hoping he doesn't)