Live data from Hacker News

Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

github.com

11–20 of 50 posts

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#11
Great to see more data engineering in the direction of SecOps.

We do something similar with VAST at https://vast.io. We’re still early, but especially live and retro detection of threat intel is what we are focusing on. Essentially operationalizing security content for detection and response, plus acquiring and extracting context of alerts and telemetry.

We have an experimental serverless deployment with Lambda and Fargate, but the majority of our users still collocate VAST near network sensors like Zeek and Suricata.

We’re running everything on top of Apache Arrow, storage of telemetry is now also Parquet. The idea is to do everything with open standards to minimize vendor lock-in.

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#12
post #9

I'm going to regret asking this, but what the hell is a "security lake"? A collection of audit logs?

Its a data lake in which you store security logs. That includes Cloud/SaaS audit logs, network security logs (Zeek, Suricata, Snort), VPN/Firewall logs, and more.

But logs are structured and filtered by their relevance to security. In what way is that a "lake"?

Is "data lake" just the new plural of "dataset"?

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#14
post #12

Earlier quoted context omitted.

Its a data lake in which you store security logs. That includes Cloud/SaaS audit logs, network security logs (Zeek, Suricata, Snort), VPN/Firewall logs, and more.

But logs are structured and filtered by their relevance to security. In what way is that a "lake"? Is "data lake" just the new plural of "dataset"?

People tend to call them "lakes" because, I think, they are "unfiltered" and contain raw data objects and blobs, originally from the source system, unmodified. In a normal data warehouse system you ETL things and the final "load" step stores them in the warehouse, and then you use that as your source of truth. Your data warehouse might be Redshift on Amazon. In the "Data lake" case you instead load everything into something like S3, and then everything uses S3 as the source of truth -- including your query engine, which in this case might be Athena (also on Amazon). I won't go into Redshift vs Athena but if you're familiar with them, this should make sense.

I'd say like 95% of the case I've seen people talking about these things, they basically mean: shove everything into S3 and use that as the canonical source of truth for your data systems, rather than some OLAP system; instead you build the OLAP system off S3.

More simply, I think of it like a term to describe a particular mindset concerning your ETL: always work on the source data. And source data is often messy and unstructured. It's a lot of potentially unstructured and underspecified bullshit. So S3 is pretty good storage for something like that versus datastores with performance/usability cliffs around things like cardinality, fields that come and go, etc...

One advantage of this design I can see is that S3 is very "commodified" by this point (lots of alternative offerings) and can be integrated with in nearly every pipeline, and your tools can be replaced more easily, perhaps. S3 is more predictable and "low level" in that regard than something like a database, with many more performance/availability considerations. Like in the example I gave, you could feasibly replace Athena with Trino for instance, without disturbing too much beyond that system. You just need to re-ingest data from S3 for a single system. While if you loaded and ETL'd all your data into a database like Redshift, you might be stuck with that forever even if you later decide it was a mistake. This isn't a hard truth (you might still be stuck with Athena) but just an example of when this might be more flexible.

As usual this isn't an absolute and there are things in-between. But this is generally the gist of it, I think. The "lake" naming is kind of weird but makes some amount of sense I think. It describes a mindset rather than any particular tech.

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#16

is this opensource snowflake-for-security-logs ?

It is similar, although Snowflake is more of a query engine whereas we are a cloud security data platform built on an open data model (Apache Iceberg). We help you ingest and normalize data from common security sources into a data lake and offer a serverless platform to deploy & run Python detections-as-code on these events in realtime.

Although the only supported query service is currently Athena, we plan to integrate with popular vendors like Snowflake and Dremio. Thanks to the growing industry support for Iceberg, we believe vendor lock-in should be a story of the past for security data.

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#17

Looks neat, but in what way is this serverless? It's a pretty complex diagram: https://github.com/matanolabs/matano/blob/main/website/src/a...

There's no servers to maintain in the entire architecture, we heavily use Lambda and even use MSK serverless for Kafka.

This sounds truely nightmarish and costly. Even moderate volumes of data are going to add up very quickly cost wise.

I see the term zero-ops. But maintaining and debugging this pipeline is going to require some ops, even if you are not managing VMs.

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#18

It: "powered by Rust + Apache Iceberg" Me: Oh cool, can I run it in my k8s cluster? It: "designed specifically for AWS" Me: disappointed and annoyed by title Looking at that service diagram, "Powered by AWS services" seems more accurate.

Related, "powered by Rust" I guess is some kind of shorthand way of saying "powered by Kotlin, TypeScript, Python, Rust, but mostly TypeScript"

In fairness to the project, I think that clickbait was just the submission title, I don't see that language in the GH page at all

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#19
Serverless sounds cool for this at first, but what are the ingest/compute costs going to look like at a modest 20Tb/day? How about 100, or 1Pb?

Honestly think at that point you’d be better off and cheaper to go with a commercial security data lake..

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#20
post #17

Earlier quoted context omitted.

There's no servers to maintain in the entire architecture, we heavily use Lambda and even use MSK serverless for Kafka.

This sounds truely nightmarish and costly. Even moderate volumes of data are going to add up very quickly cost wise. I see the term zero-ops. But maintaining and debugging this pipeline is going to require some ops, even if you are not managing VMs.

Using and maintaining Matano is a fraction of the cost compared to popular non-serverless alternatives like ELK or Spunk. Matano is specifically designed for petabyte-scale security analytics use-cases that don't fit in a traditional SIEM.

The serverless data ingestion pipeline means you don't need to over-provision for ingestion (Logstash and Splunk Forwarders are notorious for related costs / ops in high scale use-cases) in the write path. For reads, since Matano queries Iceberg tables backed by highly-compressed parquet files on object storage you won't pay anything close to what you would for a database or search engine based SIEM.

Post reply on HN