Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

github.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#21

Post by shaeqahmed »

happyopossum wrote: post #19

Serverless sounds cool for this at first, but what are the ingest/compute costs going to look like at a modest 20Tb/day? How about 100, or 1Pb? Honestly think at that point you’d be better off and cheaper to go with a commercial security data lake..

Matano is designed specifically for petabyte-scale security log analytics use cases, so performance and costs are a top priority. Our data pipeline borrows from Vector's Rust based data transformation language [0] for maximal performance, with each parallel function invocation capable of processing upwards of 20MiB/s [1] thanks to auto-vectorizaton.

Roughly this comes out to $1/(TB/day) for ingest compute costs which is much cheaper than a commercial solution. We are also working on moving over our Lambda's to ARM for even better cost-effficency.

[0] https://vector.dev/docs/reference/vrl/ [1] https://vector.dev/docs/setup/going-to-prod/sizing/#sizing

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#22

Post by wizwit999 »

electromech wrote: post #7

It: "powered by Rust + Apache Iceberg" Me: Oh cool, can I run it in my k8s cluster? It: "designed specifically for AWS" Me: disappointed and annoyed by title Looking at that service diagram, "Powered by AWS services" seems more accurate.

> serverless

> run in my k8s cluster

Those two don't really go together. ;)

Kidding aside, yeah we definitely leverage all the power of AWS services to give a completely serverless experience.

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#23

Post by warent »

remram wrote: post #12
shaeqahmed wrote: post #10

Earlier quoted context omitted.

Its a data lake in which you store security logs. That includes Cloud/SaaS audit logs, network security logs (Zeek, Suricata, Snort), VPN/Firewall logs, and more.

But logs are structured and filtered by their relevance to security. In what way is that a "lake"? Is "data lake" just the new plural of "dataset"?

data lake isn't a new term (relatively). I remember first hearing it when I worked at Google like 5 years ago, and the context was always referring to some enormous raw data store. Probably the term "lake" is supposed to evoke a sense of largeness and shapelessness. If you wanted to train a model, you would tap into a data lake which had up to petabytes.

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#24

Post by aliqot »

warent wrote: post #23
remram wrote: post #12

Earlier quoted context omitted.

But logs are structured and filtered by their relevance to security. In what way is that a "lake"? Is "data lake" just the new plural of "dataset"?

data lake isn't a new term (relatively). I remember first hearing it when I worked at Google like 5 years ago, and the context was always referring to some enormous raw data store. Probably the term "lake" is supposed to evoke a sense of largeness and shapelessness. If you wanted to train a model, you would tap into a data lake which had up to petabytes.

the FAANG job promotion "game" leads to a lot of new terms being coined because everyone wants to be the guy who invented X. My generation isn't beyond reproach either, our thing was acronyms and clever initialisms.

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#25

Post by aliqot »

aseipp wrote: post #14
remram wrote: post #12

Earlier quoted context omitted.

But logs are structured and filtered by their relevance to security. In what way is that a "lake"? Is "data lake" just the new plural of "dataset"?

People tend to call them "lakes" because, I think, they are "unfiltered" and contain raw data objects and blobs, originally from the source system, unmodified. In a normal data warehouse system you ETL things and the final "load" step stores them in the warehouse, and then you use that as your source of truth. Your data warehouse might be Redshift on Amazon. In the "Data lake" case you instead load everything into so…

None of this really drives home why a new term was necessary. I'm still seeing "dataset".

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#26

Post by aliqot »

mdaniel wrote: post #18
electromech wrote: post #7

It: "powered by Rust + Apache Iceberg" Me: Oh cool, can I run it in my k8s cluster? It: "designed specifically for AWS" Me: disappointed and annoyed by title Looking at that service diagram, "Powered by AWS services" seems more accurate.

Related, "powered by Rust" I guess is some kind of shorthand way of saying "powered by Kotlin, TypeScript, Python, Rust, but mostly TypeScript" In fairness to the project, I think that clickbait was just the submission title, I don't see that language in the GH page at all

Over enough time on HN you start to notice patterns of things that are easy low-hanging upvotes. Of course we can't do anything with upvotes, but we can do things with clicks and usage metrics, and I'd imagine that Hn is a considered market when it comes to introducing something to the greater tech community. A lot of us tend to have some degree of decision making capability and influence in our orgs.

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#27

Post by mdaniel »

shaeqahmed wrote: post #20
georgyo wrote: post #17

Earlier quoted context omitted.

This sounds truely nightmarish and costly. Even moderate volumes of data are going to add up very quickly cost wise. I see the term zero-ops. But maintaining and debugging this pipeline is going to require some ops, even if you are not managing VMs.

Using and maintaining Matano is a fraction of the cost compared to popular non-serverless alternatives like ELK or Spunk. Matano is specifically designed for petabyte-scale security analytics use-cases that don't fit in a traditional SIEM. The serverless data ingestion pipeline means you don't need to over-provision for ingestion (Logstash and Splunk Forwarders are notorious for related costs / ops in high scale use-…

> For reads, since Matano queries Iceberg tables backed by highly-compressed parquet files on object storage you won't pay anything close to what you would for a database or search engine based SIEM

Where do you show an example of querying anything? There's an empty "detector" in the examples directory, which I guess gets called once per row of this 20MiB/s alleged elsewhere?

Anyway, I find comparing this to Splunk to be a bit premature

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#29

Post by geodel »

remram wrote: post #9

I'm going to regret asking this, but what the hell is a "security lake"? A collection of audit logs?

From GitHub it looks like implementation of Random Buzzword Lake APIs in Javascript.

I am so excited, couldn't wait to see more!

Re: Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

#30

Post by glitchcrab »

aliqot wrote: post #25
aseipp wrote: post #14

Earlier quoted context omitted.

People tend to call them "lakes" because, I think, they are "unfiltered" and contain raw data objects and blobs, originally from the source system, unmodified. In a normal data warehouse system you ETL things and the final "load" step stores them in the warehouse, and then you use that as your source of truth. Your data warehouse might be Redshift on Amazon. In the "Data lake" case you instead load everything into so…

None of this really drives home why a new term was necessary. I'm still seeing "dataset".

That's because it's just marketing buzzword bingo
Post reply on HN