Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Matano (YC W23) – Open-Source Security Lake Platform (SIEM) for AWS

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Launch HN: Matano (YC W23) – Open-Source Security Lake Platform (SIEM) for AWS

#31

Post by shaeqahmed »

mdaniel wrote: post #29

I loaded the GitHub link, bracing myself for yet another AGPL license, but no, it's Apache 2! So I wanted to say thank you for that and I hope to take a deeper look when I'm back at my desk because trying to keep Splunk alive and happy is a monster pain point. There are so many data sources we'd love to throw at it but we don't have the emotional energy to put up with Splunk crying about it

Thank you! We definitely believe in open source and don't need AGPL. Sending you love as you deal with that Splunk instance.

P.S. feel free to open some issues for any log sources you'd like to see supported in Matano

Re: Launch HN: Matano (YC W23) – Open-Source Security Lake Platform (SIEM) for AWS

#32

Post by _8j50 »

What distinguishes Matano'd existing or planned products from Google Chronicle? Would you have any limits on data ingestion or retention?

Also, python detections sounds horrible! I love python but it sounds like you haven't considered the challenges of detection engineering. This one of my main "expertise" if you will. You should think more in the lines of flexible sql than python. People who write detection rules to the most part don't know python and even if they do it would be a nightmare to use for many reasons.

I hope someone from your team reads this comment: DO NOT try to invent your own query language but if you do, DON'T start from scratch. Your product could be the best people who like the fabulous splunk need to also like it. And for a security data lake, you must support Sigma rule conversion into your query/rule format. Python is a general purpose language, there are very good reasons why no one else from Splunk,elastic, graylog, Google,Microsoft use Python. Don't learn this hard lesson with your own money. Querying it needs to be very simple and most importantly you need to support regex with capture groups and the equivalent of "|stats" command from splunk if you want to quickly capture market share. I have used and evaluated many of these tools and have written a lot of detection content.

Your users are not coders, DB admins or exploit developers. They are really smart people whose focus is understand threat actors and responding to incidents -- not coding or anything sophisticated. FAANG background founders/devs have a hard time grasping this reality.

Re: Launch HN: Matano (YC W23) – Open-Source Security Lake Platform (SIEM) for AWS

#33

Post by shaeqahmed »

_8j50 wrote: post #32

What distinguishes Matano'd existing or planned products from Google Chronicle? Would you have any limits on data ingestion or retention? Also, python detections sounds horrible! I love python but it sounds like you haven't considered the challenges of detection engineering. This one of my main "expertise" if you will. You should think more in the lines of flexible sql than python. People who write detection rules to…

Some big differences:

- Matano has realtime Python + SQL detections as code with advanced correlation support. Chronicle uses inflexible YARA-like detection rules iirc

- Matano supports Sigma detections by automatically transpiling them to the Python detection format

- Matano has an OSS Vendor Agnostic Security Data Lake and can work with multiple clouds / let's you bring your own query engine (Snowflake, Spark, Athena, BigQuery Omni). Chronicle is a proprietary SIEM that uses BigQuery under the hood and cannot be used with other tooling.

There are no limits on data retention or ingestion with Matano, it's your S3 bucket and the compute scales horizontally.

Re: Launch HN: Matano (YC W23) – Open-Source Security Lake Platform (SIEM) for AWS

#35

Post by shaeqahmed »

brunes wrote: post #34

How do you position this against AWS's own Security Lake announced at re:Invent in November ( https://aws.amazon.com/security-lake/ ) ? Your architecture diagram looks like a carbon copy of theirs.

We launched before Amazon Security Lake :)

Amazon Security Lake's main value prop is that it is a single place where AWS / partner security logs can be stored and sent to downstream vendors. As such, Amazon only writes OCSF normalized logs to the parquet-based data lake for it's own data in a fully managed way (VPC flow logs, Cloudtrail, etc.) and leaves it to the customers to handle the rest.

For partner sources, the integration approach has been to tell customers to set up infrastructure themselves to accomplish OCSF normalization, parquet conversion, etc. For example, here is okta's guide using Firehose and Lambda, https://www.okta.com/blog/2022/11/an-automated-approach-to-c...

The Amazon Security Lake offering is built on top of Lake Formation, which itself is an abstraction around services such as Glue, Athena, and S3. Security Lake is built using the legacy Hive style approach and does not use Athena Iceberg. There is a per-data cost associated with the service, in addition to the costs incurred by other services for your data lake. Looks like the primary use case of the service is being able to store first-party AWS logs across all your accounts in a data lake and being able to route them to analytical partners (SIEM) without much effort. It does not seem very useful for an organization that is looking to build its own security data lake with more advanced features, as you will still have to do all the work yourself.

Matano, has a broader goal to help orgs in every step of transforming, normalizing, enriching and storing all of their security logs into a structured data lake, as well as giving users a platform to build detection-as-code using Python & SQL for correlation on top of it (SIEM augmentation/alternative). All processing and data lake management (conversion to parquet, data compaction, table management) is fully automated by Matano, and users do not need to write any custom code to onboard data sources.

Matano can ingest data from Cloud, Endpoint, SaaS, and practically any custom source using the in-built Log transformation pipeline (think serverless Logstash). We are built around the Elastic Common Schema, and use Apache Iceberg (ACID support, recommended for Athena V2+). Matano's data lake is also vendor neutral and can be queried by any Iceberg-compatible engine without having to copy any data around (Snowflake, Spark, etc.).

Re: Launch HN: Matano (YC W23) – Open-Source Security Lake Platform (SIEM) for AWS

#36

Post by 0x4e53 »

slt2021 wrote: post #11
0x4e53 wrote: post #7

Earlier quoted context omitted.

At least from my time at SpaceX - this is untrue. SIEM costs were rapidly ballooning, and we were being charged by RAM. RAM?? Of all things!! After our SIEM costs for ELK ramped up to where Splunk was - we just bought Splunk instead. I imagine there are many security teams out there that would entertain a cheaper alternative that isn't priced by RAM.

the reason for that is near real-time detection of threats requires aggregation of terabytes of data according to rules (continuous GROUP BY on thousands columns on a sliding window) - and these aggregates by design have to be stored in RAM. Otherwise these detections stop being near-realtime and become offline detection instead, just like any other sql server.

To be clear - we were hosting on-premise, and being charged for our own RAM. Servers we had to buy, and then pay for the privilege of using with ELK.

Re: Launch HN: Matano (YC W23) – Open-Source Security Lake Platform (SIEM) for AWS

#37

Post by slt2021 »

Question to Matano authors - won't your solution simply enrich AWS by blowing up my cloud bill ?

Did you estimate how many times lambda will get invoked and what will be AWS bill for 1 million events ingested? I am curious to learn the price to pay for serverless SIEM

Re: Launch HN: Matano (YC W23) – Open-Source Security Lake Platform (SIEM) for AWS

#38

Post by shaeqahmed »

slt2021 wrote: post #37

Question to Matano authors - won't your solution simply enrich AWS by blowing up my cloud bill ? Did you estimate how many times lambda will get invoked and what will be AWS bill for 1 million events ingested? I am curious to learn the price to pay for serverless SIEM

The code is written in high performance multi-threaded Rust and uses the [1] Arrow compute framework. We also batch events and target about 32MB of event data per lambda invocations. As a result it can process tens of thousands of events per second per thread, depending on the number of transformations.

That said, we are working on performance estimates and a benchmark on some real world data for Matano to help users like you better understand the cost factors. Stay tuned.

[1] https://github.com/jorgecarleitao/arrow2

Re: Launch HN: Matano (YC W23) – Open-Source Security Lake Platform (SIEM) for AWS

#39

Post by parkerhiggins »

shaeqahmed wrote: post #22
wdb wrote: post #21

Anyone aware of a similar solution for Google Cloud / GCP?

We are working on a solution for GCP and Azure :) GCP recently announced Iceberg support with BigLake and support for federation across multi-cloud lakes so it would be perfect use cases. If you are interested in using Matano for GCP, feel free to reach out and join our Discord community! We are FOSS so would love to collaborate on a solution.

Definitely looking forward to GCP.

Re: Launch HN: Matano (YC W23) – Open-Source Security Lake Platform (SIEM) for AWS

#40

Post by feanaro »

mox1 wrote: post #30
sullivanmatt wrote: post #24

Earlier quoted context omitted.

For sure. Pull a dbt and get everybody hooked on your tool, then slap a SaaS platform ecosystem to the farthest right and watch the revenue flow.

Splunk is HEAVILY pushing their SaaS offering at the moment. They are the most obnoxious vendor we currently deal with. We are fine on prem, pay big $$ license fees, but not enough. They want that sweet SaaS revenue. I would be wary of pushing this, being a non-SaaS platform could be an advantage here.

Indeed, no more SaaS. I've had enough of this cloud nonsense already.
Post reply on HN