31–40 of 50 posts
It: "powered by Rust + Apache Iceberg" Me: Oh cool, can I run it in my k8s cluster? It: "designed specifically for AWS" Me: disappointed and annoyed by title Looking at that service diagram, "Powered by AWS services" seems more accurate.
Earlier quoted context omitted.
Related, "powered by Rust" I guess is some kind of shorthand way of saying "powered by Kotlin, TypeScript, Python, Rust, but mostly TypeScript" In fairness to the project, I think that clickbait was just the submission title, I don't see that language in the GH page at all
Over enough time on HN you start to notice patterns of things that are easy low-hanging upvotes. Of course we can't do anything with upvotes, but we can do things with clicks and usage metrics, and I'd imagine that Hn is a considered market when it comes to introducing something to the greater tech community. A lot of us tend to have some degree of decision making capability and influence in our orgs.
downvote and flag buttons. you only need a handful alt accounts get any comment you dislike [dead] or [flagged], and it only takes one successful submission of a yuppie clickbait article per alt
Serverless sounds cool for this at first, but what are the ingest/compute costs going to look like at a modest 20Tb/day? How about 100, or 1Pb? Honestly think at that point you’d be better off and cheaper to go with a commercial security data lake..
Matano is designed specifically for petabyte-scale security log analytics use cases, so performance and costs are a top priority. Our data pipeline borrows from Vector's Rust based data transformation language [0] for maximal performance, with each parallel function invocation capable of processing upwards of 20MiB/s [1] thanks to auto-vectorizaton. Roughly this comes out to $1/(TB/day) for ingest compute costs which…
Given a prior exploration of a security visibility stack on AWS at scale, this is likely a colossal sized operational expense.
Earlier quoted context omitted.
But logs are structured and filtered by their relevance to security. In what way is that a "lake"? Is "data lake" just the new plural of "dataset"?
data lake isn't a new term (relatively). I remember first hearing it when I worked at Google like 5 years ago, and the context was always referring to some enormous raw data store. Probably the term "lake" is supposed to evoke a sense of largeness and shapelessness. If you wanted to train a model, you would tap into a data lake which had up to petabytes.
Earlier quoted context omitted.
People tend to call them "lakes" because, I think, they are "unfiltered" and contain raw data objects and blobs, originally from the source system, unmodified. In a normal data warehouse system you ETL things and the final "load" step stores them in the warehouse, and then you use that as your source of truth. Your data warehouse might be Redshift on Amazon. In the "Data lake" case you instead load everything into so…
None of this really drives home why a new term was necessary. I'm still seeing "dataset".
Now just wait till you hear someone reference "data lake-house" ...
is this opensource snowflake-for-security-logs ?
It is similar, although Snowflake is more of a query engine whereas we are a cloud security data platform built on an open data model (Apache Iceberg). We help you ingest and normalize data from common security sources into a data lake and offer a serverless platform to deploy & run Python detections-as-code on these events in realtime. Although the only supported query service is currently Athena, we plan to integra…
Our IDS solution outputs zeek/suricata info to s3 as dns.1234.log.gz, http.1234.log.gz, etc.
Can these files be handled automatically?
Earlier quoted context omitted.
Using and maintaining Matano is a fraction of the cost compared to popular non-serverless alternatives like ELK or Spunk. Matano is specifically designed for petabyte-scale security analytics use-cases that don't fit in a traditional SIEM. The serverless data ingestion pipeline means you don't need to over-provision for ingestion (Logstash and Splunk Forwarders are notorious for related costs / ops in high scale use-…
> For reads, since Matano queries Iceberg tables backed by highly-compressed parquet files on object storage you won't pay anything close to what you would for a database or search engine based SIEM Where do you show an example of querying anything? There's an empty "detector" in the examples directory, which I guess gets called once per row of this 20MiB/s alleged elsewhere? Anyway, I find comparing this to Splunk t…
Earlier quoted context omitted.
People tend to call them "lakes" because, I think, they are "unfiltered" and contain raw data objects and blobs, originally from the source system, unmodified. In a normal data warehouse system you ETL things and the final "load" step stores them in the warehouse, and then you use that as your source of truth. Your data warehouse might be Redshift on Amazon. In the "Data lake" case you instead load everything into so…
None of this really drives home why a new term was necessary. I'm still seeing "dataset".
It: "powered by Rust + Apache Iceberg" Me: Oh cool, can I run it in my k8s cluster? It: "designed specifically for AWS" Me: disappointed and annoyed by title Looking at that service diagram, "Powered by AWS services" seems more accurate.
We need a solid open source and portable serverless platform. Using lambda gets you locked into lambda.