41–50 of 50 posts
I'm going to regret asking this, but what the hell is a "security lake"? A collection of audit logs?
From GitHub it looks like implementation of Random Buzzword Lake APIs in Javascript. I am so excited, couldn't wait to see more!
It: "powered by Rust + Apache Iceberg" Me: Oh cool, can I run it in my k8s cluster? It: "designed specifically for AWS" Me: disappointed and annoyed by title Looking at that service diagram, "Powered by AWS services" seems more accurate.
> serverless > run in my k8s cluster Those two don't really go together. ;) Kidding aside, yeah we definitely leverage all the power of AWS services to give a completely serverless experience.
It: "powered by Rust + Apache Iceberg" Me: Oh cool, can I run it in my k8s cluster? It: "designed specifically for AWS" Me: disappointed and annoyed by title Looking at that service diagram, "Powered by AWS services" seems more accurate.
We need a solid open source and portable serverless platform. Using lambda gets you locked into lambda.
Earlier quoted context omitted.
> serverless > run in my k8s cluster Those two don't really go together. ;) Kidding aside, yeah we definitely leverage all the power of AWS services to give a completely serverless experience.
There are plenty of FOSS serverless options https://knative.dev/docs/ https://krustlet.dev https://www.openfaas.com/
So currently, one would have to maintain the OpenFaaS control plane themselves, which takes away most of the benefits for an end user using serverless (would still have ops and no multi-tenant cost benefit).
Earlier quoted context omitted.
None of this really drives home why a new term was necessary. I'm still seeing "dataset".
Think of it as "raw dataset" where all the data lands first and only copies of it are modified.
Here these are logs that were already filtered by their relevance to security and exported as structured data. Considering those "raw unstructured data" because you haven't personally done ETL on it seems wrong.
Earlier quoted context omitted.
None of this really drives home why a new term was necessary. I'm still seeing "dataset".
Because dataset doesn't tell you if the data was ETL'd or ELT'd; data warehouse and data lake do tell you. Now just wait till you hear someone reference "data lake-house" ...
> A data warehouse is a repository for structured, filtered data that has already been processed for a specific purpose.
Structured logs that have been filtered by their relevance to security really seem to fit the definition. If we must use newspeak, "log warehouse" then?
is this opensource snowflake-for-security-logs ?
It is similar, although Snowflake is more of a query engine whereas we are a cloud security data platform built on an open data model (Apache Iceberg). We help you ingest and normalize data from common security sources into a data lake and offer a serverless platform to deploy & run Python detections-as-code on these events in realtime. Although the only supported query service is currently Athena, we plan to integra…
Earlier quoted context omitted.
It is similar, although Snowflake is more of a query engine whereas we are a cloud security data platform built on an open data model (Apache Iceberg). We help you ingest and normalize data from common security sources into a data lake and offer a serverless platform to deploy & run Python detections-as-code on these events in realtime. Although the only supported query service is currently Athena, we plan to integra…
Respectfully, this guy's description of Snowflake is very wrong - it's much more than a query engine. Snowflake already supports Iceberg format. If anything Snowflake is better described as similar to what OP is making (but for all data, not just security data): a cloud data platform that supports open data models (Apache Iceberg).
Anyway, he acknowledged Snowflake has Iceberg support by planning to integrate with Snowflake.
This sounds very appealing. Our IDS solution outputs zeek/suricata info to s3 as dns.1234.log.gz, http.1234.log.gz, etc. Can these files be handled automatically?
Feel free to join our Discord, happy to walk you through the steps and learn about your use case.