Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Datomic and Content Addressable Techniques

latacora.com

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Datomic and Content Addressable Techniques

#21

Post by lvh »

FYI: we're planning a followup post for people who are less interested in the Datomic mechanics and more interested in the usefulness to investigations and other security functions. Informally, I think of it this way: your SIEM has the deltas, but often you want the contextual states in between the deltas (and a lot of investigation is about trying to reason about that state). We built this tool originally to support that, and it turned out that approach was also super useful for things like compliance, CSPM...

We already wrote a high level blog post here: https://www.latacora.com/blog/2023/11/01/our-approach-to-bui... -- but the one I'm hoping to write is more of a case study.

Re: Datomic and Content Addressable Techniques

#22

Post by lvh »

w10-1 wrote: post #3

Their content-addressing hash would seem critical, but the "combineUnordered" hash they use just adds each byte. The API is clear that this is only as good as possible, and I'm not sure I would rely on it for data used for security investigations. I suspect they'll come up with something like an arbitrary but fixed order over keys that would improve hash quality. More generally, while I can maybe understand what they…

Re: hashing: Yes, but I'll leave that one to Paul who is a lot smarter than I am :)

Re: QA: can you say a bit more about the type of coverage you're worried about? Is your concern that we'd be missing APIs, or that the storage format itself breaks, resulting in fact elision? payne (the underlying project) has a borderline obnoxious amount of tests, but that doesn't mean we didn't miss anything :)

Re: Datomic and Content Addressable Techniques

#23

Post by paulvrutledge »

w10-1 wrote: post #3

Their content-addressing hash would seem critical, but the "combineUnordered" hash they use just adds each byte. The API is clear that this is only as good as possible, and I'm not sure I would rely on it for data used for security investigations. I suspect they'll come up with something like an arbitrary but fixed order over keys that would improve hash quality. More generally, while I can maybe understand what they…

I suspect you're right and was already having similar thoughts regarding the hashing scheme. I put a patch together and am going to supplement with some additional tests of the collision space.

Unordered hashes made more sense for arbitrary Clojure data structures where the keys might be complex compound objects, but once we're in the land of datoms with finite value data types it's pretty easy to enforce a consistent ordering.

(disclaimer: I wrote much of the feature and post)

Re: Datomic and Content Addressable Techniques

#25

Post by juoitre »

w10-1 wrote: post #3

Their content-addressing hash would seem critical, but the "combineUnordered" hash they use just adds each byte. The API is clear that this is only as good as possible, and I'm not sure I would rely on it for data used for security investigations. I suspect they'll come up with something like an arbitrary but fixed order over keys that would improve hash quality. More generally, while I can maybe understand what they…

> in a way that's convincing to customers

Customers of this sort of security consulting are largely uninterested in the security as much as the audit report that allows them to say to their customers and incestors “we had these security professionals look at our stuff and this is what they said”.

Re: Datomic and Content Addressable Techniques

#26

Post by puredanger »

linotype wrote: post #2

Separate question: are there any new startups out there writing code primarily in Clojure?

The annual Clojure/conj conference is coming up next month in Alexandria, VA (https://2024.clojure-conj.org) and there will be talks from a variety of companies, both big and small.

Re: Datomic and Content Addressable Techniques

#27

Post by fiddlerwoaroof »

finnh wrote: post #10
fiddlerwoaroof wrote: post #9

Earlier quoted context omitted.

Even if you never push the code to production, it’s a crazy gorse multiplier for the explore and adventure prototype phase of a project

"gorse multiplier" wins today's "best transcription error" award, IMO category: edit-distance-two

I usually catch these too and ninja edit them out

Re: Datomic and Content Addressable Techniques

#28

Post by jacobobryant »

finnh wrote: post #10
fiddlerwoaroof wrote: post #9

Earlier quoted context omitted.

Even if you never push the code to production, it’s a crazy gorse multiplier for the explore and adventure prototype phase of a project

"gorse multiplier" wins today's "best transcription error" award, IMO category: edit-distance-two

2nd place probably goes to another comment in this thread: https://news.ycombinator.com/item?id=41560711

Re: Datomic and Content Addressable Techniques

#29

Post by lvh »

juoitre wrote: post #25
w10-1 wrote: post #3

Their content-addressing hash would seem critical, but the "combineUnordered" hash they use just adds each byte. The API is clear that this is only as good as possible, and I'm not sure I would rely on it for data used for security investigations. I suspect they'll come up with something like an arbitrary but fixed order over keys that would improve hash quality. More generally, while I can maybe understand what they…

> in a way that's convincing to customers Customers of this sort of security consulting are largely uninterested in the security as much as the audit report that allows them to say to their customers and incestors “we had these security professionals look at our stuff and this is what they said”.

Some of our customers, like Tailscale, are a helluva lot more picky than that.

Re: Datomic and Content Addressable Techniques

#30

Post by [deleted] »

lvh wrote: post #21

FYI: we're planning a followup post for people who are less interested in the Datomic mechanics and more interested in the usefulness to investigations and other security functions. Informally, I think of it this way: your SIEM has the deltas, but often you want the contextual states in between the deltas (and a lot of investigation is about trying to reason about that state). We built this tool originally to support…

[deleted]
Post reply on HN