Live data from Hacker News

In-Memory C++ Leap in Blockchain Analysis

caudena.com

1–10 of 72 posts

In-Memory C++ Leap in Blockchain Analysis

#1
Hey HN

We’re the core engineering team at Caudena (which is used globally by investigative and intelligence agencies, including: Europol, Interpol, BKA, DHS, IRS-CI, FBI, NPA and others), and we just released the technical details behind Prism - our real-time, in-memory C++ database for blockchain analysis.

To tackle the massive scale and complexity of blockchain data, we had to get creative with low-level engineering:

- We utilize barebone servers with 2TB RAM and 48 Cores.

- Implemented lock-free concurrent data structures

- Developed a custom memory management system

- Leveraging CPU-level vectorization

- Built a custom in-memory columnar/graph database from scratch

We’d love to AMA about:

- the engineering choices we made

- crazy optimizations that paid off

- pitfalls we hit

Ask us anything about scaling, memory trade-offs, building real-time analytics on immutable data, or the crypto-forensics space.

Looking forward to a great convo!

In-Memory C++ Leap in Blockchain Analysis
caudena.com

Re: In-Memory C++ Leap in Blockchain Analysis

#5
We built something very similar back in 2016, in the jvm with unsafe memory and garbage-free data structures to avoid GC pauses. The dynamic clustering is not too hard, are you able to dynamically undo a cluster when new information shows up?

Are you running separate instances per customer to separate the information they have access to?

Re: In-Memory C++ Leap in Blockchain Analysis

#6
Do you see crypto as anything more than scams/crime/speculation?

Most people involved in crypto pretend it's the future and their business models depend on pumping up crypto. That might be the same for you all but I figure of anyone in the space, a group dedicated to tracking down where coins are moving for government agencies (I assume for scams/crime reasons) might not have the wool so pulled over their eyes.

Re: In-Memory C++ Leap in Blockchain Analysis

#7
When implementing the lock-free stuff, was portability (across processors) a goal? If yes, did you have to deal with anything specific? Do you notice any difference in behavior of correct implementations when ran on different processors? How do you test for correctness of lock-free stuff?

EDIT: Oh and did you implement from scratch? Why not use eg. the RCU implementation from folly?

Re: In-Memory C++ Leap in Blockchain Analysis

#8

Do you see crypto as anything more than scams/crime/speculation? Most people involved in crypto pretend it's the future and their business models depend on pumping up crypto. That might be the same for you all but I figure of anyone in the space, a group dedicated to tracking down where coins are moving for government agencies (I assume for scams/crime reasons) might not have the wool so pulled over their eyes.

First of all, at Caudena, we are not involved in crypto projects or investments ourselves. Our expertise lies in analyzing blockchains and providing deep technical insights into how various blockchains operate. We focus on tracking and understanding the flow of digital assets, often in support of government agencies investigating scams, fraud, and other illicit activities.

That said, we absolutely believe that blockchain and cryptocurrency will shape the future of the financial system. When you look beyond the noise of scam tokens, speculative NFTs, and high-profile scandals, there is significant and meaningful financial innovation happening. This extends beyond DeFi to include the tokenization of RWA, where major institutions like BlackRock and JPM Chase are actively exploring and implementing blockchain-based solutions. Numerous projects are driving real progress, and there’s a slow but steady movement toward a more decentralized and transparent financial ecosystem.

Re: In-Memory C++ Leap in Blockchain Analysis

#10
post #5

We built something very similar back in 2016, in the jvm with unsafe memory and garbage-free data structures to avoid GC pauses. The dynamic clustering is not too hard, are you able to dynamically undo a cluster when new information shows up? Are you running separate instances per customer to separate the information they have access to?

Assuming by undoing you mean splitting the cluster:

A linked list can be split in two in O(1). When it comes to updating the roots for all the removed nodes, there is no easy way out, but luckily:

- This process can be parallelized.

- It could be done just once for multiple clustering changes.

- This is a multi-level disjoint set, not all the levels or sub-clusters are usually affected. Upper level clustering, which is based on lower confidence level, can be rebuilt more easily.

If by undoing you mean reverting the changes, we don’t use a persistent data structure. When we need historical clustering, we use a patched forest with concurrent hash maps to track the changes, and then apply or throw them away.

We use a single instance for all clients, but when one CFD server processes new block data, it becomes fully blocked for read access. To solve this, we built a smart load balancer that redirects user requests to a secondary CFD server. This ensures there's always at least two servers running, and more if we need additional throughput.

Post reply on HN