Live data from Hacker News

Stenographer – A full-packet-capture utility

github.com

21–22 of 22 posts

Re: Stenographer – A full-packet-capture utility

#21
Interesting!

An open source high performance rolling packet dump with packet index for incident response.

To be honest, I had imagined Google already had solutions like this internally :)

These have been commercially available for a few years, look at RSA Security Analytics (formerly NetWitness), or BlueCoat Security Analytics (formerly Solera). Or the (open source) Bro Time Machine. I used to work with one of these products in the past.

What make systems like this a lot more powerful is more and easier search and retrieval. While indexing IP numbers and port numbers is good, it will get much more useful if you can connect it to something like 'bro' and get session level data and then index filenames, user-agents, file hashes, and others pieces of information. I'm sure you can see the use cases.

Having an easy way to query 'all traffic with this particular user agent', together with the full packet capture, which allows you to write new rules, can significantly increase the efficiency of a security team.

Apart from the streaming analytics, once the PCAP data is stored, you can use mapreduce type operations on them to search through yesterday's data with today's IDS signatures (look at PacketPig/what Packetloop does). Maybe a lambda architecture is the way to go, or just reprocess old data through the same stream processing.

Cool work though! I'm curious where this will go next.

Re: Stenographer – A full-packet-capture utility

#22
post #19
post #4

Earlier quoted context omitted.

1) Performance. Zero copy ("The kernel writes them from the NIC to shared memory, then the kernel uses that same shared memory for O_DIRECT writes to disk. The packets transit the bus twice and are never copied from RAM to RAM."). Parallelism. 2) Disk management. Rotates old data, etc 3) Indexing and supports efficient retrieval while writing. It allows to analyse the traffic after the fact, at 10Gbps line speed.

You can get zero-copy for tcpdump with PF_RING or netmap.

I'm aware of libpcap's ability to share memory with a user buffer, but I didn't find any mention that tcpdump utility is actually written to exploit it for extra fast writes.

Look here how they handle this in stenographer: https://github.com/google/stenographer/blob/65fb928e6bce276c...

I guess that in principle they could have patched tcpdump, but it's probably easier to have a smaller software written to do exactly what you want rather than extend a general purpose mature complex tool such as tcpdump.

Post reply on HN