Live data from Hacker News

AOL Moloch: open-source, large scale, packet-capturing, indexing database system

molo.ch

131–140 of 156 posts

Re: AOL Moloch: open-source, large scale, packet-capturing, indexing database system

#131
post #6

Not the greatest choice of name: Moloch[a] (also Molech, Mollok, Milcom, or Malcam) is the biblical name of a Canaanite god associated with child sacrifice, through fire or war. From: https://en.wikipedia.org/wiki/Moloch

Perhaps they were going for an association with Allen Ginsburg's Howl ( https://www.poetryfoundation.org/poems/49303/howl ) instead?

Wasn't Allen Ginsburg also in favor of child abuse?... He was a member of NAMBLA...

At least Moloch isn't real.

Re: AOL Moloch: open-source, large scale, packet-capturing, indexing database system

#132
I read the instructions but was not clear. Is this just a pcap viewer or something more? The way I read it, it appears to be a running daemon which listens on all port which saves the pcap file, which then exposes API's for accessing such data.

If you had an application http server running, is traffic sent to Moloch first, and forwarded to the http server like a proxy?

Re: AOL Moloch: open-source, large scale, packet-capturing, indexing database system

#133

Earlier quoted context omitted.

What can you do with a packet capture on a modern high security network besides go, "yup, that's TLS"? May DoD turns off the forward secrecy stuff and escrows keys?

In large part you're not looking at TLS application-data with this stuff; you're monitoring internal networks and all the protocols they run, in part so you can retroactively see if exploits, once revealed, have been run. For that kind of stuff you often care a lot more about, say, SMB dissection than you do about what stupid websites people are looking at. The longstanding existence of tools like these --- and there…

Interesting. A lot of Big Tech has started to encrypt even internal network traffic, though. Wouldn't the spooks be way ahead of us on that one?

Re: AOL Moloch: open-source, large scale, packet-capturing, indexing database system

#134

The Filesystem Hierarchy Standard [0] has been around for ~25 years but it still took quite a long time before most Linux distros decided to adhere to it (for the most part; some still do "non-standard" things at times). Now that we're to that point, please stop screwing it up and coming up wih your own locations for application binaries, data, etc. To be clear, the "/data" directory -- under which Moloch's pre-built…

If you look at the source code of most open source software, the devs typically install by default to arbitrary directories. They either don't know about, or care about, operating system standards. When Linux distributions package that software, they choose to change how that software works to align with standards. Usually their own standards, but those can often align with distro-independent standards too. But as a…

Not everyone is using a package manager for every install, and even then it's not a magic bullet. I'm a Mac user and I've no idea why devs insist on installing to places like /usr/local or putting configuration in a dot file in my home directory when there's a standard place to put those (~/Library/Application Support), among other well thought out directory structuring[1].

Further to this, the moment you want to install somewhere else or have a dependency elsewhere (e.g. /opt/anything) builds break because of hard coded paths and poor assumptions. To top it all, I've no idea why a package manager like Homebrew then contributes to all of this by acting like no of this matters because a Mac is just like Linux, apparently.

If they don't like Apple's way then why not XDG[2]? Reinventing the wheel badly is more than just hubris, it breaks things and it's annoying.

[1] https://developer.apple.com/library/archive/documentation/Ma... [2] https://specifications.freedesktop.org/basedir-spec/latest/

Re: AOL Moloch: open-source, large scale, packet-capturing, indexing database system

#136

The Filesystem Hierarchy Standard [0] has been around for ~25 years but it still took quite a long time before most Linux distros decided to adhere to it (for the most part; some still do "non-standard" things at times). Now that we're to that point, please stop screwing it up and coming up wih your own locations for application binaries, data, etc. To be clear, the "/data" directory -- under which Moloch's pre-built…

This post 'gonna cost me, but what you write is endogenous to the FHS making little sense to most people who don't have a unix background.

"Where ARE my programs / settings that I installed, I always have to google it!"

is interrelated with someone else asking

"Where should I PUT this program / setting / data? I guess I'll chose one of those three or so locations where it may fit, or several"

For example: Why isn't there a data directory to be found? Where does the data go? Why not replace nondescript and unhelpful names like "opt" and "usr" (which is btw. not where the user data is!)?

Re: AOL Moloch: open-source, large scale, packet-capturing, indexing database system

#137

The Filesystem Hierarchy Standard [0] has been around for ~25 years but it still took quite a long time before most Linux distros decided to adhere to it (for the most part; some still do "non-standard" things at times). Now that we're to that point, please stop screwing it up and coming up wih your own locations for application binaries, data, etc. To be clear, the "/data" directory -- under which Moloch's pre-built…

If the use of /data is the reason why you're not using this software, I'm pretty sure you wouldn't find any use of it if it stored data someplace in /var either.

Personally, I think the FHS is stupid marketing wankery that barely made sense in 1995 when the "preciousness" of the root partition was relevant, and has in the annals of time, caused more problems than it ever could have solved. Hey, what's the path to python? Is it:

    /usr/bin/python
    /usr/bin/python2
    /usr/bin/python3
    /usr/local/bin/python
    /usr/local/bin/python2
    /usr/local/bin/python3
    /usr/local/bin/python2.7
    /usr/local/bin/python3.7
This is perhaps the only question I need a "standard" to solve, and I've given up[†]. I don't need to know mail is "sometimes" in /var and sometimes in /var/local. I don't need to know that some of my configuration files are in /etc and others are in /usr/local/etc whilst still others are in /lib someplace. Where are my libraries? maybe /lib, /usr/lib /usr/local/lib /usr/lib64 /usr/local/lib64 and who knows where else. Fuck it, I'll just use find /.

I saw this great quote recently -- I know it was about something else, but I still wish Dan and the other FSSTND/FHS people could have seen and understood it all those years ago:

I suppose people blindly following suggestions on the Internet will eventually learn a hard lesson -- don't.

Because that is what the FHS was: A bunch of suggestions on a mailing list, with a smattering of armchair philosophy lacking any real analysis. But I mean, that's just my opinion. What if I'm wrong? ¯\_(ツ)_/¯

[†]: The answer is whatever /usr/bin/env python says.

Re: AOL Moloch: open-source, large scale, packet-capturing, indexing database system

#138
post #136

The Filesystem Hierarchy Standard [0] has been around for ~25 years but it still took quite a long time before most Linux distros decided to adhere to it (for the most part; some still do "non-standard" things at times). Now that we're to that point, please stop screwing it up and coming up wih your own locations for application binaries, data, etc. To be clear, the "/data" directory -- under which Moloch's pre-built…

This post 'gonna cost me, but what you write is endogenous to the FHS making little sense to most people who don't have a unix background. "Where ARE my programs / settings that I installed, I always have to google it!" is interrelated with someone else asking "Where should I PUT this program / setting / data? I guess I'll chose one of those three or so locations where it may fit, or several" For example: Why isn't t…

[deleted]

Re: AOL Moloch: open-source, large scale, packet-capturing, indexing database system

#139
post #136

The Filesystem Hierarchy Standard [0] has been around for ~25 years but it still took quite a long time before most Linux distros decided to adhere to it (for the most part; some still do "non-standard" things at times). Now that we're to that point, please stop screwing it up and coming up wih your own locations for application binaries, data, etc. To be clear, the "/data" directory -- under which Moloch's pre-built…

This post 'gonna cost me, but what you write is endogenous to the FHS making little sense to most people who don't have a unix background. "Where ARE my programs / settings that I installed, I always have to google it!" is interrelated with someone else asking "Where should I PUT this program / setting / data? I guess I'll chose one of those three or so locations where it may fit, or several" For example: Why isn't t…

/usr is absolutely where user data was. That's what it stands for "users".

First when / ran out of space, new programs were put on the other disk /usr in /usr/bin so everyone could assume programs were in /bin except ones that were newly installed so they would be in /usr/bin

When /usr ran out of space, a new disk was added as /home and things that were easier to move (user directories) were moved first leaving things that everyone's script was depending on (#!/usr/bin...) where they were.

This was done out of necessity, not out of good taste.

Re: AOL Moloch: open-source, large scale, packet-capturing, indexing database system

#140
post #139
post #136

Earlier quoted context omitted.

This post 'gonna cost me, but what you write is endogenous to the FHS making little sense to most people who don't have a unix background. "Where ARE my programs / settings that I installed, I always have to google it!" is interrelated with someone else asking "Where should I PUT this program / setting / data? I guess I'll chose one of those three or so locations where it may fit, or several" For example: Why isn't t…

/usr is absolutely where user data was. That's what it stands for "users". First when / ran out of space, new programs were put on the other disk /usr in /usr/bin so everyone could assume programs were in /bin except ones that were newly installed so they would be in /usr/bin When /usr ran out of space, a new disk was added as /home and things that were easier to move (user directories) were moved first leaving thing…

/usr stands for Unix System Resources
Post reply on HN