Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

github.com

Post reply on HN
Print view View original

31–40 of 82 posts

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#31

Post by xorcist »

Fetching logs regularly sounds hard? Wouldn't you need to keep track of the position of all files, with heuristics around file rotations? And if something catastrophic happens, the most interesting data would be in that last block which couldn't be polled?

Normally you'd avoid all that complexity by shipping logs the other way, sending from each machine. That way you can keep state locally should you need to. All unix-like systems do this out of the box, and almost all software supports the syslog protocol to directly stream logs. But you can also use something like filebeat and a bunch of other modern alternatives.

The analyzer can then run locally on the log server and a whole lot of complexity just disappears.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#32

Post by simonw »

I love this!

Log analysis isn't one of the core use-cases for Datasette, but I've done my own experiments with it that have worked pretty well - anything up to 10GB or so of data is likely to work just fine if you pipe it into SQLite, and you could go a lot larger than that with a bit of tuning.

I added some features to my sqlite-utils CLI tool a while back to help with log ingestion as well: https://simonwillison.net/2022/Jan/11/sqlite-utils/

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#33

Post by dynisor »

emodendroket wrote: post #27
rollcat wrote: post #10

"This simple tool solves X at my org" is probably the most underrated type of project. There's not enough room to overcomplicate something that isn't a core part of the business, it must be practical to maintain, simple&stupid enough so that onboarding is not a hurdle, etc. I encourage everyone to share your "splunk in 1kloc of Python" projects! Some of my own: - https://github.com/rollcat/judo is Ansible without Pyt…

> There's not enough room to overcomplicate something that isn't a core part of the business, it must be practical to maintain, simple&stupid enough so that onboarding is not a hurdle, etc. You would think. But no, there is lots of room to make it over complicated without the corresponding efforts to manage the complexity.

Don’t worry, that’s just tech debt and we will deeefinitely come back to it next sprint.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#34

Post by xorcist »

rollcat wrote: post #10

"This simple tool solves X at my org" is probably the most underrated type of project. There's not enough room to overcomplicate something that isn't a core part of the business, it must be practical to maintain, simple&stupid enough so that onboarding is not a hurdle, etc. I encourage everyone to share your "splunk in 1kloc of Python" projects! Some of my own: - https://github.com/rollcat/judo is Ansible without Pyt…

Your software is cool but the description is a bit unfair to Ansible. Ansible works by solving for desired state. This software runs scripts, it replaces "for host in; do ssh $host < script.sh; done".

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#35

Post by 10000truths »

dsXLII wrote: post #9
hliyan wrote: post #5

A long long time ago, I used a series of tail -f's and unix pipes to aggregate logs, and grep, less and awk to analyse them. There were about 20 different services written in C++, each producing over 1GB of logs each day. Managed to debug some fairly complex algorithmic trading bugs. Twenty years later, I still can't fathom why we're spending so much money on Splunk, DataDog an the like.

Volume. 1GB of data per day is rounding error. If you have tens of thousands of servers, each generating hundreds of gigabytes of data per day, tail -f and grep don't scale especially well.

They scale perfectly fine, as long as you filter locally before aggregating. Lo and behold:

  mkdir -p /tmp/ssh_output
  while read ssh_host; do
      ssh "$ssh_host" grep 'keyword' /var/my/app.log > "/tmp/ssh_output/${ssh_host}.log" &
  done
Tweak as needed. Truncation of results and real-time tailing are left as an exercise to the reader.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#37

Post by RexM »

egwor wrote: post #26
eigenvalue wrote: post #14

Earlier quoted context omitted.

Thanks, based on the dismissive replies to my original comment in the Splunk acquisition discussion, I thought this would get a lot of hostile takes saying that it was dumb, that I reinvented the wheel because I didn't want to spend 2 weeks trying to figure out opentelemetry nonsense and tools X, Y, and Z, that it was trivial, that it wouldn't scale, etc. But people are actually being surprisingly nice and friendly!…

I suggest you sell it to Oracle, get some popcorn and watch the Cisco vs Oracle log war begin!

mired in antitrust lawsuits

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#38

Post by eigenvalue »

xorcist wrote: post #31

Fetching logs regularly sounds hard? Wouldn't you need to keep track of the position of all files, with heuristics around file rotations? And if something catastrophic happens, the most interesting data would be in that last block which couldn't be polled? Normally you'd avoid all that complexity by shipping logs the other way, sending from each machine. That way you can keep state locally should you need to. All uni…

I considered doing it the way you described, but then you need to deploy software on every single one of your machines and make sure it's running, that it's not accidentally using up 99% of your CPU (I've had bad experiences with the monitoring agents for Splunk and Netdata misbehaving and slowing down the machines and causing problems), etc. Whereas with the "pull" approach I used in my tool, you don't need to deploy ANY software to the machines you are monitoring-- you just connect with SSH and grab the files you need and do all the work on your control node.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#39

Post by [deleted] »

rollcat wrote: post #10

"This simple tool solves X at my org" is probably the most underrated type of project. There's not enough room to overcomplicate something that isn't a core part of the business, it must be practical to maintain, simple&stupid enough so that onboarding is not a hurdle, etc. I encourage everyone to share your "splunk in 1kloc of Python" projects! Some of my own: - https://github.com/rollcat/judo is Ansible without Pyt…

[deleted]

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#40

Post by runjake »

Neat! Definitely a better solution for single source logs. Splunk is ridiculous and Cisco acquiring it isn't going to make that better.

For others with a bit more complex needs, take a look at the free (or paid) versions of Graylog Open[1].

It's really improved over the years. I had messed with Graylog in it's early days but was turned off by it. A few years back, I encountered someone doing some neat stuff with it. It looked much improved. I stood up a "pilot project" to test, and it's now been running for years and several different people use it for their areas of responsibility.

It does log collection/transforming and graphing and dashboarding and we use the everloving crap out of it at work. I wish I could publicly post some of the stuff we're doing with it.

It takes input from just about any source.

1. https://graylog.org/products/source-available/

Post reply on HN