Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

github.com

Post reply on HN
Print view View original

11–20 of 82 posts

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#11

Post by eigenvalue »

Okkef wrote: post #7

"log files of several several gigabytes ... Process them in minutes" Thanks but no thanks.

I misspoke there-- meant to say:

"The application has been tested with log files several gigabytes in size from dozens of machines and can process all of it in minutes."

That's the time it takes to connect to 20+ machines, download multiple gigs of log files from all of them, and parse/ingest all the data into a sqlite. If you have a big machine with a lot of cores and a lot of RAM, it's incredibly performant for what it does.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#13

Post by vincnetas »

dsXLII wrote: post #9
hliyan wrote: post #5

A long long time ago, I used a series of tail -f's and unix pipes to aggregate logs, and grep, less and awk to analyse them. There were about 20 different services written in C++, each producing over 1GB of logs each day. Managed to debug some fairly complex algorithmic trading bugs. Twenty years later, I still can't fathom why we're spending so much money on Splunk, DataDog an the like.

Volume. 1GB of data per day is rounding error. If you have tens of thousands of servers, each generating hundreds of gigabytes of data per day, tail -f and grep don't scale especially well.

100GB of logs per day? what kind of applications are that chatty?

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#14

Post by eigenvalue »

rollcat wrote: post #10

"This simple tool solves X at my org" is probably the most underrated type of project. There's not enough room to overcomplicate something that isn't a core part of the business, it must be practical to maintain, simple&stupid enough so that onboarding is not a hurdle, etc. I encourage everyone to share your "splunk in 1kloc of Python" projects! Some of my own: - https://github.com/rollcat/judo is Ansible without Pyt…

Thanks, based on the dismissive replies to my original comment in the Splunk acquisition discussion, I thought this would get a lot of hostile takes saying that it was dumb, that I reinvented the wheel because I didn't want to spend 2 weeks trying to figure out opentelemetry nonsense and tools X, Y, and Z, that it was trivial, that it wouldn't scale, etc.

But people are actually being surprisingly nice and friendly! I guess people just really hate Splunk!

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#15

Post by getrealyall »

dsXLII wrote: post #9
hliyan wrote: post #5

A long long time ago, I used a series of tail -f's and unix pipes to aggregate logs, and grep, less and awk to analyse them. There were about 20 different services written in C++, each producing over 1GB of logs each day. Managed to debug some fairly complex algorithmic trading bugs. Twenty years later, I still can't fathom why we're spending so much money on Splunk, DataDog an the like.

Volume. 1GB of data per day is rounding error. If you have tens of thousands of servers, each generating hundreds of gigabytes of data per day, tail -f and grep don't scale especially well.

And I bet a hang glider can't fly from New York to Paris, either! The nerve!

Recall that the poster said this was for a small startup. If you're Google, by all means, use Google logging tools. If you aren't, then solve the problem you have, not the problem your résumé needs.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#16

Post by KaiserPro »

I used to work at a splunk shop. It was used for alerting, graphing & prediction. It was critical to how the company functioned.

There was lots of stuff that relied on splunk, and we had splunk specialists who knew the magic splunkQL to get the graph/data they wanted.

However, we managed to remove most of the need for splunk by using graphite/grafana. It took about 2-3 years but it meant that non techs could create dashbaords and alerts.

As someone once told me, splunk is the most expensive way you can ignore your data.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#17

Post by oblvious-earth »

Quickly skimming some points that would irratate me if I had to maintain this script:

* Importing Paramiko but regularly call `ssh` via subprocess

* Unused functions like `execute_network_commands_func`

* Sharing state via a global instead of creating a class

Overall it's fit for purpose, but makes a lot of assumptions about the host and client machines. As you said in the thread you're running a very small number of servers (less than 30). I've written similiar things over the years and they are great for what you need.

When I heavily used Splunk (back in 2013) I was in an application production support team that managed over 100 productions servers for over a dozen applications, there were dozens of other teams in similar situations across the company. The Splunk instance was managed by a central team, minimal assumptions about the client environment, had well defined permissions, understood common and essoteric logging formats, and could reinterpret the log structure at query time. A script like this is not competiting in that kind of situation.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#18

Post by getrealyall »

hliyan wrote: post #5

A long long time ago, I used a series of tail -f's and unix pipes to aggregate logs, and grep, less and awk to analyse them. There were about 20 different services written in C++, each producing over 1GB of logs each day. Managed to debug some fairly complex algorithmic trading bugs. Twenty years later, I still can't fathom why we're spending so much money on Splunk, DataDog an the like.

Financialization and mediocre developers. I haven't worked with too many people I could actually trust to even emit logs correctly, let alone develop a tool to collect and aggregate them.

I've also been told, time and again, in no uncertain terms, to "buy as much as possible". We've reached the logical conclusion of SaaS-everything: every company just cobbles together expensive, overcomplicated computers from other expensive, overcomplicated computer providers, resulting in expensive, bloated systems that barely work.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#19

Post by olafalo »

rollcat wrote: post #10

"This simple tool solves X at my org" is probably the most underrated type of project. There's not enough room to overcomplicate something that isn't a core part of the business, it must be practical to maintain, simple&stupid enough so that onboarding is not a hurdle, etc. I encourage everyone to share your "splunk in 1kloc of Python" projects! Some of my own: - https://github.com/rollcat/judo is Ansible without Pyt…

For me, it's configinator[0]. Write a spec file for a config like [1], get a Go file that loads a config from environment variables like [2]. Code-gen only, no reflection, fairly type-safe, supports enums, string, bool, and int64. I made it because it was gross to add new config vars in a project at work, and it's come in handy a lot!

[0] https://github.com/olafal0/configinator

[1] https://github.com/olafal0/configinator/blob/0576a53970bcb4d...

[2] https://github.com/olafal0/configinator/blob/0576a53970bcb4d...

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#20

Post by catlover76 »

That's cool!

I disagree though that the deal shows any bad judgment on Cisco's part; the gravamen of whether the acquisition was good is not whether many software developers can quickly develop replacements for their own use-cases, or how ergonomic the software is, but whether Splunk is a profitable business with a bunch of paying subscriptions/contracts that aren't going to go away any time soon.

Post reply on HN