Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

github.com

Post reply on HN
Print view View original

61–70 of 82 posts

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#61

Post by rollcat »

eigenvalue wrote: post #14
rollcat wrote: post #10

"This simple tool solves X at my org" is probably the most underrated type of project. There's not enough room to overcomplicate something that isn't a core part of the business, it must be practical to maintain, simple&stupid enough so that onboarding is not a hurdle, etc. I encourage everyone to share your "splunk in 1kloc of Python" projects! Some of my own: - https://github.com/rollcat/judo is Ansible without Pyt…

Thanks, based on the dismissive replies to my original comment in the Splunk acquisition discussion, I thought this would get a lot of hostile takes saying that it was dumb, that I reinvented the wheel because I didn't want to spend 2 weeks trying to figure out opentelemetry nonsense and tools X, Y, and Z, that it was trivial, that it wouldn't scale, etc. But people are actually being surprisingly nice and friendly!…

> But people are actually being surprisingly nice and friendly! I guess people just really hate Splunk!

Best things in life come through love and passion. Frustration can be a good motivator but don't let it guide you.

> I thought this would get a lot of hostile takes [...]

To be entirely honest with you, recognizing and praising the good parts is a lot easier than giving proper feedback on what needs to be improved;)

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#62

Post by msm_ »

eigenvalue wrote: post #48
msto wrote: post #45

Earlier quoted context omitted.

* barely any comments and not a single docstring in the entire kiloline file

I find comments annoying to read and write and distracting. I’d rather fit more code on the screen at once and instead focus on making the variable names and function names really descriptive and clear so you immediately grasp what it’s doing from context alone. Nowadays, if you really need comments to tell you what code is doing, you can just throw it into ChatGPT and get it that way.

I disagree with comments being distracting. You can overdo them, but one good comment - like "baz = 1 # the default is 1 instead of 0, because most real-world production servers foobars 1s into 7s" can save days of frustration.

In general, comments often add a very vaulable context to your code in a way that readaable function/variable names can't.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#63

Post by oblvious-earth »

eigenvalue wrote: post #46
oblvious-earth wrote: post #17

Quickly skimming some points that would irratate me if I had to maintain this script: * Importing Paramiko but regularly call `ssh` via subprocess * Unused functions like `execute_network_commands_func` * Sharing state via a global instead of creating a class Overall it's fit for purpose, but makes a lot of assumptions about the host and client machines. As you said in the thread you're running a very small number of…

Thanks for the feedback. I do use Paramiko for some things. I tried to use it for everything in the project but ran into some weird stuff that wouldn't work reliably for me, which is why I switched some of it over to using SSH directly via subprocess (it was a few months ago so I don't even remember now what it was; I believe it was also performance related, since I'm trying to SSH to tons of machines at the same tim…

> I don't think globals are so awful for certain things. I prefer a more functional approach where you have simple composable standalone functions instead of classes. Obviously classes have a role, but I find they sometimes overly complicate things and make the logic harder to follow and debug.

But "globals" and "composable standalone functions" are contradictory, if you're mutating global state your function is neither composable nor standalone.

What you've got is a poor mans class instance using global instead of self.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#64

Post by moqmar »

vincnetas wrote: post #13
dsXLII wrote: post #9

Earlier quoted context omitted.

Volume. 1GB of data per day is rounding error. If you have tens of thousands of servers, each generating hundreds of gigabytes of data per day, tail -f and grep don't scale especially well.

100GB of logs per day? what kind of applications are that chatty?

Probably Java/JVM... Never seen something where all kinds of libraries log more.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#65

Post by quickthrower2 »

diarrhea wrote: post #41
getrealyall wrote: post #18

Earlier quoted context omitted.

Financialization and mediocre developers. I haven't worked with too many people I could actually trust to even emit logs correctly, let alone develop a tool to collect and aggregate them. I've also been told, time and again, in no uncertain terms, to "buy as much as possible". We've reached the logical conclusion of SaaS-everything: every company just cobbles together expensive, overcomplicated computers from other e…

Buying everything and SaaSing the whole place up is a true killjoy. I giggle with joy whenever I am allowed to write code. And then a support request comes in that I get assigned to, “thing in SaaS doesn’t work please fix”. And all you have to debug that SaaS is their UI. The checkbox in question is on, you notice, so it can only be a bug on their side. Off to contacting support as the only available avenue. Incredib…

Yay I get to write code! Oh it’s Pulumi code that wraps terraform to deploy Elastic Search and configure networking to allow logs from a kubernetes we deployed the same way.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#66

Post by matrss »

eigenvalue wrote: post #56
matrss wrote: post #54

Earlier quoted context omitted.

Last time I checked, ansible playbooks were also essentially just sequential steps to execute via ssh, albeit in yaml format. There was certainly no way to describe a desired state nor did ansible accomplish consistently bringing a system into some desired state. Two executions of the same playbook could result in a very different system state for example, depending on what happened in between. The only systems I am…

Ansible is definitely all about solving from a specified target state and ensuring that it is followed. It’s even at the level of syntax for ansible, which is how it can be used totally declaratively. And if you stick to the native idiomatic ansible way of doing everything (as opposed to doing hacky stuff with ad hoc shell commmands), you get automatic idempotence and other nice stuff “for free”.

So, the last time I used ansible, which was quite a while ago to be fair, there was a builtin way to install packages with apt on the target system. You could add packages to a list to be installed and that would work. But removing them from that "declared state" would not remove them on the target system on the next playbook run. You would have to add an explicit uninstall command. And that is where ansible failed to be declarative. Did that change in the meantime?

Ansible might provide idempotence for the builtin things (although I would argue it doesn't, at least not on a bit-by-bit level, since you can't pin down specific versions of package repositories and stuff like that), but to be declarative it would need to provide a 1-to-1 mapping from declared state to running system state. And if what I described above is still the case, then it simply does not do that.

In my experience, ansible tries to build a declarative interface to an imperative mode of system management, which works to some extent, but breaks down in more complex cases because building this declarative interface can only be a leaky abstraction without the right foundation.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#67

Post by matrss »

xorcist wrote: post #59
matrss wrote: post #54

Earlier quoted context omitted.

Last time I checked, ansible playbooks were also essentially just sequential steps to execute via ssh, albeit in yaml format. There was certainly no way to describe a desired state nor did ansible accomplish consistently bringing a system into some desired state. Two executions of the same playbook could result in a very different system state for example, depending on what happened in between. The only systems I am…

You can run sequential commands with ansible, but then you're just using it as a replacement for "ssh $host < script.sh". You would be missing out on most of the usefulness of the tool. It's meant to be used declaratively. The manual describes it quite well. In the same type of tools are puppet and indeed nix, but with one important difference for the latter: nix is also a package manager, which allows for more a fin…

True, nix is also a package manager, and that is the crucial step necessary to actually provide a declarative interface to system state. Without it you can only get the leaky abstraction that is ansible.

To illustrate my point, imagine this playbook:

  ---
  - name: Bring system into some state
    hosts: localhost
  
    tasks:
      - name: Install hello
        ansible.builtin.apt:
          name: hello
        become: true
Apply it and you get GNU hello installed. Now remove the package installation step, which really is just a glorified "ssh $host 
  ---
  - name: Bring system into some state
    hosts: localhost
  
    tasks:
Apply that and you will still have hello installed, even though it was removed from the "declared state". This just pretends to be declarative but really isn't, the tasks are still imperative steps.

Not to blame ansible for this, it is just that ansible is build on a foundation that inherently makes declarative system management impossible to begin with and ansible is more or less the best thing you could do given the constraints.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#68

Post by hjvalente »

eigenvalue wrote: post #38
xorcist wrote: post #31

Fetching logs regularly sounds hard? Wouldn't you need to keep track of the position of all files, with heuristics around file rotations? And if something catastrophic happens, the most interesting data would be in that last block which couldn't be polled? Normally you'd avoid all that complexity by shipping logs the other way, sending from each machine. That way you can keep state locally should you need to. All uni…

I considered doing it the way you described, but then you need to deploy software on every single one of your machines and make sure it's running, that it's not accidentally using up 99% of your CPU (I've had bad experiences with the monitoring agents for Splunk and Netdata misbehaving and slowing down the machines and causing problems), etc. Whereas with the "pull" approach I used in my tool, you don't need to deplo…

having a Netdata agent taking your machine's CPU to 99% shouldn't happen, not sure when was the list time you tried it but a lot of recent improvements have been done on the Netdata Agent

also, with Netdata you can achieve the same architecture design using a Netdata Parent that could be your "control node" and to where you stream the metrics of the nodes you want to keep running with as less load as possible - you can even offload the health engine and the machine learning

take a look at https://learn.netdata.cloud/docs/streaming/ and https://learn.netdata.cloud/docs/configuring/how-to-optimize...

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#69

Post by knowsuchagency »

oblvious-earth wrote: post #63
eigenvalue wrote: post #46

Earlier quoted context omitted.

Thanks for the feedback. I do use Paramiko for some things. I tried to use it for everything in the project but ran into some weird stuff that wouldn't work reliably for me, which is why I switched some of it over to using SSH directly via subprocess (it was a few months ago so I don't even remember now what it was; I believe it was also performance related, since I'm trying to SSH to tons of machines at the same tim…

> I don't think globals are so awful for certain things. I prefer a more functional approach where you have simple composable standalone functions instead of classes. Obviously classes have a role, but I find they sometimes overly complicate things and make the logic harder to follow and debug. But "globals" and "composable standalone functions" are contradictory, if you're mutating global state your function is neit…

It's a single script. Globals are fine--they're even marked as such.

Re: Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup

#70

Post by vincnetas »

moqmar wrote: post #64
vincnetas wrote: post #13

Earlier quoted context omitted.

100GB of logs per day? what kind of applications are that chatty?

Probably Java/JVM... Never seen something where all kinds of libraries log more.

Log level configuration is a cheap solution in this case.
Post reply on HN