Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go is portable, until it isn't

simpleobservability.com

Post reply on HN
Print view View original

121–130 of 138 posts

Re: Go is portable, until it isn't

#121

Post by tkone »

combiBean wrote: post #84

Was there not a third option: Calling the journalctl CLI as a child process and consume the parsed logs from the standard output? This might have avoided both the requirement to use CGO and also to write a custom parser. But I guess I am missing something.

Ironically this is EXACTLY what the journald receiver for OpenTelemetry does, which, as they noted, is written in go.

Specifically because you're only supposed to use that OR the c bindings by design because they want the ability to change in the internal format when it's necessary.

Re: Go is portable, until it isn't

#122

Post by jeremyjh »

khazit wrote: post #119
jeremyjh wrote: post #109

Earlier quoted context omitted.

journalctl is designed for these use cases and has options to solve those issues. The lazy part here is you not doing any research about this tool before dismissing it as "not best practice", which is exactly what the fuckups who wrote this article did.

We dismissed using journalctl at the very start. We’ve had similar experiences with other CLI tools: the moment you start embedding them inside a program, you introduce a whole new class of problems. What if journalctl exits? What if it outputs an error? What if it hangs? On top of that, you have to manage the subprocess lifecycle yourself. It’s not as easy as it may seem. You can also argue that sd_journal (the C AP…

Does Go really not have any libraries capable of supervising an external program? If you'd considered journalctl, why didn't you mention it in the article? As many have pointed out here, it is the obvious and intended way to do this, and the path you chose was harder for reasons that seemed to surprise you but were entirely foreseeable.

Re: Go is portable, until it isn't

#123

Post by dwattttt »

regularfry wrote: post #112

All of this, every last bit of complexity and breakage and sweat, is downstream of this: > Journal logs are not stored in plain text. They use a binary format And it was entirely predictable and predicted that this sort of problem would be the result when that choice was made.

That's why I hexify all binary files, to make it easier to understand them.

Re: Go is portable, until it isn't

#124

Post by mxey »

khazit wrote: post #119
jeremyjh wrote: post #109

Earlier quoted context omitted.

journalctl is designed for these use cases and has options to solve those issues. The lazy part here is you not doing any research about this tool before dismissing it as "not best practice", which is exactly what the fuckups who wrote this article did.

We dismissed using journalctl at the very start. We’ve had similar experiences with other CLI tools: the moment you start embedding them inside a program, you introduce a whole new class of problems. What if journalctl exits? What if it outputs an error? What if it hangs? On top of that, you have to manage the subprocess lifecycle yourself. It’s not as easy as it may seem. You can also argue that sd_journal (the C AP…

> You can also argue that sd_journal (the C API) exists for this exact reason, rather than shelling out to journalctl.

Quoting from https://systemd.io/JOURNAL_FILE_FORMAT/

> If you need access to the raw journal data in serialized stream form without C API our recommendation is to make use of the Journal Export Format, which you can get via journalctl -o export or via systemd-journal-gatewayd.

Certainly sounds like running journalctl, or using the gateway, is a supported option.

Re: Go is portable, until it isn't

#125

Post by mxey »

jeremyjh wrote: post #122
khazit wrote: post #119

Earlier quoted context omitted.

We dismissed using journalctl at the very start. We’ve had similar experiences with other CLI tools: the moment you start embedding them inside a program, you introduce a whole new class of problems. What if journalctl exits? What if it outputs an error? What if it hangs? On top of that, you have to manage the subprocess lifecycle yourself. It’s not as easy as it may seem. You can also argue that sd_journal (the C AP…

Does Go really not have any libraries capable of supervising an external program? If you'd considered journalctl, why didn't you mention it in the article? As many have pointed out here, it is the obvious and intended way to do this, and the path you chose was harder for reasons that seemed to surprise you but were entirely foreseeable.

JFTR, of course it has a library for it https://pkg.go.dev/os/exec

Re: Go is portable, until it isn't

#126

Post by mxey »

zbentley wrote: post #113
IshKebab wrote: post #94

Earlier quoted context omitted.

It's generally less robust to run CLI tools and scrape the output. Usually it isn't intended to be machine readable, and you have to handle extra failure modes, like incompatible tool versions, missing tools, incorrect parsers, etc. It's the lazy-but-bad solution.

I think a lot is riding on that “generally”. You’re right that the default approach/majority of cases should avoid shelling out wherever possible, but there are a large minority of situations where doing that does make sense, including: Calling a CLI tool which will be present everywhere your program might reasonably be installed (e.g. if your program is a MySQL extension, it can probably safely assume the existence…

I recently wrote some Go code for running containers and chose to use the docker CLI instead of an API client. The CLI is more well known and better documented, and this is what replacements like Podman support. When there’s a problem, it’s easier to reproduce it by running the same CLI command. It also meant I wouldn’t need a whole lot of dependencies, and we needed the docker CLI anyway.

Obviously you shouldn’t try to parse human-readable output.

Re: Go is portable, until it isn't

#127

Post by 1718627440 »

IshKebab wrote: post #94
combiBean wrote: post #84

Was there not a third option: Calling the journalctl CLI as a child process and consume the parsed logs from the standard output? This might have avoided both the requirement to use CGO and also to write a custom parser. But I guess I am missing something.

It's generally less robust to run CLI tools and scrape the output. Usually it isn't intended to be machine readable, and you have to handle extra failure modes, like incompatible tool versions, missing tools, incorrect parsers, etc. It's the lazy-but-bad solution.

> Usually it isn't intended to be machine readable

It usually is, because that is the UNIX philosophy and programs that intermingle output with layout often stop doing that, when they don't write to a terminal.

Re: Go is portable, until it isn't

#128

Post by 1718627440 »

spijdar wrote: post #24
dekhn wrote: post #22

Earlier quoted context omitted.

In general, cross compilers can do dynamic linking.

In my experience, the cross-compiler will refuse to link against shared libraries that "don't exist", which they usually don't in a cross compiler setup (e.g. cross compiling an aarch64 application that uses SDL on a ppc64le host with ppc64le SDL libraries) The usual workaround, I think, is to use dlopen/dlsym from within the program. This is how the Nim language handles libraries in the general case: at compile time…

Well, you need to link against them and you can't do that when they don't exist. I don't understand the purpose of a stub library, it is also only a file and if you need to provide that, you can also provide the real thing right away.

Re: Go is portable, until it isn't

#129

Post by CGamesPlay »

immibis wrote: post #69
CGamesPlay wrote: post #44

Use dlopen? I haven’t tried this in Go, but if you want a binary that optionally includes features from an external library, you want to use dlopen to load it.

It only works in a dynamically-linked binary, because the dynamic linker needs to be loaded.

That makes sense. Some digging turns up some workarounds; the best I found was https://github.com/jart/cosmopolitan/releases/tag/3.1

Re: Go is portable, until it isn't

#130

Post by vrighter »

nly wrote: post #68
mmulet wrote: post #9

I ran into this issue when porting term.everything[0] from typescript to go. I had some c library dependencies that I did need to link, so I had to use cgo. My solution was to do the build process on alpine linux[1] and use static linking[2]. This way it statically links musl libc, which is much friendlier with static linking than glibc. Now, I have a static binary that runs in alpine, Debian, and even bare container…

> and even bare containers. Strange, i thought the whole point of containers was to solve this problem.

The whole point of containers is to ship almost the whole OS with the application (It is a technical implementation of the "works on my machine" concept). If the OS you put in your container (by just pulling in a prebuilt image from somewhere) doesn't have the necessary things, then the application would fail to work just the same as if you ran it on the bare operating system with the the same missing libraries.
Post reply on HN