Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Streamdal – an open-source tail -f for your data

github.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Show HN: Streamdal – an open-source tail -f for your data

#31

Post by dsies »

defanor wrote: post #30

The repository's README.md links to https://docs.streamdal.com/sdks >, which is not there. Those SDKs are simply libraries though, aren't they? "SDK" often stands for more than that (e.g., development tools, bits of code not properly packaged), and may be appalling if you don't want to wrap a project around such an SDK, as opposed to merely incorporating a library. But then I wonder why it has to be a library at all,…

Re: docs - oops. We were frantically putting stuff together and linking before the docs were in that location - the link is supposed to be https://docs.streamdal.com/en/core-components/sdk/ . Fixed it in the readme.

re: libs vs sdk - we named it that in anticipation of exactly having to do some funky stuff. As it stands, we are already doing grpc, Protobuf, wasm and having it all interop across all languages is not easy - so having to introduce some sort of a “helper” binding/lib is not at all unlikely.

Besides that, the “tail” part is really a small part of the functionality - the overall idea is that the sdk/lib has access to most/all I/O of the app and is able to interact with the payload that the caller provides before it is sent on its way.

Traditional pipes aren’t really in the equation.

We went with calling it “tail” because it’s easier to explain instead of “it’s a lib that an app owner can wrap their i/o calls with to enable calling dynamic wasm”… and that’s still not the whole thing haha

Last, here’s a diagram depicting the flow: https://docs.streamdal.com/_astro/StreamdalSdk.cd7c8d45.png

Re: Show HN: Streamdal – an open-source tail -f for your data

#32

Post by dsies »

smivan wrote: post #29

Congratulations on the launch! I have been a long time customer of the enterprise version of Streamdal, and I can confidently say Daniel and Ustin are absolutely KILLER engineers. Any time we spoke I always was impressed by their super deep experience and understanding of modern challenges! So good to see you getting some love on HN. Excited to implement this in some personal projects as well!

Thanks Ivan - really appreciate the kind words and support!

Re: Show HN: Streamdal – an open-source tail -f for your data

#33

Post by avgcorrection »

tfsh wrote: post #25
dsies wrote: post #22

Earlier quoted context omitted.

Hey there - we have documented the tech stack here: https://docs.streamdal.com/en/resources-support/open-source/ Tldr: go, grpc, Protobuf, wasm, deno, reactflow, ts And yep, you’re right - we are using protobuf to have a common schema between all SDKs, the server and UI. Re: sdk implementation - it’s basically implementing grpc methods, knowing how to exec wasm and doing a couple of extra things at instantiation. In…

Thanks for the info, sounds like you have a pretty solid tech stack :) Re Java - If you're looking to maximise compatability, then yeah you should aim to target an older JDK. Virtually all Java projects use at least JDK 8 so that can be a baseline, however many enterprise projects would use closer to JDK 18 at a guess (Google's internally aiming to migrate to 21 in 2024). Generally if there are libraries or features…

> Re Java - If you're looking to maximise compatability, then yeah you should aim to target an older JDK. Virtually all Java projects use at least JDK 8 so that can be a baseline,

Oracle says that JDK 11 is on “Extended Support” which comes after “Premier Support”.[1] Why not just support JDK 17 and higher?

[1]: https://endoflife.date/oracle-jdk

Re: Show HN: Streamdal – an open-source tail -f for your data

#34

Post by jtcap »

Maybe I need to read the docs in more detail, but a question that comes to mind is what is the impact on latency and reliability? Can StreamdalSDK.Process be made asynchronous so that any delay in processing the data, or unavailability of the server, has minimal impact on the flow of the application? That is, to make the observing as passive as possible?

Re: Show HN: Streamdal – an open-source tail -f for your data

#35

Post by jacobheric »

jtcap wrote: post #34

Maybe I need to read the docs in more detail, but a question that comes to mind is what is the impact on latency and reliability? Can StreamdalSDK.Process be made asynchronous so that any delay in processing the data, or unavailability of the server, has minimal impact on the flow of the application? That is, to make the observing as passive as possible?

In the node-sdk, process pipeline is async , see: https://github.com/streamdal/node-sdk/blob/main/src/streamda... (I'm the author of that). I believe this is also the case for the python and go sdks as well. So you can call it asynchronously for passive observability.

However, we implemented the pipeline rules in wasm with the goal of keeping the overhead as minimal as possible. So you could also use it as more of a data security or governance tool and invoke the pipelines synchronously and mask or prevent sensitive data before passing it along.

Re: Show HN: Streamdal – an open-source tail -f for your data

#36

Post by jacobheric »

tfsh wrote: post #19

This is awesome, the UI looks beautiful. I've noticed you've provided Go, Python, and Node SDKs. What's the general tech stack for these? I assume your usage of Protobufs is for a consistent schemas between languages? I ask because I'm curious as to how much work it is to define new SDKs for other languages, as I'd love a Java implementation - Ideally the SDK should be a pretty thin wrapper, simply calling the gRPC s…

> This is awesome, the UI looks beautiful.

Thank you! I wrote the UI! It's a pretty tricky UI stack as we update everything to the browser realtime in protobuf over grpc streaming (using grpc-web and protobuf-ts). There is a lot mapping we have to do to shape the data properly for React Flow so we do that server side in Deno before passing it along to the browser. We still have some optimization to do to keep the live tail view zippy, but it's a pretty solid foundation.

Re: Show HN: Streamdal – an open-source tail -f for your data

#38

Post by a1369209993 »

dsies wrote: post #23
a1369209993 wrote: post #20

Obligatory: We already have a open-source tail -f; it's called `tail -f`. (Kind of /s, but much like the infamous dropbox comment[0], it sucks that there exists a problem for this to solve in the first place.) 0: https://news.ycombinator.com/item?id=9224

I wonder if we screwed up by calling out “tail” - it is so much more then that - it executes wasm rules on the client that are pushed to it by the server AND because we have access to the data - we can expose a UI to see it flowing… like a “tail -f” - but that doesn’t quite flow off the tongue :) I’d urge you to check out the live demo and “tail” an app at runtime - it might be able to explain what we we are doing be…

I don't actually have a problem for this to solve - like the rsync guy, I know how to use actual `tail -f` (and eg `perl -E` et al) - I'm just self-aware enough to realise that most people either aren't up to the task of hacking together whatever functionality they'd actually end up using out of generic utilities at all, or at least wouldn't consider that so easy as to be the path of least resistance to get something done.

Re: Show HN: Streamdal – an open-source tail -f for your data

#39

Post by sshine »

dsies wrote: post #24
sshine wrote: post #21

Earlier quoted context omitted.

But can `tail -f` handle data ?

I know you kid - but the _data_ in this context is the data that the app is processing at runtime. Ie. If the app is reading from a DB - that’s what we are tailing. And if that’s already clear - sorry :)

It wasn’t, thanks. :)
Post reply on HN