Show HN: Streamdal – an open-source tail -f for your data • Hacker News Board

Re: Show HN: Streamdal – an open-source tail -f for your data

#11

Post by jacobheric » Tue, Oct 31, 2023, 6:31 PM UTC

kensavage wrote: post #10

Node support - sweet!! Is there any way to interact with the data as it comes in real-time?

Hey there! I wrote the node sdk for this, you can use the node-sdk to to execute wasm-based rules on your data and interact with the data in real time. With the wasm rules you can do things like detect and mask PII, etc. The node-sdk is here: https://github.com/streamdal/node-sdk. There are some minimal examples of these pipelines in the readme and examples/sandbox directories.

Re: Show HN: Streamdal – an open-source tail -f for your data

#12

Post by openquery » Tue, Oct 31, 2023, 6:55 PM UTC

Congrats on the launch! Been waiting for this.

Re: Show HN: Streamdal – an open-source tail -f for your data

#13

Post by ollybee » Tue, Oct 31, 2023, 7:44 PM UTC

Is there not a literal "tail -f" type client to send data from log files?

Re: Show HN: Streamdal – an open-source tail -f for your data

#14

Post by uzarubin » Tue, Oct 31, 2023, 7:52 PM UTC

ollybee wrote: post #13

Is there not a literal "tail -f" type client to send data from log files?

We live more on the preventive side where we tap into the real-time data flow before it reaches logs or data stores. Therefore, enabling you to be preventive vs reactive.

Re: Show HN: Streamdal – an open-source tail -f for your data

#15

Post by bradbrewer » Tue, Oct 31, 2023, 8:07 PM UTC

I'd like to use it but I have strict requirements for data not leaving our network

Re: Show HN: Streamdal – an open-source tail -f for your data

#16

Post by dsies » Tue, Oct 31, 2023, 8:17 PM UTC

bradbrewer wrote: post #15

I'd like to use it but I have strict requirements for data not leaving our network

Good news then :) Everything stays on your network. Actually, in most situations - everything stays completely client-side. Because the rules that the client executes are Wasm modules, all data inspections and transformations occur in the client itself.

There is a server component (that you host) - but it is only used for pushing rules/Wasm down to the SDKs and for facilitating tail - that's it.

Re: Show HN: Streamdal – an open-source tail -f for your data

#17

Post by tim-- » Tue, Oct 31, 2023, 9:41 PM UTC

That console UI looks beautiful!

Re: Show HN: Streamdal – an open-source tail -f for your data

#18

Post by uzarubin » Tue, Oct 31, 2023, 10:19 PM UTC

tim-- wrote: post #17

That console UI looks beautiful!

Thanks Tim! We're working toward having visually appealing UI, especially for open source!

Re: Show HN: Streamdal – an open-source tail -f for your data

#19

Post by tfsh » Tue, Oct 31, 2023, 11:12 PM UTC

This is awesome, the UI looks beautiful.

I've noticed you've provided Go, Python, and Node SDKs. What's the general tech stack for these? I assume your usage of Protobufs is for a consistent schemas between languages?

I ask because I'm curious as to how much work it is to define new SDKs for other languages, as I'd love a Java implementation - Ideally the SDK should be a pretty thin wrapper, simply calling the gRPC service with some minimal error handling, is this the case?

Re: Show HN: Streamdal – an open-source tail -f for your data

#20

Post by a1369209993 » Tue, Oct 31, 2023, 11:17 PM UTC

Obligatory: We already have a open-source tail -f; it's called `tail -f`. (Kind of /s, but much like the infamous dropbox comment[0], it sucks that there exists a problem for this to solve in the first place.)

0: https://news.ycombinator.com/item?id=9224