Didn't know what NATS is, page doesn't explain it. Tried to click on the Docs link at the bottom of the page but that doesn't work. Stopped looking.
Stream Compression in NATS
11–20 of 24 posts
Re: Stream Compression in NATS
#12Didn't know what NATS is, page doesn't explain it. Tried to click on the Docs link at the bottom of the page but that doesn't work. Stopped looking.
I didn't stop looking and followed link to the version announcement, then went to the front page of nats.io from the top left of that page. And I also still have absolutely no fucking idea what any of this is.
if you've got a bunch of programs running that want to send messages to one another, in various paradigms and various languages, nats.io may well help you.
Re: Stream Compression in NATS
#13I was hoping the blog post would include a tentative roadmap for adding tiered storage.
Re: Stream Compression in NATS
#14Honestly, I'm more interested in auth callout, someone making a "how to use this stuff effectively" video and/or article, for dummies, ideally with keycloak. If you look at [0] how does this explain anything? Seems like I need to write a service utilizing NATS which talks to the OIDC server. NATS auth story is a complicated one, and now with auth callout it's even more complicated. What I also quite didn't succeed in…
> Seems like I need to write a service utilizing NATS which talks to the OIDC server.
Auth callout was designed to be a generic extension point to delegate authentication and generate dynamic a user JWT that NATS understands (permissions, limits, etc). It enables an arbitrary backend to be integrated with, not tied specifically to OIDC. But indeed, this requires implementing a service that does this integration.
> NATS auth story is a complicated one, and now with auth callout it's even more complicated.
There is a spectrum of auth options, starting with simple config-based, token or user/pass leading up to decentralized auth for use cases that need it. Auth callout is an opt-in thing, so it should only be adopted if it is truly necessary.
> you have to have dedicated functions which you would call in the request handler [...] instead of being able to define a chain of middleware functions
I don't quite understand this statement. Wrapping a NATS handler is the same approach as wrapping an HTTP handler (within the same client app). The function would take a handler and return a handler. There can be inspection of the message within that function and the choice of calling the next handler, responding early, doing some external call, or doing nothing.
If you want to distribute this middleware, then you would need to have separate, explicit subjects that each middleware would subscribe to and then publish to for the next element in the chain (for a choreography approach).
There is also the "message slip pattern" where the ingest component sets the path as metadata (e.g. headers) that each middleware component uses that to know (at request time) which subject to publish the result to next.
Re: Stream Compression in NATS
#15Honestly, I'm more interested in auth callout, someone making a "how to use this stuff effectively" video and/or article, for dummies, ideally with keycloak. If you look at [0] how does this explain anything? Seems like I need to write a service utilizing NATS which talks to the OIDC server. NATS auth story is a complicated one, and now with auth callout it's even more complicated. What I also quite didn't succeed in…
Same. I tried out some toy use-cases using nats.js over websocket a few months ago. The prospect of possibly being able to "directly" consume messaging or key/value store from the browser with only a thin gateway between was really interesting to me but I couldn't square up the NATS-internal JWT cookie thing with how you would handle auth in a traditional web-app (OAuth client on a gateway plus a session cookie). I f…
Auth callout is post-NATS client authentication, so it would not solve the "auth web flow" for authentication. Instead, the resulting token from that would be set as a cookie that then would be passed into the nats.ws client connection. The auth callout service would use that token to map to the concrete NATS user. The mechanism of doing that is up to the implementation. One option is to manage NATS claims into the OIDC provider (for the user authenticating) and then the auth service would decode that source JWT and extract the NATS claims and generate the NATS user JWT in the response.
Re: Stream Compression in NATS
#16Earlier quoted context omitted.
Same. I tried out some toy use-cases using nats.js over websocket a few months ago. The prospect of possibly being able to "directly" consume messaging or key/value store from the browser with only a thin gateway between was really interesting to me but I couldn't square up the NATS-internal JWT cookie thing with how you would handle auth in a traditional web-app (OAuth client on a gateway plus a session cookie). I f…
Auth callout is post-NATS client authentication, so it would not solve the "auth web flow" for authentication. Instead, the resulting token from that would be set as a cookie that then would be passed into the nats.ws client connection. The auth callout service would use that token to map to the concrete NATS user. The mechanism of doing that is up to the implementation. One option is to manage NATS claims into the O…
Thanks that observation is extremely helpful. If I have it down then the intended flow looks something like?
- Web client is directed to some token vending service. This service implement authn in a manner of its choosing (i.e. OAuth) then sets a NATS client JWT in the cookie per https://docs.nats.io/running-a-nats-service/configuration/se... - Nats.ws client connection provides cookie during connection to perform client auth - If further authz/fine-grained control is needed the auth callout mechanism can be used. This would have access to the provided cookie/token so any claims needed for access control could be stapled on during step one and used at this point?
For GPs original question -- I'm running a fairly old Keycloak version (v8) but it does appear to set a JWT in KEYCLOAK_IDENTITY and KEYCLOAK_IDENTITY_LEGACY.
Am I right in understanding that IFF the token is signed with Ed25519 and both sub and iss are an NKEY value this is sufficient for NATS to accept that cookie as a credential?
Re: Stream Compression in NATS
#17Earlier quoted context omitted.
Auth callout is post-NATS client authentication, so it would not solve the "auth web flow" for authentication. Instead, the resulting token from that would be set as a cookie that then would be passed into the nats.ws client connection. The auth callout service would use that token to map to the concrete NATS user. The mechanism of doing that is up to the implementation. One option is to manage NATS claims into the O…
Thanks that observation is extremely helpful. If I have it down then the intended flow looks something like? - Web client is directed to some token vending service. This service implement authn in a manner of its choosing (i.e. OAuth) then sets a NATS client JWT in the cookie per https://docs.nats.io/running-a-nats-service/configuration/se... - Nats.ws client connection provides cookie during connection to perform cl…
Yes that reads correct. The `sub` would a NATS user public nkey, the `iss` would be the NATS account public nkey (either the issuer nkey in config-mode or existing nkey in decentralized auth).
As long as it can verify the chain of trust for the user JWT that is returned, it should work.
The three schema types are shown here: https://docs.nats.io/running-a-nats-service/configuration/se...
auth request comes in -> generate user jwt, sign + encode -> respond with auth response.
As long as the necessary bits of the response and user JWT conform, it will work.
Re: Stream Compression in NATS
#18I was hoping the blog post would include a tentative roadmap for adding tiered storage.
This post is about a product for NATS.
I presume you are talking about the roadmap after the 2.10 release?
Re: Stream Compression in NATS
#19Honestly, I'm more interested in auth callout, someone making a "how to use this stuff effectively" video and/or article, for dummies, ideally with keycloak. If you look at [0] how does this explain anything? Seems like I need to write a service utilizing NATS which talks to the OIDC server. NATS auth story is a complicated one, and now with auth callout it's even more complicated. What I also quite didn't succeed in…
> Seems like I need to write a service utilizing NATS which talks to the OIDC server. Auth callout was designed to be a generic extension point to delegate authentication and generate dynamic a user JWT that NATS understands (permissions, limits, etc). It enables an arbitrary backend to be integrated with, not tied specifically to OIDC. But indeed, this requires implementing a service that does this integration. > NA…
Regarding middleware, last time I tried there was a lot of information to process.
Now I tried again and
func loggingMiddleware(next micro.Handler) micro.Handler {
return micro.HandlerFunc(func(req micro.Request) {
spew.Dump(req.Data())
next.Handle(req)
})
}
Regarding auth, yes, spectrum of options, but for "production" use it's very complicated and you don't want a shared key, you need the whole operator, account, user thing, and external auth aka auth callout. The documentation is lacking. nkey, xkey what? It's confusing is complicated. Managing auth is a pain, more than once I've been locked out of my test server because of how complicated auth is and had to wipe it. It's so complicated because you have have isolated "namespaces" or operator spaces? Naming an organization operator, or a solution or service group doesn't help. user is synonymous with client. All that you don't get from the documentation, you have to watch Youtube videos to understand it. The documentation is bad.
How do I consume a stream from the deno/ws client without the whole client app blocking? No idea. The code snippets don't help. All there is, is a pay for company that offers to host NATS clusters, which you can pay to ask. Slack is not a good medium.
I'm guessing that the documentation is bad so one has to pay for support. And that's a no-go for me. I get that you spent time and effort into it, I'm assuming you're one of the NATS people, but what good is it when I don't know how to effectively use the software? The natsbyexample page is also hard to digest.
You're in your own little bubble and expect people to be mind readers.
If it's too complicated people won't adopt it. I know I won't because it's too complicated.
I would've loved to build upon it, because the core functionality is nice and works, easy to grasp.
I've not seen a single Jetstream stream and consumer tutorial with the new API and nats.ws and Go. Old API sure, STAN sure. New API no. Some real world stuff, not just code demo snippets. Why not? Because it's complicated.
Auth callout, that was the big thing everyone was waiting for? I would work on actually solving problems instead of building components so I can use this extra piece of software who's documentation is so bad.
You know how many Kafka tutorials there are? A LOT.
I want to like it, it seems very promising, but it suffers from the "coder no likey documentation" illness.
Today I saw a Twitch stream of some Netflix dev, talking about DX, developer experience.
NATS' DX has a lot of room for improvement, to put it friendly.
Re: Stream Compression in NATS
#20Earlier quoted context omitted.
> Seems like I need to write a service utilizing NATS which talks to the OIDC server. Auth callout was designed to be a generic extension point to delegate authentication and generate dynamic a user JWT that NATS understands (permissions, limits, etc). It enables an arbitrary backend to be integrated with, not tied specifically to OIDC. But indeed, this requires implementing a service that does this integration. > NA…
Regarding middleware, last time I tried there was a lot of information to process. Now I tried again and func loggingMiddleware(next micro.Handler) micro.Handler { return micro.HandlerFunc(func(req micro.Request) { spew.Dump(req.Data()) next.Handle(req) }) } I'll remove the middleware part. EDIT: Seems I can't edit the post anymore. Regarding auth, yes, spectrum of options, but for "production" use it's very complica…
> You're in your own little bubble and expect people to be mind readers.
Out of curiosity, have you asked questions in Slack or Github? If so, and you had a bad experience with the interaction, I get the sentiment and would offer help. But this comment is not constructive without context.
> I'm guessing that the documentation is bad so one has to pay for support.
The NATS project has been open source for around 12 years, and part of the CNCF since 2018. This is an incorrect statement and very poor assumption to make because the documentation doesn't make sense for you.
I 100% agree it can be improved and we are working on a new docs site, but it is not quite ready.
In case its helpful, there is an increasing collection of examples on https://natsbyexample.com with new JetStream client examples among others. If you have specific requests, feel free to open issues in the corresponding repo: https://github.com/ConnectEverything/nats-by-example