Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Signadot (YC W20) – Lightweight Test Environments for Microservices

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 21 posts

Re: Launch HN: Signadot (YC W20) – Lightweight Test Environments for Microservices

#11

Post by volkk »

One thing that maybe is just wooshing over my head, but how does persistence fit into this, at least as best practices?

The way I understand this tool from the docs, is that a request is duplicated to go to the baseline service A as well as service A' -- so service A on `main` branch is actually serving the up to date code, while service A' has modifications that developers can quickly see either work or blow up. What happens if the change is a DB write change? Do both A and A' point to a primary prod DB and if A' changes something in that results in bugged data, wouldn't that screw up prod data? How do I go about accounting for that? Or am I just entirely misunderstanding the point of this tool?

EDIT: I think I just bumped into my own answer within the docs -- "Sandbox Resources." I see so far you have Mysql/Maria, SQS, and Rabbit plugins. What's next on the roadmap? Kafka/PgSQL soonish? :)

Re: Launch HN: Signadot (YC W20) – Lightweight Test Environments for Microservices

#12

Post by anirudhrx »

volkk wrote: post #11

One thing that maybe is just wooshing over my head, but how does persistence fit into this, at least as best practices? The way I understand this tool from the docs, is that a request is duplicated to go to the baseline service A as well as service A' -- so service A on `main` branch is actually serving the up to date code, while service A' has modifications that developers can quickly see either work or blow up. Wha…

I was just about to post this :)

Depending on the request context (typically a set of headers), a request will end up going to one of baseline service A or A’, but never both. Depending on the scenario being tested, you can choose to isolate additional stateful components. In the case of testing a change to the database itself (DDL, etc), you’re right in that the database must be isolated so that A’ does not talk to the same shared database as A. So, as you pointed out, typically this would involve using a resource plugin (https://docs.signadot.com/docs/using-resources-with-sandboxe...) to set up that new database shard (or table, or schema) that is isolated as part of the test sandbox environment, and then linking A’ to talk to that isolated instance instead.

Kafka is on our roadmap next! We're actually working right now on making custom resource plugins easy to write, because invariably with databases, there's a data seeding step that's pretty custom for users. Right now, for building a custom plugin, aside from writing the logic in the provision / deprovision scripts (for ex. https://github.com/signadot/plugins/tree/main/src/mariadb), there's a bit of docker image building and helm packaging needed that we're hoping to simplify.

Re: Launch HN: Signadot (YC W20) – Lightweight Test Environments for Microservices

#13

Post by wjossey »

anirudhrx wrote: post #9
wjossey wrote: post #7

As someone who has built out a similar internal tool, one of the things I'm excited to see someone do is the route propagation technique. It's something I've evaluated adding to our internal solution as we rolled out a service mesh, but ultimately we manage routing in a different way under the hood. Either way, this notion of slices of environments being deployed for testing, with "baseline" or fallback environments…

Thank you! We're really excited about covering the entire dev lifecycle using this approach. Re: routing, do you make use of a library that is directly included in each service? We've seen an approach in the wild where such logic is embedded into gRPC interceptors and HTTP middleware. Curious to hear how you thought about the choice between that and using the mesh.

Right now, we just require intermediate services to be spun up if you need to connect with a custom downstream service.

So, as an example, if I have three services where A talks to B, B talks to C, and I have a custom version of A & C that require testing E2E, we have to spin up B in the middle.

It definitely is a point of confusion for engineers who have to understand what intermediate services are in play when working on a frontend and some distant backend microservice. Fortunately, that intermediate layer tends to just always be the same two services, so folks learn pretty quickly. At scale, especially hyper-scale in a microservices architecture, this becomes untenable, and either automatic dependency discovery OR routing (like you designed) is the path forward.

Needless to say, it's a "niche" issue I think folks don't run into for a while in their setups; however, once you run into it, it's a PIA for those involved.

Re: Launch HN: Signadot (YC W20) – Lightweight Test Environments for Microservices

#14

Post by volkk »

anirudhrx wrote: post #12
volkk wrote: post #11

One thing that maybe is just wooshing over my head, but how does persistence fit into this, at least as best practices? The way I understand this tool from the docs, is that a request is duplicated to go to the baseline service A as well as service A' -- so service A on `main` branch is actually serving the up to date code, while service A' has modifications that developers can quickly see either work or blow up. Wha…

I was just about to post this :) Depending on the request context (typically a set of headers), a request will end up going to one of baseline service A or A’, but never both. Depending on the scenario being tested, you can choose to isolate additional stateful components. In the case of testing a change to the database itself (DDL, etc), you’re right in that the database must be isolated so that A’ does not talk to…

> a request will end up going to one of baseline service A or A’, but never both

that's interesting, i totally didn't gleam that. so X percentage of users are getting a potentially bugged experience? is this kind of like a more robust feature flagging tool? i.e "rollout this thing to 1% of people for X time, and then roll it back to control"

and if so, if you have 200 engineers all testing all sorts of stuff on prod users/data, the chances of buggy experience for small amounts of people increases? am i understanding that correctly?

Re: Launch HN: Signadot (YC W20) – Lightweight Test Environments for Microservices

#15

Post by anirudhrx »

volkk wrote: post #14
anirudhrx wrote: post #12

Earlier quoted context omitted.

I was just about to post this :) Depending on the request context (typically a set of headers), a request will end up going to one of baseline service A or A’, but never both. Depending on the scenario being tested, you can choose to isolate additional stateful components. In the case of testing a change to the database itself (DDL, etc), you’re right in that the database must be isolated so that A’ does not talk to…

> a request will end up going to one of baseline service A or A’, but never both that's interesting, i totally didn't gleam that. so X percentage of users are getting a potentially bugged experience? is this kind of like a more robust feature flagging tool? i.e "rollout this thing to 1% of people for X time, and then roll it back to control" and if so, if you have 200 engineers all testing all sorts of stuff on prod…

Ah, the primary use-case here is for developers to test microservices as changes are made to them - so one would typically be running this setup in pre-production / staging only and no prod user data would be involved. I would think of it as a way to run integration / e2e tests for each commit or pull request. Running these tests in the high fidelity test environment constructed this way reduces the bottlenecks on staging environments, and bugs are discovered / addressed sooner.

Re: Launch HN: Signadot (YC W20) – Lightweight Test Environments for Microservices

#16

Post by anirudhrx »

wjossey wrote: post #13
anirudhrx wrote: post #9

Earlier quoted context omitted.

Thank you! We're really excited about covering the entire dev lifecycle using this approach. Re: routing, do you make use of a library that is directly included in each service? We've seen an approach in the wild where such logic is embedded into gRPC interceptors and HTTP middleware. Curious to hear how you thought about the choice between that and using the mesh.

Right now, we just require intermediate services to be spun up if you need to connect with a custom downstream service. So, as an example, if I have three services where A talks to B, B talks to C, and I have a custom version of A & C that require testing E2E, we have to spin up B in the middle. It definitely is a point of confusion for engineers who have to understand what intermediate services are in play when work…

Makes sense. I have seen a couple of instances of this pattern - with service dependencies being stitched together using configuration. Thanks for sharing!

Re: Launch HN: Signadot (YC W20) – Lightweight Test Environments for Microservices

#17

Post by volkk »

anirudhrx wrote: post #15
volkk wrote: post #14

Earlier quoted context omitted.

> a request will end up going to one of baseline service A or A’, but never both that's interesting, i totally didn't gleam that. so X percentage of users are getting a potentially bugged experience? is this kind of like a more robust feature flagging tool? i.e "rollout this thing to 1% of people for X time, and then roll it back to control" and if so, if you have 200 engineers all testing all sorts of stuff on prod…

Ah, the primary use-case here is for developers to test microservices as changes are made to them - so one would typically be running this setup in pre-production / staging only and no prod user data would be involved. I would think of it as a way to run integration / e2e tests for each commit or pull request. Running these tests in the high fidelity test environment constructed this way reduces the bottlenecks on st…

AH! derp. thank you that makes total sense :) -- seemingly awesome tool, will certainly look into this. thanks for the quick responses
Post reply on HN