Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gremlin Free – Run chaos experiments to prevent outages

gremlin.com

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Gremlin Free – Run chaos experiments to prevent outages

#31

Post by espeed »

vorg wrote: post #23
espeed wrote: post #19

Earlier quoted context omitted.

I was confused when I first saw the project, cartoon logo and "Gremlin Free " name, and I'm intimately familiar with both the Apache TinkerPop Gremlin open-source project, its third-party libraries, and the extensive ASF legal process we went through registering and identifying the use of marks. Read up on your trademark, IP and copyright law. The use of similar marks is not permitted when there is potential for conf…

> the names of all Apache® projects, software products, and their logos are trademarks owned by the Apache Software Foundation on behalf of our project communities. Note that while some Apache project names and logos are registered in the US and various countries, even unregistered names and logos are still trademarks of the ASF and should be treated with respect. Only trademarks registered with various legal jurisdi…

Gremlin is the original/previous moniker and has been since 2009. Apache TinkerPop was added upon joining Apache in 2016.

Re: Gremlin Free – Run chaos experiments to prevent outages

#32

Post by perfmode »

Negitivefrags wrote: post #28

So here is what I don't get about this stuff. What happens to the in-flight requests? Don't a few users run into random errors whenever a host is killed unexpectedly? You could have your loadbalancer retry everything that fails, but then wouldn't every single request in your app have to be idempotent?

Server crashes happen. This forces you to deal with them instead of pretending they won’t.

Re: Gremlin Free – Run chaos experiments to prevent outages

#33

Post by Negitivefrags »

perfmode wrote: post #32
Negitivefrags wrote: post #28

So here is what I don't get about this stuff. What happens to the in-flight requests? Don't a few users run into random errors whenever a host is killed unexpectedly? You could have your loadbalancer retry everything that fails, but then wouldn't every single request in your app have to be idempotent?

Server crashes happen. This forces you to deal with them instead of pretending they won’t.

Well yes, but I would suggest that they are uncommon enough that a few requests failing isn't a problem when those happen.

It's an entirely different story when you are killing processes constantly.

Re: Gremlin Free – Run chaos experiments to prevent outages

#34

Post by big_whoop »

It's reached a point where I actually want outages, because I just don't fucking care anymore.

Burn everything to the ground. None of the perpetual availability is actually productive. Most of these websites are bullshit. If an outage occurs, five nines out of ten, the lost money is hypothetical, not yours, and founded on a dubious hypothosis of how advertising hypotizes and subliminally influences all the mindless simpletons we presume our users to be.

Re: Gremlin Free – Run chaos experiments to prevent outages

#35

Post by Negitivefrags »

LoSboccacc wrote: post #30
Negitivefrags wrote: post #28

So here is what I don't get about this stuff. What happens to the in-flight requests? Don't a few users run into random errors whenever a host is killed unexpectedly? You could have your loadbalancer retry everything that fails, but then wouldn't every single request in your app have to be idempotent?

well for example in our systems all api calls only moves from a know state to another known state and any call failure redirects the client/user to the dashboard trough an error handler so they have to reload the last good state saved on the database. not perfect, but having a server crash is not much different than having a connection reset by a wifi status change or an upload timing out due the mobile network going…

It sounds like you are saying "The in-flight requests fail" to me.

I really don't like the idea of saying that it's simply okay to give random users a bad user experience like that when you are actually killing servers yourself all the time.

Re: Gremlin Free – Run chaos experiments to prevent outages

#36

Post by flowardnut »

Negitivefrags wrote: post #28

So here is what I don't get about this stuff. What happens to the in-flight requests? Don't a few users run into random errors whenever a host is killed unexpectedly? You could have your loadbalancer retry everything that fails, but then wouldn't every single request in your app have to be idempotent?

idempotent requests, stateless services, etc are all parts of a fault tolerant system.

your service has a few ways to deal with a dependency going down -- maybe it's a retry, maybe it's opening a circuit breaker and returning a default payload instead of calling that service.

It really depends on what specifically the service is and what it's calling (so it's a very case by case issue).

One of the very neat features of istio is that you can do this tuning in real time -- spin up your services, simulate faults, and then test your service while tuning your retry logic to see what the best user experience is.

Re: Gremlin Free – Run chaos experiments to prevent outages

#37

Post by perfmode »

Negitivefrags wrote: post #33
perfmode wrote: post #32

Earlier quoted context omitted.

Server crashes happen. This forces you to deal with them instead of pretending they won’t.

Well yes, but I would suggest that they are uncommon enough that a few requests failing isn't a problem when those happen. It's an entirely different story when you are killing processes constantly.

It doesn’t _need_ to be constantly. The important part is that it is done _deliberately_ to understand what happens when failures occur.

Re: Gremlin Free – Run chaos experiments to prevent outages

#38

Post by dbaggerman »

Negitivefrags wrote: post #35
LoSboccacc wrote: post #30

Earlier quoted context omitted.

well for example in our systems all api calls only moves from a know state to another known state and any call failure redirects the client/user to the dashboard trough an error handler so they have to reload the last good state saved on the database. not perfect, but having a server crash is not much different than having a connection reset by a wifi status change or an upload timing out due the mobile network going…

It sounds like you are saying "The in-flight requests fail" to me. I really don't like the idea of saying that it's simply okay to give random users a bad user experience like that when you are actually killing servers yourself all the time.

It's a different approach to managing risk -- minimizing impact of failure rather than minimizing the likelihood of failure.

It's nice to know that you can kill a process and the only impact is that in-flight requests fail, rather than having a more significant outage if a process crashes and the failover doesn't work, or the process doesn't automatically restart, etc.

If you accept that requests will fail you can build retries into the system. It's a lot harder to make a system more resilient if you avoid testing the failure scenarios.

Re: Gremlin Free – Run chaos experiments to prevent outages

#39

Post by lklig »

dbaggerman wrote: post #38
Negitivefrags wrote: post #35

Earlier quoted context omitted.

It sounds like you are saying "The in-flight requests fail" to me. I really don't like the idea of saying that it's simply okay to give random users a bad user experience like that when you are actually killing servers yourself all the time.

It's a different approach to managing risk -- minimizing impact of failure rather than minimizing the likelihood of failure. It's nice to know that you can kill a process and the only impact is that in-flight requests fail, rather than having a more significant outage if a process crashes and the failover doesn't work, or the process doesn't automatically restart, etc. If you accept that requests will fail you can bu…

Exactly! Chaos engineering is all about thoughtfully planned out experiments, to observe what the user experience will be when something fails. Doing this on your own terms allows you to improve the experience so that your customers aren't affected.

You can decide what happens when an in-flight request is dropped, whether you hold onto the state somehow and retry or the client could fail gracefully with a relevant error message.

Re: Gremlin Free – Run chaos experiments to prevent outages

#40

Post by lklig »

isuckatcoding wrote: post #24

How do you prevent abuse of this tool?

Security is extremely important to us. Clients authenticate to our control plane either with a secret string or a certificate. Clients can be revoked at any point from our webapp and as well if the client loses communication to our control plane, any ongoing attack is halted.

Check out our security page for more: https://gremlin.com/security

Post reply on HN