Viewing profile — binarylogic

I agree with the framing. The goal isn't less data for its own sake. The goal is understanding your systems and being able to debug when things break. But here's the thing: most te…

Agree to an extent. There are absolutely unknown unknowns. But I think you'd be surprised how much data is obviously waste. Not the grey area, just pure garbage: health checks, deb…

Thank you! And you're right, it shouldn't cost that much. Financials are public for many of these vendors: 80%+ margins. The cost to value ratio has gotten way out of whack. But ev…

Thanks for the comment! Yes, I read that post. Great post. Feel free to reach out if you ever need help with Vector or have questions.

What you're describing is very real and it works to a degree. I've seen this same manual maintenance play out over and over for 10 years: cleaning dashboards, chasing engineers to …

You're right, it's not always binary. That's why we broke it down into categories: https://docs.usetero.com/data-quality/logs/malformed-data You'd be shocked how much obviously-saf…

The question is answered in the post: ~40% on average, sometimes higher. That's a real number from real customer data. But I'm an engineer at heart. I wanted this post to shed ligh…

My apologies, I fixed the link. So much for restructuring the docs the night before posting this. You can read more here: https://docs.usetero.com/data-quality/overview To loosely …

Yeah, it's funny, I never went down the regex rabbit hole until this, but I was blown away by Hyperscan/Vectorscan. It truly changes the game. Traditional wisdom tells you regex is…

100% accurate. It is very much political. I'd also add that the problem is perpetuated by a disconnection between engineers who produce the data and those who are responsible for p…

Thank you for the nice comment. I'm glad you enjoy Vector. I poured myself into that software for many years. I'm a bit bummed with its current trajectory, though. We hope to bring…

Hey Peter, I absolutely remember you! Thanks for the nice comment. And yes, data waste in this space is absurdly bad. I don't think people realize how bad it actually is. I estimat…

I spent a decade in observability. Built Vector, spent three years at Datadog. This is what I think is broken with observability and why.

Yeah, we're trying to solve for this over at https://vector.dev . We intend to decouple the pipeline from the method. As demonstrated by OT, this stuff changes, and you shouldn't h…

Hey Lemmox, good questions: 1) Not currently, but it's certainly possible. We don't foresee any major fundamental changes. The most likely change is around transforming. Specifical…

I'm biased because my team and I created Vector [0], but I'd highly recommend investing in a vendor-agnostic data collector to start. You can use this to collect your data and send…

Ah, I'm amazed that I did not know this. I just updated everything to use MiB. Thanks for pointing that out.

They do, we actually put together a correctness test for this behavior: https://github.com/timberio/vector-test-harness/tree/master/... Logstash is not graceful. Our testing shows …

Absolutely, this is likely the next integration we'll be working on. There were a few features schema-wise that we needed support before we started, but we're _very_ close. We'd lo…

Telegraf is nicely done. We spent a lot of time testing solutions in our test harness ( https://github.com/timberio/vector-test-harness ) and Telegraf was the most impressive of th…

It's very likely we're doing something wrong with this test, but after many hours of trying we couldn't get our simple test to pass for Logstash, even though it passed for others: …

This discussion is moot since Authlogic easily supports BCrypt: http://www.binarylogic.com/2008/11/22/storing-nuclear-launch...