241–250 of 261 posts
The first ten years of my career, I worked with distributed systems built on this stack: C++, Oracle, Unix (and to some extent, MFC and Qt). There were hundreds of instances of dozens of different type of processes (we would now call these microservices) connected via TCP links, running on hundreds of servers. I seldom found this exhausting. The second ten years of my career, I worked with (and continue to work on) m…
Earlier quoted context omitted.
I would respectfully say that you are wrong. I speak from experience. At Netflix we tried to hire for around the clock coverage. But what ended up working much better was taking that same team and having each person on call for a week at a time, all based in Pacific Time. Yes, you would get calls at 2am, sometimes multiple days in a row. But you were only on call once every six to eight weeks, and we scheduled out we…
> you were highly incentivized (and had the time) to build tools or submit patches to fix the problems that woke you at 2am. Ah, so you worked on a team where the SRE needs were prioritized over the feature requests? Because in most companies where I've worked, Product + Customer Service + Sales + Marketing + Executives don't really have time or patience for the engineers to get their diamond polishing cloths out. Th…
Should this decision happen at 2am? Can it wait until 10am?
Earlier quoted context omitted.
> But who are the customers? Business, engineering, or finance? :) The business's customers. The ones who pay your company so they can pay you, and your reason for having a job at all. > Why should internally developed services be any different? Because they're your core competency and you have control over it. If you could page the Jenkins developers you probably wouldn't hesitate to do it, because you'll get better…
> As an operator the last thing I want is devs to throw whatever they write over the wall and then say "not my problem anymore!", and have to rely on getting retrained every time the code changes. And as a dev I wouldn't want to be in an organization that accepts sloppy developers who aren't responsible for building solid code that can run under adverse conditions and who don't get to experience the issues in product…
Look at all that on-prem software you just mentioned. The developers of every one of those complain that they need better bug reports, and the people who operate them complain they need better documentation. Things would be much better if those devs worked directly for every company that uses them, and in fact in a lot of cases one of the contributors is an operator at a company. Why do you think companies like to hire open source devs? To get better access to someone who knows the codebase!
It's far better if the operator is the developer. Sometimes we live with that not being the case because the software is made by others. But when given the choice, I will always opt for the dev running the software themselves.
Earlier quoted context omitted.
Pre covid I would have laughed at this. But now, no one knows what a user story should be unless you can reas it off jira and there are no backups of course.
Gives me a fun idea: a program that randomly deletes items out of your backlog.
Earlier quoted context omitted.
Sure: the network request time of a person making a request over the open internet is going to be an order of magnitude longer than a DB lookup (in the right style, with a reverse-index) on the scale of data this person is describing. So making the lookup 10x faster saves you...1% of the request latency. And at the qps they've described, it's not a throughput issue either. So I'm pretty confident in saying that this…
This reads to me as if you have never really used mmap in a dedicated C/C++ application. Just to give you a data point, looking up one word_id in the LUT and reading 20 document_ids from it takes on average 0.0000015 ms. So if that alternative database takes on average 0.1ms per index read, then it's starting out roughly 65000x slower. "than a DB lookup (in the right style, with a reverse-index)" Unless, of course, y…
What kind of servers are you running? What's your max QPS?
The fact is with your mmap impl. you probably use ram + virtual memory, and have more ram than needed to compensate for the fact that you don't keep the most used keys in memory, which a DB will do for you.
Point is if you have petabytes of data and access patterns only mean you access a subset of it, even Mongo might be cheaper to run.
Earlier quoted context omitted.
I've long wondered what I might be able to keep an eye out for during onboarding/transfer that would help me tell overstuffed kitchens apart from optimally-calibrated engineering caves from a distance. I'm also admittedly extremely curious what (broadly) had 1000 (and 500) engineers dedicated to it, when arguably only 50 were needed. Abstractly speaking that sounds a lot like coordinational/planning micromanagement,…
> what I might be able to keep an eye out for during onboarding/transfer that would help me tell overstuffed kitchens apart from optimally-calibrated engineering caves from a distance The biggest thing I've been able to correlate are command styles: imperative vs declarative. I.e. is management used to telling engineering how to do the work? Or communicating a desired end result and letting engineering figure it out?…
There really is an art to scaling problems to humans so the individual work (across management and engineering) falls within the sweet spot of cognitive saturation. TIL yet another dimension that can go sideways.
The signal to noise ratio is very appreciated.
Earlier quoted context omitted.
I've long wondered what I might be able to keep an eye out for during onboarding/transfer that would help me tell overstuffed kitchens apart from optimally-calibrated engineering caves from a distance. I'm also admittedly extremely curious what (broadly) had 1000 (and 500) engineers dedicated to it, when arguably only 50 were needed. Abstractly speaking that sounds a lot like coordinational/planning micromanagement,…
> a lot like coordinational/planning micromanagement, where the manglement had final say on how much effort needed to be expended where instead of allowing engineering to own the resource allocation process Yep, that's a fair assessment! The 1000+ one was an ERP for mid-large businesses. They had 10 or so flagship products (all acquired) and wanted to consolidate it all into a single one. The failure was more on tryi…
> trying to join the 10 teams together
oh no
(insert https://webcomicname.com/ here)
> rather than picking a solid foundation that they already owned and handpicking what needed.
Mmmm.
I wonder if a close alternative (notwithstanding lack of context to optimally calibrate ideas off of) might have involved leaving all the engineers alone to compare notes for 6-12 months with the singular top-down goal of "decide what components and teams do what best." That could be interesting... but it leans very heavily on preexisting competence, initiative and proactivity (not to mention conflict resolution >:D), and is probably a bit spherical-cow...
> The 500+ was an online marketplace. They had that many people because that was a condition imposed by investors.
*Constructs getaway vehicle in spare time* AAAAAaaaaaa
Sad engineering face :> I don't think it's impossible to notice, but it's hard... you can ask during interviews about numbers of employees, what each one does, ask for examples of what each team does on a daily basis.
Noted. Thanks.
> Honestly 100, 500, 1000 people for a company is not really a lot, but 100, 500, 1000 for a single project is definitely a red flag for me now, and anyone trying to pull the "but think of the scale!!!" card is a bullshit artist.
That makes a lot of sense, and also filed away.
Also, I recently read this which resonates quite strongly with the economy-of-efficiency scale problem (which I totally agree with): https://rachelbythebay.com/w/2022/01/26/swcbbs/, and the update, https://rachelbythebay.com/w/2022/01/27/scale/
Earlier quoted context omitted.
Which FAANG is rewriting to another hip language and chasing trends (especially when it comes to infra services??)? I don't mean to be rude, but it doesn't sound like you are talking about any of the FAANGs, this sounds completely made up.
https://blog.pragmaticengineer.com/uber-app-rewrite-yolo/
Earlier quoted context omitted.
Sure: the network request time of a person making a request over the open internet is going to be an order of magnitude longer than a DB lookup (in the right style, with a reverse-index) on the scale of data this person is describing. So making the lookup 10x faster saves you...1% of the request latency. And at the qps they've described, it's not a throughput issue either. So I'm pretty confident in saying that this…
This reads to me as if you have never really used mmap in a dedicated C/C++ application. Just to give you a data point, looking up one word_id in the LUT and reading 20 document_ids from it takes on average 0.0000015 ms. So if that alternative database takes on average 0.1ms per index read, then it's starting out roughly 65000x slower. "than a DB lookup (in the right style, with a reverse-index)" Unless, of course, y…
Are...are you saying that you've purchased petabyte(s) of RAM, and that that multi-million dollar investment is somehow cheaper than...well really anything else?
> But before that your 10gbit/s node-to-node link will saturate. Oops.
Only if you're returning dense results, which it sounds like you aren't (and there are ways to address this anyhow), which is why I said the issue of saturating network before disk probably wasn't an issue for you ;)
Yup. Spent more than a decade doing it. Got so frustrated that I started a company to try abstract it all away for everyone else. It's called M3O https://m3o.com . Everyone ends up building the same thing over and over. A platform with APIs either built in house or an integration to external public APIs. If we reuse code, why not APIs. I should say, I've been a sysadmin, SRE, software engineer, open source creator, m…
>Consume public APIs as simpler programmable building blocks Is the 'r' in simpler there intentionally? In which way are the building blocks more simple than simple blocks?