Behaviours to avoid in a software architecture role

To be fair, xml-rpc truly is terrible. To start with, XML is extremely verbose and you add a lot of network overhead for that, plus a lot of parsing overhead, which makes rpcs significantly more expensive than they should be. Use https://capnproto.org/ instead. (Based on protobuff, which is an open sourced version of what Google uses internally.) As far as I'm concerned, the only valid reason to use xml-rpc is because you're interfacing with someone else's system and they have chosen to use it.

Moving on, here is an excellent example of an important architecture system that almost everyone gets wrong. In any distributed system you should transparently support having rpc calls carry an optional tracing_id that causes them to be traced. Which means that you log the details of the call with a tracing id, and cause all rpcs that get emitted from that one to carry the same flag. You then have a small fraction of your starting requests set that flag, and collect the logs up afterwards in another system so that you can, live, see everything that happened in a traced rpcs. To make this easy for the programmer, you build it in to the rpc library so that programmers don't even have to think about it.

You then flag a small random small fraction of rpcs at the source for tracing. This minimizes the overhead of the system. But now when there is a production problem that affects only a small percentage of RPCs you just look to see if you have a recent traced RPC that shows the issue, look at the trace, and problems 3 layers deep are instantly findable.

Very few distributed systems do this. But those that do quickly discover that it is a critical piece of functionality. This is part of the secret sauce that lets Google debug production problems at scale. But basically nobody else has gotten the memo, and no standard library will do this.

Now I don't know why they reinvented xml-rpc for themselves. But if they had that specific feature in it, I am going to say that it wasn't a ridiculous thing to do. And the reason why not becomes obvious the first time you try to debug an intermittent problem in your service that happens because in some other service a few calls away there is an issue that happens 1% of the time based on some detail of the requests that your service is making.

I did work at Google, and their engineering architecture is insanely good. Their product design, not so much. Both aspects are visible when you use their site.

I acknowledge that Google is able to hire incredible people. They also got a lucky break in hiring Jeff Dean early. However it is my considered belief that when you call Google an exception, you're reversing cause and effect. When people who come up with the architecture have to write code, work with their system, and see from experience what is wrong with it, they produce better architectures.

It is not whether coding produces more value than architecture. It is that continuing to code informs their architectural decisions.

For a similar idea in a very different industry, when Robert Townsend was CEO of Avis back in the 1960s he turned the company around. He made it profitable, and made it grow.

One of the things he did that he says made a huge difference was to make it a rule that everyone worked the rental desk. Didn't matter whether you were the CEO, VP, big manager or whatever. One day a month you stood behind the rental desk and had to deal with live customers. And having that regular experience meant that problems in the organization that would otherwise go missed became instantly visible. Just like how actually having to code and debug things in your architecture makes architecture problems visible that otherwise you'd discount. It made Robert Townsend a better CEO. The corresponding exercise would make you a better architect.

See https://hbr.org/2010/02/make-the-change-be-an-undercov for more.

>I acknowledge that Google is able to hire incredible people. They also got a lucky break in hiring Jeff Dean early. However it is my considered belief that when you call Google an exception, you're reversing cause and effect. When people who come up with the architecture have to write code, work with their system, and see from experience what is wrong with it, they produce better architectures.

Totally agree with this. I personally started with installing trumpet winsock and netscape navigator on win3.11. I did customer support while been developer. I went through most of positions that are usually part of software development org and dealt with with most aspects of software. And I consider myself lucky to been able to do so.

But the unfortunate truth it's that in majority of companies that develop software (lets get outside of silicone valley for a moment), developers tend to be very far removed from production environments and from clients. As example, I was hired to work on a new, from scratch product, in one of biggest suppliers of software for telecoms (8B in sales). Over there "core engineering" was tasked with developing "core functionality" and there were totally separate delivery organization that will take this code and adapt it to client needs and will write installers/deployment systems, figure out how it actually integrate, run, etc. Clients weren't accessible to core engineering and internal product management couldn't talk to them either.

Also, the overall approach was that it's perfectly normal that software will crash and leave everything in inconsistent mess, hence there were mandatory tooling to be developed for manual operation to "fix things".

I spent first few months arguing with my boss that it's not normal and that we must deliver software that can be installed and work without crashing left and right. I got to built first devops org in company, first private cloud lab, pretty much first in company integrated ci/cd, jira+confluence (because company used internally developed "agile" tools that were unusable. and i actually got visit from internal auditors questioning me why we spent money on something that already exists in company), lobby to create track for engineers to interact with clients so they will understand what is real operational environment of software they are writing and what are the real requirements so they will be able to do better job, get training for developers so they will learn that there is something outside of "enterprise java" (as anecdote, when i presented rabbitmq as part of architecture, i got asked to what java standard it corresponds and when i said to none, i was told that they don't really understand why would I use it). Project was amazing success and delivered more in less time than was common in the company. (manager got fast tracked to vp/svp/gen manager overseeing 2B in business within 4 years. I got relocation to states :).

my point i guess, it's that most of software development in most companies is disorganized mess with developers been completely detached from realities production environments. And when I am hired, pardon the pathos, my job is to bring order into the chaos or light into the darkness or whatever. In cases when I am out of my depth in some areas (i well aware about limitations of my knowledge) , i usually have enough pull to hire some domain specialists that I can outsource to them designs/implementation.

It seems to me that in most organisation, the equivalence to 'rental desk' is facing the customer, not having to write some code. So we should be taking developers/architects/etc. and bringing them to interact with end users.

In other words you're saying that "software architect" exists as a role in these organizations because bad organizational structures create problems that it is a band-aid for.

And I'm saying that the band-aid doesn't work that well, and you do better if you fix the organizational structure to not need it. And furthermore, as a developer, those organizations are no fun to be in and you can't pay me enough to want to deal with them. Particularly not when I have the option of finding better organizations to be in.

Which adds to the problem. When good developers choose not to be in your company because your company is a disaster for developers, you don't get good developers. Which reinforces the idea that developers aren't that good, and causes them to try to put guard-rails around developers to prevent the problems that bad ones cause. Thereby creating a negative reinforcement cycle.