My Salary Progression in Tech

Sorry if this is too much text, but I feel like I have a decent sense about this, since I've been interviewing recently. Of course, we all live in bubbles, and there may be some tech bubbles extremely different from mine.

Your tech stack in a distributed system will often be diverse and wide-ranging. In my opinion, no specific combination of tools is very important for hireability, though commonly recurring characters will certainly help your resume.

I write or work on lots of small services, mostly in Java, some Scala, occasionally Go or Python. The JVM is probably the most popular programming platform for this domain but some companies do without it entirely (Go, Python, Node.js are all somewhat popular).

Most services run in Docker containers on AWS. I feel like almost everyone is in the cloud these days.

There are lots of REST APIs, so you'll want a good handle on HTTP. In my work, even more than HTTP, I've dealt with streaming/realtime data coming over Apache Kafka or AWS Kinesis or AWS SQS. Lots of people use RabbitMQ, but I never have.

And I interface with lots of different data stores - Postgres, S3, Dynamodb, Elasticsearch, Cassandra, Redis. And Zookeeper, more for coordination than storage. Zookeeper is fun, but very niche.

In terms of building a resume to get hired in this domain, these are some things you want:

- Be able to work in a few languages and with several data stores. Be able to talk about which kind of DB is appropriate for what use.

- Demonstrate proficiency in concurrent programming and asynchrony via multithreading/futures/promises/channels or whatever the relevant constructs on your platform of choice are.

- Have used some kind of streaming solution or message queue (kafka is neat and widely used, if don't already have one in mind)

- Have experience with provisioning and orchestration on a cloud platform (AWS, GCP, Azure,...)

- Be able to talk about tools and techniques for monitoring, alerting, and tracing in systems where a request or a stream element must traverse many services.

- Be able to talk about typical distributed-systems challenges at a conceptual level:

   * Why and how to partition your data, both in a stream and at rest

   * How to avoid or resolve consistency issues, when and how to trade consistency for availability, coordinations strategies, CRDTs

   * At-least-once processing, at-most-once, exactly-once, idempotency, why you would want each - how and when you can achieve each.

As a side note, I feel like, once you've built a decent distributed system or application, it's easy to make it look good on a resume. Either it's a high-throughput system and you can say you processed X thousands of requests per second with no downtime for a year, or else it's low throughput and you can double down on the fault-tolerance bit "even when we deliberately crashed half of the servers, the service remained available and end-to-end latency was only reduced by X amount". Of course, you need measure some stuff in order to have numbers to brag about. But hopefully you're collecting those metrics anyway.

Hey Retra, I looked at your comment history, to see why you'd possibly be saying this.

I've noticed two type of comments that you made:

1. Comments that have substantial reasoning, mostly technical but not always. These seemed to be appreciated.

2. Short comments that seemed to attack the person or a particular concept. These short comments did not have any reasoning from your side. Also, most of these comments are non-technical.

My first thought was: it seems that you have a lot of technical knowledge, and people appreciate this. I know I do.

Two observations on the category 2 comments you make:

1. In the comments that I skimmed, no one seemed to tell you to read the guidelines. I'll post the link [1]. Saying "then get cancer and die young" is an offensive thing to say because you're attacking the person. For some people this is obvious, but based on your comment history, I doubt whether this is obvious to you. Attacking a person is not a civil thing to do, not a nice thing to do (unless it's meant in a constructive fashion, even then it'd be debatably constructive). Regardless of that, it is in violation of the guidelines. As a technical person, I hope you'd appreciate that I'm pointing to a source appointed by Y Combinator. They make the rules on how to behave here, so then behave like that, the guidelines are reasonable. You seem to do so in most of your comments.

2. In most cases, short comments don't work quite well on Hacker News (many counter examples exist, I'm describing a trend which I observe). This is not because HN'ers dislike discussion, they dislike a discussion that does not have scientific evidence (again, a trend), most already have a problem when someone says "based on my experience". Do with this observation what you will, it's just my observation which is biased as well.

I hope these two points will help you to improve your comments like these ones, because it is obvious that you have a lot of insightful things to offer. And HN would be a more fun place if we'd see more of that and less of "then get cancer and die young" type of comments.

If you want a private conversation about this, you can always email me (check my profile).

[1] https://news.ycombinator.com/newsguidelines.html -- read the section "In Comments"

Philadelphia to Seattle.

Philly Team Lead Architect @ 185k w2 contract with health insurance and a week of PTO

Seattle System Admin -> 115K (There was an intent to promote, but we didn't agree on my career path).

Yep, that's the whole problem with this "follow your passion" BS. I'm the opposite: I work in a tech job on something I really don't care much about, but the pay is excellent, and the stability is actually pretty good too. I work 8h a day, I have an easy commute, I don't stress about money. I have a pretty easy life that way. But it also feels pretty empty in some ways, but the way I see it I've removed what for me was a big stressor in my earlier life. Maybe later, after I've saved up a lot, I can think about doing something that brings me more fulfillment, but for now I just concentrate on enjoying my free time, and look at work as just that: work.

I also look at the janitors in my building when I leave work in the evening, and remind myself that they're not following their passion either. Most workers do not get to do anything that really excites them. So I count myself lucky that I can do something that doesn't ruin my health or make me hate life, and I can get paid well for it, so instead of worrying how to pay rent, I can worry about less urgent things like what to spend my weekend doing. Plus, with my free time (since I don't have a job that's overworking me, like some people), I'm free to pursue my other passions there.

I guess my pithy answer is I've never got tired of learning. Four straightforward steps that you've no doubt heard before, but I can confirm they do the trick:

Establish internal credibility. That is, the confidence to speak credibly on a given subject, be it a platform or tool, a technique, a concept, a prediction. Roll your sleeves up and get dirty coding. Even on-rails tutorials force your brain to make the connection between the concepts and the technical reality. Especially for Big Data and ML/DL workloads. Put this all on a public repo. Search GH for relevant repos, fork them, and code walk them - especially Python/R/Scala scripts. Compete on Kaggle, that's obvious. Answer questions on SO, and attempt to replicate problems you find on SO yourself - this yields a lot of benefits, as most SO questions are edge cases. Join the user communities and help forums for popular platforms and read the newest entries - again, you learn the most from others actually practicing in the space. And still an extremely effective technique is to just directly search for e.g. "Spark shuffle joins" on Google and look for real people with real opinions - that you can then test yourself in an environment. When you've already read the blog post or watched the YouTube video or taken the Udacity course the guy across the table is talking about - you really establish credibility.

Teach others what you know. True trial by fire; share what you've learned. You'll learn how to overprepare, communicate and speak plainly, connect with those different than you in personality, behavior and skill, read rooms, provide feedback, take criticism, and most importantly - you'll really, really learn the ins and outs of the material.

Read voraciously. I skim-consume about 25 books a month using my company's Safari subscription; have built up a pretty good "reading recommender" system of Twitter users, subreddits, aggregators, blogs, etc; And not just technical books - books on business strategy, behavioral economics and psych, industry-specific blogs and journals - if you can't explain what you're doing in terms of risk, dollars, or time, you're doing it wrong. (Also a good life pro tip: download every interesting thing you read into a single repo and put a crawler on top of them. Even if you don't know the answer, you know you've read it somewhere and can retrieve it.)

Learn from others. Reading is great, in-person is better, mentoring is the best. Reach out to everyone in the field. Ask them (like you just did to me) what they're reading, what they did, ask follow up questions, share what you're doing with them. I find keeping a list of "questions you'd like to ask" while you're working through material (especially on industry/business) to be invaluable when you meet someone who can potentially answer them.

These days I spent probably 15-20 hours a week just reading, learning, doing, in prep for the 20-30 hours a week I spend educating, communicating, and (hopefully) selling.

One thing to add: I usually am focusing on 2 things at a time, no more, no less. If you can combine them, even better - learn Kafka for streaming and Grafana to visualize it; learn Azure SQL DW or Redshift for distributed DW and Spark for processing; etc.

My real evolution came when I realized I was happy being the dumbest person in the room, and that curiosity is a superpower.