Ask HN: How can I grow as an engineer without good seniors to learn from?

First off, don't get too distracted by what software engineers are talking about on the internet. Despite the enormous overlap in what kinds of technologies you use at work, the jobs are not the same, and the details of what constitutes a job well done are often not the same, either.

In particular: data engineers' main task is to produce clean data. Few people care about the cleanliness of the code used to produce it. They care about whether or not the data makes sense, whether or not it's accurate, how well it's documented, and whether or not others can understand the ETL pipelines' dependencies and assumptions so that people can predict how changes to business processes and IT systems will affect their business intelligence and data science efforts.

Cleanliness of code is, at best, a secondary concern. Especially if you're mostly working solo. The code for managing data pipelines is often relatively small and tightly scoped, to the point where even a messy ETL implementation can still be easier to understand than very clean application source code. Or at least, that's been my experience as someone who's worked as both a software engineer and a data engineer.

The biggest thing is, again, just to make sure the documentation is good. In one data engineering role I found a bunch of things in the data engineering pipeline that I was pretty sure were junk that we could get rid of to simplify the whole thing. Doing so would have reduced the code for the pipeline in question by ~25%, with comparable reductions in run time and server load. But I couldn't be sure because the original author left no documentation to indicate why those bits were there, so we ultimately decided to leave it there for the sake of safety. The original author was a software engineer, not a data engineer, so the code in question was incredibly clean and readable. I had no trouble figuring out how it was doing what it was doing. But what I really needed to know was what it was doing and why.

I'd also encourage caution about adopting fancy tech stack. It's amazing what you can accomplish with just a SQL database (or Parquet files) and some Python scripts. Data engineering is a space with a lot lot lot of vendors and open source projects selling solutions to problems you might not even have. And every additional solution you bring on is another piece of technology to understand, which can quickly get overwhelming if you're on a small team. Especially be careful about committing to the open source versions of technologies like Apache Spark. I can tell you from experience that, if you can't name the specific person who will personally be responsible for supporting and operating the technology, that person will be you, you will end up spending a lot of time doing it, and it will probably not be a use of time that the person who does your performance reviews will consider valuable. Especially if "I had to spend a bunch of time diagnosing Spark executor failures" becomes the reason why you couldn't get other things done.

You're conflating manipulating with engineering. Communicating with people can be part of a job but it isn't engineering.

  engineering
  noun

    the art or science of making practical application of the knowledge of pure sciences, as physics or chemistry, as in the construction of engines, bridges, buildings, mines, ships, and chemical plants.

 Digital Technology. the art or process of designing and programming computer systems:

  computer engineering;
  software engineering.

https://www.dictionary.com/browse/engineering

Read voraciously. A membership in IEEE and ACM journals will keep your stack full although it can be a pain sorting through all of the wheat. Keep track of books that interest you in subject areas that seem challenging and interesting: if you're interested in data that will definitely be in the realm of statistics, combinatorics, linear algebra.

Do exercises. Take 30-40 mins a day to do practice problems. Use a site like codewars, leetcode, even past advent of code challenges. Do the exercises in the text books you read.

Write. Write about what you learn. Use your writing to explain it to someone who is encountering the subject area for the first time. Doesn't matter what you write it on: a paper notebook is just as serviceable as a blog. The act of reinforcing what you discover by writing it down helps to reinforce it and challenges your knowledge by explaining it to someone else. Whatever you do make sure that the act of writing works for you: it must be seamless and flow easily.

Don't over-engineer the perfect writing system that use a ton of frameworks and require maintaining deployment scripts. You will avoid writing if it is hard to do. Writing itself is hard enough and that's what you need to focus on. If you want to go the blog route make sure it's no harder than opening up a text file and running a simple script when you're done.

Lastly, find a mentor. It can be someone outside of work -- and in your case sounds like it will have to be. Contribute to an open source project you use regularly. Go to meetups in your local area or online. Find someone you can talk shop with. And when the opportunity arises ask if they'd be interested in meeting up once in a while to review your work or talk about what you've discovered.

I have found myself in a similar position between independence and isolation in my role, though it's not as high-profile.

I am interested in the other answers here and will add my two cents.

Recent hiring: KEEP IT STUPID-SIMPLE. Being new at the company, less experienced, and empowered, you will probably be tempted by lots of "solutions," whether they're products, frameworks, management buzzwords, etc. Most of them are not designed for a one-man-team. Make sure your solutions actually help YOU.

Non-technical boss: AGREE ON METRICS. Your boss/es will measure you eventually, whether either of you are aware of it or not. It may be better to discuss some metrics while you have a chance to do something about it. At the end of the day, it's almost always about money (and sometimes ego), especially with cantankerous O&G types. Just figure out how you can show you enable, save, or make money within the system.

Feedback and standards: YOUR BEST FEEDBACK AND STANDARDS WILL COME FROM YOU. You are in a position to act on your best judgement, and then own your mistakes. Yes, you can feel like a fish-out-of-water a lot of the time. I have tried to find a balance between pushing myself to learn personal skills on projects, innovating for the company, just getting things "done," and keeping myself sane. Your peers, "clients", and supervisors are probably not used to (or aware of) "best practices", which is a double-edged sword. They might expect you to deliver things you know could be better, but you also have a ton of power to build a solid environment for yourself. Think of progress like driving a big boat, not a racecar.

I've been in similar positions throughout most of my career (early promotion above my current skill level with no mentor). If I look back, I think only my first internship ever did I have a good mentor to learn from. So I think I know how you feel.

There's lots of good advice in this thread, but I think the most important thing to keep in mind is to know that you will make mistakes and do stupid stuff. Always be evaluating yourself and use that to adjust how you do the next thing.

Anecdote: in my first real job after school, I was essentially give a one person R&D project that reported directly to the CTO to design a framework for "the future of our data analytics". I ended up designing this crazy over-engineered highly abstracted OO framework that was a nightmare to document and use. But since it was fast and met the requirements, the CTO pushed it through as the framework for the rest of the engineering team.

I spent the next couple of years justifying it and supporting it. Once I recognized that I made many dumb (well maybe not dumb, but novice) decisions. I used that as a learning experience for my next project.

I've had many of those kinds of learning experiences over the years, and as long as I keep self-examining and being self-critical, I learn from them and don't make the same mistake twice. Now 25-ish years later, I have a long list of internalized knowledge of what not to do.

I guess that's a long way of saying, don't be afraid to make mistakes, you will. Just pay attention to them and learn from them.