Live data from Hacker News

Ask HN: How to transition from academic programming to software engineering?

news.ycombinator.com

1–10 of 105 posts

Ask HN: How to transition from academic programming to software engineering?

#1
I taught myself how to code and ended up doing a PhD in a computational discipline. Programming has been a big part of my life for at least the last decade, during which I've written code almost every day, but always by myself. After graduating I joined a medium-sized company (~10^2 developers) as a machine learning engineer and realized how much I don't know about software engineering. I feel very comfortable with programming in the small, but programming in the large still feels mostly opaque to me. Issues like testing / mocking, code review, management of dev / stage / prod workflows and, most importantly, the judgment / taste required to make maintainable changes to a million LOC repository, are areas where I can tell I need to improve.

Former academics who moved into software engineering, which resources did you find most useful as you made the transition? Python-specific or language-agnostic books would be most helpful, but all advice would be welcome.

Re: Ask HN: How to transition from academic programming to software engineering?

#2
I'm also self taught, have done a lot of research code and product code, have worked on 'production' teams for over ten years and have had several PhD students as contributing team members.

The number one thing you can do is read through other people's code. If your colleagues are very good then you will learn a lot and pick up good habits, if they are so-so then you will build your self-esteem and sharpen your critical thinking skills. Some developers are shifty, and others love to talk about what they are doing and share insights. Spend time with the latter type.

Don't try to be an expert at everything, most teams should have self-selected individuals that choose to specialize in different areas that the team depends on.

Re: Ask HN: How to transition from academic programming to software engineering?

#3
If you're smart enough to do a PhD, you're smart enough to figure out all of the scaling and operational bits. It's not rocket science. It's just operations.

Simply by asking relevant questions, practicing some things a bit, you'll get used to it.

And consider that it's different in every organization, and nobody does it really well frankly.

Given the pace of change, the varying technologies and flavour-of-the-month processes, it may always seem a little unwieldy an opaque: the feeling of 'not knowing everything' never goes away.

And I concur with the itronitron: read other people's code on the team, who are known to 'code well'.

It does not mean that algorithmically it's genius or even good - it just means that those are the styles/patterns that may be expected of you. It's like learning to say certain words a certain way. You'll get it soon and then forget you're doing it.

Don't fret you'll get all of this quickly.

Re: Ask HN: How to transition from academic programming to software engineering?

#4
Read "Software engineering" by Ian Sommerville. Any edition (maybe from 6 onwards, though they are slighty different.. pick latest u can get). Maybe skim/skip the (technical) parts u think u know, and read the rest. Most will not make sense initialy.. does not matter, keep reading. u need to get all that "uploaded" in brain in order to be able to grasp it one day.

It took me 10 years to be able to skip all the technicals. And another 10+ years to understand why u may ever need the rest..

for judgment etc... Maybe pick some big-enough open-source project in a domain u know well and follow it - how and when they do change what. Dont worry, it does take years to really form your own judgment.

btw u will need some philosophy/methodology/human-side too.. there's not many of it in the above book.

For more, see the recommended readings on www.svilendobrev.com/rabota/

have fun

Re: Ask HN: How to transition from academic programming to software engineering?

#6
Some general advice I've given multiple junior developers over the years, you probably aren't a junior but most likely applicable to the advice you are seeking. These were passed down to me by other developers. Other HN folk will have links to literature but hopefully my advice will give you a precursor.

* testing - write your functions small enough to be readable, but not so small their abstractions are meaningless (because you have to test them all)

* testing - don't reach into your code's modules and mock. Instead use dependency injection with non-testing defaults

* code review - It shouldn't be personal, if it is are you reading it wrong or are they attacking you personally?

* code review - when referencing style complaints ask for reference material. Don't get caught in cyclic-pedantic style war between lead devs.

* code - your code should be environment agnostic, if you have environment/context specific things to do, pass along a environment/configuration dict or make a global config singleton. As long as your code depends on that you can write code more discretely.

* code - personal preference but try to not nest your loops too deeply, when you can use itertools.

* code - if you can help it, try not to mutate dicts/objects in place while in a loop. Makes testing a difficult.

* code - exit early if possible, test for failures instead of nesting your entire function inside a single `if`. Helps identify the bad inputs faster as well.

Above all, remember code isn't perfect. It's a tool to get to an end goal. If you aren't solving for the end goal you aren't solving the right problem. At the end of the day, you are employed to build a product and that product needs to perform it's job. (that isn't a pass to write super shitty code)

edit: formatting

Re: Ask HN: How to transition from academic programming to software engineering?

#7
I have made the jump from writing academic code to working on a product where actual software engineering was encouraged. Although I did jump back to academia pretty quickly.

Hitchhiker's Guide to Python is a very good book (freely available online, or get a copy from O'Reilly); some of it may be obvious but some might not be.

It is true IMO that making your code testable will also make it better designed. It might even be worthwhile to do completely dogmatic test-driven development (i.e. always write tests first, then stub out everything with NotImplementedError, then write actual code until all tests pass) for a while to get used to it, and force yourself to become familiar with tools for dependency injection/mocks/etc.

This is complicated by the fact that unit-testing machine learning code can be unusually tricky; normal unit-testing practices and metrics (e.g. code coverage) may not be very effective.

Re: Ask HN: How to transition from academic programming to software engineering?

#8

If you're smart enough to do a PhD, you're smart enough to figure out all of the scaling and operational bits. It's not rocket science. It's just operations. Simply by asking relevant questions, practicing some things a bit, you'll get used to it. And consider that it's different in every organization, and nobody does it really well frankly. Given the pace of change, the varying technologies and flavour-of-the-month…

This. I transitioned from a (non-STEM) PhD and am now working as a "real" engineer. It is called a "practice" for a reason, you will get better with time as long as you are self-reflective about it.

One practical tip - take a look at Dan Bader's book if you are deep in Python, it has a lot of good stuff.

One philosophical tip - depending on your organization, remember that slow is fast in engineering. This is somewhat different from more academic computing environments (at least that I know of). So take the time to get it right and really deeply understand your solution.

Re: Ask HN: How to transition from academic programming to software engineering?

#9
Code Complete 2 is kinda the bible on disciplined software engineering practice. That's a good start.

Other than that - there is not that much theoretical basis. The core principle is that software engineering is about dealing with a situation where you have far too many variables to fit into a single persons working memory, and how to organize a group of people in a way that they can co-operate without turning the thing into a mess, really fast.

It's more about having a set of understood processes, rather than what those processes really are, so that people can communicate about and co-ordinate their work effectively. Of course the processes need to make sense, but there's no "silver bullet" process that either would fix everything, or, conversely not following would lead to the end of the world.

"Issues like testing / mocking"

Testing has sadly developed bookish dogma around itself. But it's extremely practical. The most important automated test is the high-level integration testing - will this and that work when the customer uses it.

Unit test are about creating enforceable rules to the production system, which makes thing break faster, and, hence, faster to fix.

You don't want someone else to accidentally to break your code - especially that kinda weird cornercase? Write a test - now the rule becomes enforced as a part of your domain model.

" code review,"

Same principle as in writing text. Having someone proofread the things you write generally improves the quality. No need to be dogmatic.

The second aspect is the zenlike increase in code quality. People know that their work will be looked at by someone, hence they have a higher intrinsic motivation not to fudge things.

"management of dev / stage / prod workflows"

The only thing that matters there is that there is one agreed process inhouse. Otherwise things turn really messy, real fast. It's kinda tricky to wrap ones head the first time around the ramifications of the chosen rules, so that's why there are lot of published ways of working .

"the judgment / taste required to make maintainable changes to a million LOC repository,"

"Working effectively with legacy code" by Michael C. Feathers is a good start. Now, if the corpus of code has a thorough integration and unit test suite, you can change things, and if you accidentally break something, the tests will tell you.

If there are no tests, then, better start writing them. You can't do any large scale modifications - especially to production code - without them.

Have some tool that automatically tells you the test coverage.

Re: Ask HN: How to transition from academic programming to software engineering?

#10
This really depends on what specifically you're struggling with, so going to take some shots in the dark:

* "Refactoring" by Martin Fowler would probably help with writing good quality code and doing code reviews (or understanding the reasons for changes requested in others' code reviews).

* In my experience, "academic" code tends to be far more prone to very long functions. Understanding the Single Responsibility principle was a very important part of the transition from academic scripts to software engineering for me. If you regularly write functions of 30+ lines, start looking at breaking these down better -- what are you actually doing in each chunk of code? Can it be broken down further.

* In Academia, building software is 90% coding. Now, reading code will be far more important, and you might even spend more time reading code than writing code. Relatedly, the readability of your code is now the most important thing to optimize for (sometimes even at the expense of computational efficiency, you should aim to reduce the number of developer-hours spent wherever possible). This means writing more readable code (good variable names, learning and following style guides and other conventions, following a process even if it feels like a waste of time), using better tooling, more focus on testing, more time documenting, and more time communicating with programmers than actual programming.

* At times this will be frustrating because you'll remember when you could just go into the zone for several days straight and produce something fairly significant. Remember that a lot of the stuff that feels like a waste of time isn't - it's necessary to get out of the local maximum of what was possible when you did everything yourself and could keep the entire project scope in your mental model at all times.

* Whenever you have written code and are 99% sure that it will run fine and not break any other parts of the system ("it was such a small change"), rather assume that it is very likely to break something else. I still find I constantly need to recalibrate my confidence that what I'm changing has a limited area of affect.

* Spend a decent amount of time getting more familiar with things like source control. Assuming you use git, go down this rabbit hole to read about the different git workflows and get as comfortable as possible with flows that your team uses, dealing with conflicts, reversing mistakes, etc.

* Most of what you specifically asked for can only be learned through experience. Experience is gained the most quickly by doing, even if you make mistakes. For software quality, the best (only?) way to learn is to have someone more experienced review your code. The more pedantic they are, the more you will learn. If it takes you 3-4 attempts to get a code review through, then you have team that will accelerate your learning. If you're getting mainly LGTMs, you might be super competent and had no need to ask this question, but more likely you need to try find people to push you harder to learn new standards.

Post reply on HN