Ask HN: I feel like an 'expert beginner' and I don't know how to get better

imho, if you work on an open source project and you find it (or your relation to it) being "without purpose," you're doing it wrong. You've picked the wrong open source project, the project is an undead zombie, whatever. If you do it right, it will most certainly (edit: "usually, at least") not feel "without purpose."

Perhaps consider looking at open source projects and your possible personal relation to any particular one from a kind of two-layer perspective:

1. General idea behind the project; its general goals; and how you feel about them. It's probably a fruitless thing to work on something you don't believe in. On the other hand, finding something that shares your values to a great extent can bring out actual passion.

2. This is (usually, sometimes) not enough: hence find particular lower-level goals (these are the "ideologicaltechnical interfaces" so to speak, or, purely technical particulars (think of some very particular problem or proposed enhancement on a bug tracker.)) In my experience, the lack of attainable (to you) lower level "what do i do" things (which you can use to deduce "how do i do", and then end up with a todo list of particulars, which always helps to curb general laziness (well, sometimes)) might run you into trouble (and maybe this was something the parent was actually commenting on, so we probably disagree on details only.) You have to also keep re-evaluating these lower-level particulars, and set out new ones, etc. (and this is not always easy, of course.)

A good combination of the two can be quite a powerful thing indeed. And if the project is lead by good people with lotsa experience, and if a wider community of smart folk is involved, you can learn a lot.

> My advise would be to look at your own network (friends, family, etc) and find those who are in business and ask them about their pain; and there is always some pain that a business has.

But I agree that this is probably also a great idea, and sometimes more practical, etc. (also, learning to translate business (and sometimes it is an actual business) needs into solutions / technical designs is a very practical skill indeed)

Do other people do this too? It would be interesting to hear how other's solve this problem?

However: it sounds like your interests are actually more in the realm of Computer Science than Programming, though: type theory, theory of computation, compilers, AI, etc. are less in the realm of programming and much more in the realm of computer science. You don't really need to know how an NFA or DFA works in most day to day programming, frankly. For learning Computer Science, I would suggest finding an advanced course with all of its material online (MIT OpenCourseware is a good place to start, or any of the other free online courses places), and working through a course on the topic. If it has prerequisites that you don't know any/most of the material from, then find a course on those. In my opinion, the other common options (reading through a textbook, reading papers, working on a project related and just learning what you need for that) all have flaws: textbooks are usually designed to accompany a course, which means they usually have much more than you need to learn a lot about the subject (and, in my experience, many are unbelievably boring and poorly written; if you want to go that route, make sure you find ones that are appropriate for self study). Reading papers is really interesting (and a lot more fun than reading textbooks), but without context or knowledge about the area, it's hard to evaluate the paper's meaningfulness/claims/etc., and hard to decide which papers are important to read. Working on just a project (e.x. just writing a compiler) leads to learning just enough to make that project, and not more. If you want to improve at programming: follow the advice about either making a personal project or working on an open source project. The fastest way to get better at programming is to program more.

Frankly: it sounds like you are decidedly not as much of an 'expert beginner' as you think you are. Familiarity with basic data structures+complexity, an understanding of theory, and an understanding of math through linear, ODEs, and discrete already puts you on a very firm grounding. (Outside of certain very specific parts of CS, you /do not/ need more math than that. If you're going into graphics research, knowing differential topology+geometry might be handy. I'm having a hard time thinking of things other than that, though.) From your description, you have more or less finished the 'beginner' stage, moved well beyond the 'novice' stage, and are moving towards becoming someone with a lot of advanced knowledge. Just knowing that you're interested in things like type theory+compilers/machine learning+expert systems+AI already puts you /way/ beyond being a beginner.

Shameless plug of someone else's stuff: if you're not sure where to start on the courses front, and want to start on the programming languages side of things (compilers, etc.), this course might be a good place to start: https://www.udacity.com/course/cs262 I haven't worked through this course in particular, and I don't know its exact difficulty, but I took undergraduate programming languages from this professor and he's an /amazing/ teacher.

Edit: don't know why I had two copies of that written, but now there should only be one. Also, cryptography is another part of CS that you need more math (in this case, abstract algebra). (More multiple copies? I seem to be pretty bad at this "say things once" thing, and need to make some preference changes.)

+ There's a traditional sequence in skilled trades of apprentice -> journeyman -> master. A contemporary trade probably throws student on the front end and perhaps renames apprentice to 'intern'.

What separates a student from an apprentice/intern is the type of problems they work on. The same idea distinguishes the journeyman from the apprentice except that the journeyman is expected to successfully solve the problems they work on. Etc for the master.

What I am getting at is that there is a range of expertise and that what marks someone as an expert is the sort of problems they solve. But it's critical to realize that the context in which they solve those problems matters. This week thousands of students will be writing quicksort code. In 1960 it was the stuff of ...well CA Hoare probably was an expert programmer in an absolute sense, and perhaps a new journeyman among those who were programmers. Here again, context matters when talking about expertise...the one-eyed man in the kingdom of the blind and all that.

+ It's not really clear what you mean by 'expertise'. An expert AI programmer is someone who has solved expert level problems - one in an academic setting earns a credential. In a vocational setting it's going to take several.

But in both settings the context is years, not hours per day.

+ Sure you're free to define what it means to be an expert. Regardless of what definition you choose, the question remains, what does an expert do that you aren't doing? This isn't a question of research. It's not a question of methodology. It's a question of personal opinion - that is, what activities would you have to do to meet your definition of an expert. Maybe that's working at Google. Maybe it's earning a phd. Maybe it's writing a replacement for HotSpot.

+ Once you know what sort of problems you need to solve to move toward being an expert, perhaps just trying to solve some of those problems is the place to start. I.e. what sort of problems does an apprentice or journeyman tackle?

Good luck.