Challenging projects every programmer should try (2019) • Hacker News Board

Re: Challenging projects every programmer should try (2019)

#231

Post by kragen » Tue, Dec 26, 2023, 1:01 PM UTC

fragmede wrote: post #215

vkazanov wrote: post #186

Earlier quoted context omitted.

A simple search engine is certainly doable from scratch in a matter of weeks, complete with most things expected from a search engine. Similar to compilers or tiny OSes, one can go as hardcore as necessary, or just stick to basic stuff. Of all the typical personal challenge style projects, databases are probable the hardest to build, and even that is not impossible.

making a database is easy. making a fast database is what's hard.

even implementing all of sql92 or datalog without concern for efficiency is fairly complex

Re: Challenging projects every programmer should try (2019)

#232

Post by wiseowise » Tue, Dec 26, 2023, 1:08 PM UTC

p-e-w wrote: post #210

wiseowise wrote: post #200

Earlier quoted context omitted.

> I have never once thought "man, that thing is slow". You’re either lying or incapable to detect typing latency. Try typing while CLion indexes stuff in background or open couple hundred thousand line file in VS Code with Vim plugin and let me know how it goes.

There's a difference between noticing latency and considering it a problem. My car doesn't go at the speed of sound. That doesn't mean I think "man, this car is slow" each time I drive it. But if the same car suddenly stops, and I have to open all the doors and close them again in order to keep going, that sure is a problem.

> There's a difference between noticing latency and considering it a problem.

Just because you don't consider it a problem, doesn't mean it's not a problem.

Weren't you the one lamenting about ML having potential to being 100-1000x faster if only there were real™ software engineers to implement them?

> My car doesn't go at the speed of sound. That doesn't mean I think "man, this car is slow" each time I drive it.

This isn't even remotely valid example. Put your car on track against faster cars and you'll quickly arrive to `man, this car is slow`.

Re: Challenging projects every programmer should try (2019)

#233

Post by wiseowise » Tue, Dec 26, 2023, 1:09 PM UTC

sfn42 wrote: post #216

wiseowise wrote: post #196

Earlier quoted context omitted.

Software engineering is whatever definition that allows me to procrastinate and sleep at night knowing I don’t need to learn more.

If you don't want to learn you chose the wrong profession.

Say that to people in this thread.

Re: Challenging projects every programmer should try (2019)

#234

Post by kragen » Tue, Dec 26, 2023, 1:11 PM UTC

p-e-w wrote: post #102

tehnub wrote: post #83

Earlier quoted context omitted.

This is like telling a student learning an instrument never to practice scales or études because there are other skills necessary for playing in an orchestra. Those other skills are important, but you have to develop your chops at some point, and being a better a programmer will certainly help with basically all those skills you listed anyway.

> This is like telling a student learning an instrument never to practice scales or études No, it's like recognizing that an orchestra conductor doesn't need to play every instrument (or even any instrument!) in order to make music. Or alternatively, that a violin player doesn't need to understand the physics of acoustic dispersion in order to be the best in the world at playing the violin.

> it's like recognizing that an orchestra conductor doesn't need to play (...) any instrument

i heard this before and it seemed dubious to me

on investigating, it turned out that there were literally zero famous orchestra conductors who don't know how to play any instruments. most of them know how to play numerous instruments and do so in their spare time, though there were one or two who had stopped. from my experience with musicians i think it's likely that they could play any conventional orchestra instrument after a short exploration period, even if they don't have documented histories of playing it

i suspect that this is true of non-famous orchestra conductors as well, but i could only find public information about the famous ones

of course that doesn't prove anything about programming, which is not the same skill as playing a musical instrument or conducting an orchestra, but it does show that you're constructing your arguments without much concern for veracity

the fact that in https://news.ycombinator.com/item?id=38769008 you claimed that web search engines are not built by people who work on string matching or distributed database transaction consistency further calls your credibility into question

Re: Challenging projects every programmer should try (2019)

#235

Post by hombre_fatal » Tue, Dec 26, 2023, 1:11 PM UTC

p-e-w wrote: post #24

While writing a text editor, a compiler, an operating system, or a raytracer might make you a better programmer, it won't make you a better software engineer. In fact, it might make you worse at software engineering, because it embodies the disastrous "Not Invented Here" doctrine. Hackers like to obsess about Big-O, data structures, HoTT, and other high-theory stuff, yet the following skills, essential for software e…

Honestly this just sounds like me when I first encountered coding puzzles like project euler / leetcode and interview challenge practice problems. I made up excuses like this because I simply didn't want to spend time on lower level problems and instead wanted to feel good sticking to my high level application comfort zone.

I eventually learned to like solving lower level problems and I'm much better off for it. And these days I can tell people who haven't paid their dues because the only solutions they ever come up with are slow quadratic ones no matter the occasion.

No, building your own text editor, compiler, OS, ray tracer is not going to make you a worse engineer nor keep you from learning how to google and evaluate libraries or think critically, nor are those things reserved for an engineer who isn't building those things.

Re: Challenging projects every programmer should try (2019)

#236

Post by jvanderbot » Tue, Dec 26, 2023, 1:18 PM UTC

p-e-w wrote: post #107

bobsomers wrote: post #103

Earlier quoted context omitted.

I don't understand why you think a search engine requires the use of "real engineering skills" like picking and choosing libraries and identifying which opportunities will yield fruitful optimization. Literally all of the listed projects, text editors, compilers, operating systems, and ray tracers, can exercise the exact same activities. I'm more inclined to think that your comment is really more revealing about what…

> Literally all of the listed projects, text editors, compilers, operating systems, and ray tracers, can exercise the exact same activities. In the linked article, these projects are all explicitly described as opportunities to learn about low-level stuff like how to efficiently store editable text. The difference with a web search engine is that nobody today can build such a thing completely from scratch, therefore…

If you want to build a compiler from scratch, you must first invent the universe. Peeling back abstractions to see how things could or should work is perfectly fine, even for professionals.

Case in point, I've spent a year excising bloated frameworks from my stack at work and replacing the few corners we needed from those frameworks with, e.g. 50 lines of curl calls. The C compiles instantly and is tailored for our tiny use case, produced much quicker delivery on our one related feature we wanted, and removed chains of dependencies.

Being reliant on far-too-abstract libraries and frameworks to do simple jobs is also a curse. But nobody at work has the experience to know that curl was sitting right there on our image available for our use. And nobody has that experience because nobody took the time to build something from low level libraries. Now we know how and can make an intelligent decision without defaulting in either direction because we were afraid to try.

Re: Challenging projects every programmer should try (2019)

#237

Post by kragen » Tue, Dec 26, 2023, 1:23 PM UTC

sfn42 wrote: post #142

bee_rider wrote: post #72

Earlier quoted context omitted.

I don’t really see how one could have any hope of performing engineering with any sort of rigor while throwing away big-O. Then again, big-O is useless in many cases because real computers have too many arbitrary performance thresholds. I suspect software engineering is impossible, or at least, nobody has made the model required to do it.

Going from O(n) to O(1) on an operation where n is 10 could be a performance downgrade. That doesn't mean it's useless, it just means there's more to it than the big O. Asymptotic complexity is about how things scale up. You should care about it when you're working with things that scale. Doesn't mean it's useless if your data is static, just means you need to understand when to go for the high scaling solution and w…

if your input data doesn't scale then either

① you're writing a real-time control program such as a motor controller and you care about not just the asymptotic complexity of your algorithms but their worst-case execution time, in microseconds; or

② you should just do the computation by hand with pencil and paper instead of writing and debugging a program to do it for you

at the point that the input data becomes too big for ② to be an appealing option, n is at least 100, which means you probably care a lot about whether your program is O(n) or O(n⁴), at least if you wrote it in something slow like python

maybe the time to start worrying about it is after you start the program debugged and running for the first time

Re: Challenging projects every programmer should try (2019)

#238

Post by flir » Tue, Dec 26, 2023, 1:24 PM UTC

auggierose wrote: post #141

p-e-w wrote: post #130

Earlier quoted context omitted.

> If you are only using ready-made building blocks, there is a lot of great software you just cannot write. And I think we all see that, because there isn't much great software around. I claim that the opposite is true: The reason there are so few software projects that are really great is because high-level skills, not low-level ones, are in short supply. Every CS graduate can implement Paxos. The number of people w…

I guess it all depends on your definition of low-level. High-level can also be low-level, just with different building blocks, if the building blocks are based on well-specified abstractions. I am actually currently writing a novel text editor, and I can guarantee you, my requirements mean that an AI cannot help me with that directly, because the underlying mechanisms I need don't exist yet. The AI is still helpful t…

If you feel like telling us about your novel approach to text editing, I'd like to hear it (or I'll just wait for the announcement post. That's cool too).

Re: Challenging projects every programmer should try (2019)

#239

Post by kragen » Tue, Dec 26, 2023, 1:27 PM UTC

p-e-w wrote: post #39

Turing_Machine wrote: post #34

Earlier quoted context omitted.

What if we don't want to be "software engineers"? I'm still not totally convinced that "software engineer" is even a thing , frankly.

Who do you think puts systems like a web search engine together? It's certainly not people who are lost in details like string matching or how ACID is implemented in distributed databases. Of course you need all these things in order for the search engine to work – but if you write them yourself, even if you think about them excessively, you will never finish. Software engineering is knowing that you don't need to kn…

> Who do you think puts systems like a web search engine together?

i've met a good fraction of the first 128 google employees, and this could not be farther from the truth:

> It's certainly not people who are lost in details like string matching or how ACID is implemented in distributed databases.

the people who thought those things were unimportant details were the ones who tried to compete against google and failed, like lycos, inktomi, and pets.com

like, check out the wikipedia article on udi manber, who is best known for spending 15 years 'lost' in string matching:

> In 2002, he joined Amazon.com, where he became "chief algorithms officer" and a vice president. (...) In 2004, Google promoted sponsored listings for its own recruiting whenever someone searched for his name on Google's search engine. In 2006, he was hired by Google as one of their vice presidents of engineering. (...) In October 2010, he was responsible for all the search products at Google.

the fact that in https://news.ycombinator.com/item?id=38769506 you claimed that orchestra conductors don't need to play any instrument further calls your credibility into question

Re: Challenging projects every programmer should try (2019)

#240

Post by phkahler » Tue, Dec 26, 2023, 1:32 PM UTC

p-e-w wrote: post #107

bobsomers wrote: post #103

Earlier quoted context omitted.

I don't understand why you think a search engine requires the use of "real engineering skills" like picking and choosing libraries and identifying which opportunities will yield fruitful optimization. Literally all of the listed projects, text editors, compilers, operating systems, and ray tracers, can exercise the exact same activities. I'm more inclined to think that your comment is really more revealing about what…

> Literally all of the listed projects, text editors, compilers, operating systems, and ray tracers, can exercise the exact same activities. In the linked article, these projects are all explicitly described as opportunities to learn about low-level stuff like how to efficiently store editable text. The difference with a web search engine is that nobody today can build such a thing completely from scratch, therefore…

>> The difference with a web search engine is that nobody today can build such a thing completely from scratch

Really? I think there are a couple out there. The main issue today is scale, but you could constrain that by limiting your crawler to a fixed set of sites.