More challenging projects every programmer should try • Hacker News Board

Re: More challenging projects every programmer should try

#121

Post by KMag » Mon, Dec 21, 2020, 7:01 AM UTC

pjc50 wrote: post #17

henning wrote: post #9

As someone who has played with writing trading bots but never traded them with real money, some advice: if your results seem too good to be true, they probably are. Your trading bot may be doing unrealistic things or its results may not be reliable if the following are true: - You are trading in a market with low liquidity or one that is controlled by a small number of market participants. I'm not an expert but I thi…

I worked on a unique HFT system which was capable of starting the response packet before the incoming packet's final byte had arrived. Negative latency. If it mispredicted the future, it would simply scramble the trailer and cause the packet to fail UDP checksum. It didn't make money in the real world because the quality of decision that could be made at that speed wasn't good enough.

To get such changes reliably with low latency, I presume you were modifying the DMA buffer, without any syscalls or other mechanisms to synchronize between the CPU and the NIC's processor(s), right? Did you just set up the race condition such that when you got bitten, it resulted in a failed checksum, or was the time window so large that in practice the race condition never bit you?

Re: More challenging projects every programmer should try

#122

Post by na85 » Mon, Dec 21, 2020, 7:18 AM UTC

macintux wrote: post #44

timvisee wrote: post #30

Fantastic list! By the way, adventofcode.com is currently ongoing. Though the challenges are easy compared to the projects in this list, I highly recommend it. It covers problems you might face in big projects. With these small puzzles it's easy to experiment. It prepares you for bigger things.

For better or worse, adventofcode relies heavily on mathematical literacy that I suspect is neither all that common among developers nor all that important to most programming jobs. Plus there’s no good way within the context of the puzzles to find out what mathematical trick you need if you don’t already know; you need to go find a virtual water cooler. I may simply be biased because each year it reveals how little…

Not only that but adventofcode seems to leverage purposely-obtuse input formatting as an artificial difficulty enhancement.

I stopped doing advent this year when I realized I was spending more time debugging my parsing than I was actually solving the problems. It's just not that fun for me to sometimes spend 2+ hours finding parse bugs before moving on to the actual puzzle.

Re: More challenging projects every programmer should try

#123

Post by fergie » Mon, Dec 21, 2020, 7:19 AM UTC

I would strongly recommend building something that you yourself think is cool, and not feeling that you have to conform to what other people tell you to do.

Re: More challenging projects every programmer should try

#124

Post by lordnacho » Mon, Dec 21, 2020, 7:25 AM UTC

IgorPartola wrote: post #106

atomicnumber3 wrote: post #21

Earlier quoted context omitted.

I worked in this industry. 2 more common issues: * Doing a latency-sensitive trade when you don't have good execution. It's easy to go wild in simulation and think you can flip in and out of positions. But if you're a retail trader (and in this context, by that I mean "not connected directly to the exchanges, at the minimum") * Not taking into account the impact of your own trading on markets. This is obviously impos…

Would you mind if I asked you a question since I have never worked in this industry but did play with crypto trading a while back. Before Mt Gox was shut down I was trading on a couple of tertiary small exchanges and at the time there was a lot of talk of arbitrage between different exchanges and how transaction latency and fees made it very risky at best and a losing proposition in most cases. But what I was wonderi…

Quant trading guy here. Most likely your simulation didn't account for market impact. It's maybe the hardest bit. See liquidity based bullet points above.

Other place to look is whether the data was recorded with timestamps of where the trading happened, but you probably thought of that one.

Idea makes sense though.

Re: More challenging projects every programmer should try

#125

Post by inopinatus » Mon, Dec 21, 2020, 7:26 AM UTC

pjc50 wrote: post #17

henning wrote: post #9

As someone who has played with writing trading bots but never traded them with real money, some advice: if your results seem too good to be true, they probably are. Your trading bot may be doing unrealistic things or its results may not be reliable if the following are true: - You are trading in a market with low liquidity or one that is controlled by a small number of market participants. I'm not an expert but I thi…

I worked on a unique HFT system which was capable of starting the response packet before the incoming packet's final byte had arrived. Negative latency. If it mispredicted the future, it would simply scramble the trailer and cause the packet to fail UDP checksum. It didn't make money in the real world because the quality of decision that could be made at that speed wasn't good enough.

From time to time there comes a remark that is the absolute epitome of Hacker News, and I mean that earnestly and without any backhanded rancour, and I thank you for this flawless specimen.

Re: More challenging projects every programmer should try

#126

Post by lordnacho » Mon, Dec 21, 2020, 7:27 AM UTC

pjc50 wrote: post #17

henning wrote: post #9

As someone who has played with writing trading bots but never traded them with real money, some advice: if your results seem too good to be true, they probably are. Your trading bot may be doing unrealistic things or its results may not be reliable if the following are true: - You are trading in a market with low liquidity or one that is controlled by a small number of market participants. I'm not an expert but I thi…

I worked on a unique HFT system which was capable of starting the response packet before the incoming packet's final byte had arrived. Negative latency. If it mispredicted the future, it would simply scramble the trailer and cause the packet to fail UDP checksum. It didn't make money in the real world because the quality of decision that could be made at that speed wasn't good enough.

What was the rest of the infrastructure? Was this at a proper HFT firm with everything else streamlined?

Re: More challenging projects every programmer should try

#127

Post by djeiasbsbo » Mon, Dec 21, 2020, 7:31 AM UTC

For some simpler projects, I can only recommend doing some digital signal processing. For example, an audio signal is just a list of values, so you can do things like:

- Count the number of zero crossings - Find out where they are - Create any shape of wave by adding together multiple sine waves - Hard clip the signal - Stretch a signal and interpolate it with new samples - Invert and revert a signal

For level 2, you can start processing "live":

- Create a sine synthesizer - Create a small ring buffer of samples - Find out how to output that audio (system audio, soundcard) - Add MIDI support - Add polyphony support

DSP gets hard once it has to be in real time and the latency has to be minimal. It's great exercise to mess around with it.

Re: More challenging projects every programmer should try

#128

Post by Aeolun » Mon, Dec 21, 2020, 7:39 AM UTC

xyzelement wrote: post #88

phist_mcgee wrote: post #79

Earlier quoted context omitted.

1. No idea who Prof Stark or CS219 is. Maybe some context here would help. 2. I f a course is super hard, maybe the class isn't 'full' of people who are just doing it to 'be cool, maybe that is your judgement and does not affect reality. 3. Great that you solved Tetris beforehand, but is there a point here? Are you implying that high school you was smarter than university peers? Sorry, but your post seems a little el…

Holly crap! First, there's not really a point. The preface to the post is that we're sharing nostalgia, right? Second, the "cool" thing is I got lucky that a test that I nearly blew myself up on happened to be on a subject I had already thought about a lot. However your #2 is off-base. There was something like 400% the applicants to the CS program in my university in 1999 vs 1998 and I bet that was true across the bo…

It was certainly true in my sort of CS program in 2005, where half the class had disappeared by the end of the first year. And that was after the bubble.

Re: More challenging projects every programmer should try

#129

Post by jamil7 » Mon, Dec 21, 2020, 7:39 AM UTC

djeiasbsbo wrote: post #127

For some simpler projects, I can only recommend doing some digital signal processing. For example, an audio signal is just a list of values, so you can do things like: - Count the number of zero crossings - Find out where they are - Create any shape of wave by adding together multiple sine waves - Hard clip the signal - Stretch a signal and interpolate it with new samples - Invert and revert a signal For level 2, you…

I came across this area recently while trying to build an iOS app that could reliably detect knocks or taps on the body of the phone, I was ultimately unsuccessful but I learned a lot about collecting and analysing microphone and gyroscope data. Also built some nice tooling for collecting live iOS sensor data and transmitting it to prometheus/grafana over mqtt.

Re: More challenging projects every programmer should try

#130

Post by anonytrary » Mon, Dec 21, 2020, 7:39 AM UTC

This article is aimed towards students. It's great advice for students who are in college, know very little, and want to improve their CS skills.

It's poor advice for someone who already has a STEM degree and wants to build something useful and profitable. If you already know how these things work, your time is better spent on the "edge of the circle": http://matt.might.net/articles/phd-school-in-pictures/ which applies to businesses and startups as well.

If you're in the latter group -- you've already got the skills to build real shit. Don't waste your time on homework problems. Find a problem you have and build a solution for it. Don't listen to people who tell you to work on homework problems that have already been solved; it's a complete waste of your time if you already know the fundamentals.

As for stock trading bots -- if you don't have a mathematics degree or equivalent (e.g. having incredible math skills), don't even bother. You won't be profitable, and you will learn nothing useful in the process, because you will approach the problem as a naive CS student would. Smarter people than you have made trading bots and have failed miserably. Without having an extremely strong foundation in mathematics, your trading bot will amount to nothing more than a futile exercise in gluing APIs together.