More challenging projects every programmer should try • Hacker News Board

Re: More challenging projects every programmer should try

#141

Post by djoldman » Mon, Dec 21, 2020, 8:26 AM UTC

pjc50 wrote: post #17

henning wrote: post #9

As someone who has played with writing trading bots but never traded them with real money, some advice: if your results seem too good to be true, they probably are. Your trading bot may be doing unrealistic things or its results may not be reliable if the following are true: - You are trading in a market with low liquidity or one that is controlled by a small number of market participants. I'm not an expert but I thi…

I worked on a unique HFT system which was capable of starting the response packet before the incoming packet's final byte had arrived. Negative latency. If it mispredicted the future, it would simply scramble the trailer and cause the packet to fail UDP checksum. It didn't make money in the real world because the quality of decision that could be made at that speed wasn't good enough.

This was an open secret. You can also send the start of a datagram on every incoming packet and then abort if you don't want to add/modify/delete an order by splitting your datagram into multiple packets. This worked on some exchanges because they ordered add/modify/deletes by when the beginning of the packet was received as opposed to the end.

If you do too much of this you will get angry exchange network people yelling at you as they may consider it spam/ddos. Some exchanges explicitly limit this.

It's also worth considering the possible gain. If you're in the fpga (0-150ns) or cpu(300ns-3us) space, the math can come out differently.

Re: More challenging projects every programmer should try

#142

Post by pjc50 » Mon, Dec 21, 2020, 8:30 AM UTC

KMag wrote: post #121

pjc50 wrote: post #17

Earlier quoted context omitted.

I worked on a unique HFT system which was capable of starting the response packet before the incoming packet's final byte had arrived. Negative latency. If it mispredicted the future, it would simply scramble the trailer and cause the packet to fail UDP checksum. It didn't make money in the real world because the quality of decision that could be made at that speed wasn't good enough.

To get such changes reliably with low latency, I presume you were modifying the DMA buffer, without any syscalls or other mechanisms to synchronize between the CPU and the NIC's processor(s), right? Did you just set up the race condition such that when you got bitten, it resulted in a failed checksum, or was the time window so large that in practice the race condition never bit you?

It was in a ridiculously expensive 10Gbit ethernet switch with an FPGA built in. Having to write trading algos in Verilog was an extra layer of difficulties.

I don't think it stored the packet at all, just advanced a state machine as each word arrived from the MAC.

Re: More challenging projects every programmer should try

#143

Post by pjc50 » Mon, Dec 21, 2020, 8:34 AM UTC

inopinatus wrote: post #125

pjc50 wrote: post #17

Earlier quoted context omitted.

I worked on a unique HFT system which was capable of starting the response packet before the incoming packet's final byte had arrived. Negative latency. If it mispredicted the future, it would simply scramble the trailer and cause the packet to fail UDP checksum. It didn't make money in the real world because the quality of decision that could be made at that speed wasn't good enough.

From time to time there comes a remark that is the absolute epitome of Hacker News, and I mean that earnestly and without any backhanded rancour, and I thank you for this flawless specimen.

The original case study: https://web.archive.org/web/20180224141009/http://www.argond...

Re: More challenging projects every programmer should try

#144

Post by pjc50 » Mon, Dec 21, 2020, 8:36 AM UTC

lordnacho wrote: post #126

pjc50 wrote: post #17

Earlier quoted context omitted.

I worked on a unique HFT system which was capable of starting the response packet before the incoming packet's final byte had arrived. Negative latency. If it mispredicted the future, it would simply scramble the trailer and cause the packet to fail UDP checksum. It didn't make money in the real world because the quality of decision that could be made at that speed wasn't good enough.

What was the rest of the infrastructure? Was this at a proper HFT firm with everything else streamlined?

We were consultants to an HFT firm; supposedly set up correctly but I didn't see that part of the project.

Re: More challenging projects every programmer should try

#145

Post by tijsvd » Mon, Dec 21, 2020, 8:39 AM UTC

jameskilton wrote: post #2

I would also recommend, if someone is interested in games, to do Tetris. It's a simple concept that is trickier than expected once you have to figure out the details of how it all comes together.

Even more fun, make tetris on a microprocessor, play on terminal via serial port. I did this as a diversion during my master's thesis - multiplayer tetris with two microprocessors, communicating through the radio protocol that I was building.

Re: More challenging projects every programmer should try

#146

Post by anonytrary » Mon, Dec 21, 2020, 8:44 AM UTC

watwut wrote: post #139

fergie wrote: post #123

I would strongly recommend building something that you yourself think is cool, and not feeling that you have to conform to what other people tell you to do.

Most people draw blank when they are trying to come up with project idea. They are programmers, not creative after all. Which is why lists like this help.

If you are trying to think of an idea, you're already doing it wrong. The best ideas are motivated by problems you encounter yourself, not by trying to think of ideas. This is the biggest mistake you see with 20-something founders. Best way to come up with ideas is to use existing solutions and realize how shitty they are. Pretty much every single company was formed this way.

> They are programmers, not creative after all.

Damn, this is a bold comment on HN. Since when does profession determine how creative you are? I've met "programmers" and "geeks" who are more creative than "artists". Your profession has little to do with your innate creativity -- it just determines how you are able to express that creativity.

Unpopular opinion, but lists like this are stupid for people who are trying to build companies. You have to try things and be pissed off at the status quo to find real problems. Nobody is going to find real problems for you, in the same way that no quant school is going to reveal their hedge fund's trading strategy to you. Finding ideas in a list is the last advice I would give to anyone. If it's public, it's probably not a profitable idea.

Re: More challenging projects every programmer should try

#147

Post by ma2rten » Mon, Dec 21, 2020, 8:47 AM UTC

fergie wrote: post #123

I would strongly recommend building something that you yourself think is cool, and not feeling that you have to conform to what other people tell you to do.

This comment is like the liar paradox, if people do what the comment suggests they are conforming to what another person tells them to do.

Re: More challenging projects every programmer should try

#148

Post by T_O_T_O » Mon, Dec 21, 2020, 8:51 AM UTC

anonytrary wrote: post #146

watwut wrote: post #139

Earlier quoted context omitted.

Most people draw blank when they are trying to come up with project idea. They are programmers, not creative after all. Which is why lists like this help.

If you are trying to think of an idea, you're already doing it wrong. The best ideas are motivated by problems you encounter yourself, not by trying to think of ideas. This is the biggest mistake you see with 20-something founders. Best way to come up with ideas is to use existing solutions and realize how shitty they are. Pretty much every single company was formed this way. > They are programmers, not creative afte…

For a beginning programmer, these could be good projects. For someone with a lot of experience, I mostly agree with your comment.

Re: More challenging projects every programmer should try

#149

Post by mukwenhac » Mon, Dec 21, 2020, 8:52 AM UTC

watwut wrote: post #139

fergie wrote: post #123

I would strongly recommend building something that you yourself think is cool, and not feeling that you have to conform to what other people tell you to do.

Most people draw blank when they are trying to come up with project idea. They are programmers, not creative after all. Which is why lists like this help.

Being a programmer doesn't mean one is not creative, those two are not mutually exclusive. Not to mention that one can observe a problem that can be fixed by a way they already know too.

Re: More challenging projects every programmer should try

#150

Post by mobiuscog » Mon, Dec 21, 2020, 8:54 AM UTC

anonytrary wrote: post #146

watwut wrote: post #139

Earlier quoted context omitted.

Most people draw blank when they are trying to come up with project idea. They are programmers, not creative after all. Which is why lists like this help.

If you are trying to think of an idea, you're already doing it wrong. The best ideas are motivated by problems you encounter yourself, not by trying to think of ideas. This is the biggest mistake you see with 20-something founders. Best way to come up with ideas is to use existing solutions and realize how shitty they are. Pretty much every single company was formed this way. > They are programmers, not creative afte…

> Finding ideas in a list is the last advice I would give to anyone. If it's public, it's probably not a profitable idea.

Why do they have to be profitable ? Many programmers like solving challenges to learn and have fun - it's not all about money. Advent of Code is a great example of this.