Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More challenging projects every programmer should try

web.eecs.utk.edu

Post reply on HN
Print view View original

181–190 of 232 posts

Re: More challenging projects every programmer should try

#181

Post by logarhythmic »

anonytrary wrote: post #130

This article is aimed towards students. It's great advice for students who are in college, know very little, and want to improve their CS skills. It's poor advice for someone who already has a STEM degree and wants to build something useful and profitable. If you already know how these things work, your time is better spent on the "edge of the circle": http://matt.might.net/articles/phd-school-in-pictures/ which appl…

These toy projects are worth doing if you want to continuously build your skills and like knowing how things work at a fundamental level. If you want to create an OS, it will obviously never be used by anyone ever and in that sense it's "useless", but you learn a lot in the process and it's fun.

Re: More challenging projects every programmer should try

#182

Post by guscost »

_huayra_ wrote: post #118
pjc50 wrote: post #17

Earlier quoted context omitted.

I worked on a unique HFT system which was capable of starting the response packet before the incoming packet's final byte had arrived. Negative latency. If it mispredicted the future, it would simply scramble the trailer and cause the packet to fail UDP checksum. It didn't make money in the real world because the quality of decision that could be made at that speed wasn't good enough.

Wow this is wild! I never thought things would get so latency-sensitive that one would have to overlap request and response!

HFT is so latency-sensitive that it matters how physically close you are on Manhattan island.

It is so latency-sensitive that new “hollow” fiber-optic cables are being installed, because light moves faster in air than in glass.

Re: More challenging projects every programmer should try

#183

Post by vntok »

guscost wrote: post #182
_huayra_ wrote: post #118

Earlier quoted context omitted.

Wow this is wild! I never thought things would get so latency-sensitive that one would have to overlap request and response!

HFT is so latency-sensitive that it matters how physically close you are on Manhattan island . It is so latency-sensitive that new “hollow” fiber-optic cables are being installed, because light moves faster in air than in glass.

See also how they slow down the whole network to level the field: https://hackaday.com/2019/02/26/putting-the-brakes-on-high-f...

Re: More challenging projects every programmer should try

#184

Post by dxdm »

imtringued wrote: post #134
anonytrary wrote: post #130

This article is aimed towards students. It's great advice for students who are in college, know very little, and want to improve their CS skills. It's poor advice for someone who already has a STEM degree and wants to build something useful and profitable. If you already know how these things work, your time is better spent on the "edge of the circle": http://matt.might.net/articles/phd-school-in-pictures/ which appl…

You should grow a second spike. I'm at the point where visual design skills are a greater bottleneck than my computer science skills. I can build purely functional CAD models but man, making them functional AND look good is the real challenge.

What's a "spike" in this context? This one's hard to google. I'm assuming something like "area of competence", but a more precise definition would be nice to know.

Re: More challenging projects every programmer should try

#185

Post by twic »

tijsvd wrote: post #135
pjc50 wrote: post #17

Earlier quoted context omitted.

I worked on a unique HFT system which was capable of starting the response packet before the incoming packet's final byte had arrived. Negative latency. If it mispredicted the future, it would simply scramble the trailer and cause the packet to fail UDP checksum. It didn't make money in the real world because the quality of decision that could be made at that speed wasn't good enough.

This is not unique, it's standard practice. Many exchanges send large UDP packets with the most valuable information at the front. Or the packet is structured such that you can make an informed bet based on size and first sub-message. Failing the checksum was sort of common as well, but exchanges don't like it. These days most tricks have to do with avoiding as much serialisation time as possible, e.g. by sending ahe…

Exchanges have cracked down on this sort of thing a lot this year.

From a letter from the CME to the CFTC dated July 24, 2020:

> On July 26, 2020, an enhancement to the Market Segment Gateway (“MSGW”) will be introduced to further safeguard the CME Globex electronic trading platform (“CME Globex”) infrastructure by introducing a delay of at least three microseconds if the MSGW receives a partial order message as a means of ensuring the stability of the platform. Certain participants intentionally submit partial order messages to reduce latency, and only complete the order message upon the happening of an event or trading signal. Implementation of the enhancement is expected to reduce the frequency of intentionally split order messages as the additional processing time will serve as a deterrent.

The letter is on the web, but it's a PDF; the Google search result has a redirect link to it, DuckDuckGo can't seem to find it, and Firefox on Android won't tell me the URL it downloads things from, so I'm afraid I can't link to it!

CME also made a more general change, where if they decide a participant is sending dodgy messages, they will reroute all their packets to a special gateway for "additional checks", but in practice, to impose a latency penalty. Can't find the documentation on that at all, though.

Re: More challenging projects every programmer should try

#186

Post by majormjr »

dxdm wrote: post #184
imtringued wrote: post #134

Earlier quoted context omitted.

You should grow a second spike. I'm at the point where visual design skills are a greater bottleneck than my computer science skills. I can build purely functional CAD models but man, making them functional AND look good is the real challenge.

What's a "spike" in this context? This one's hard to google. I'm assuming something like "area of competence", but a more precise definition would be nice to know.

Check the link from GP. You're correct a "spike" of knowledge is some topic that you specialize in.

Re: More challenging projects every programmer should try

#187

Post by contingencies »

henning wrote: post #9

As someone who has played with writing trading bots but never traded them with real money, some advice: if your results seem too good to be true, they probably are. Your trading bot may be doing unrealistic things or its results may not be reliable if the following are true: - You are trading in a market with low liquidity or one that is controlled by a small number of market participants. I'm not an expert but I thi…

Good list. You forgot "over-fitting your algorithm to historical data".

Re: More challenging projects every programmer should try

#188

Post by ignoramous »

simias wrote: post #27

I would add "build a toy regex engine" to the list. A couple of years ago I implemented a toy regex engine from scratch (building NFAs then turning them into DFAs). I thought it was an enlightening experience because it showed me that the core principles behind regular languages are fairly simple, although you could spend years optimizing and improving your implementation. How do you deal with unicode? How do you mod…

See also this Elevator game: https://play.elevatorsaga.com/

Re: More challenging projects every programmer should try

#189

Post by yodsanklai »

anonytrary wrote: post #130

This article is aimed towards students. It's great advice for students who are in college, know very little, and want to improve their CS skills. It's poor advice for someone who already has a STEM degree and wants to build something useful and profitable. If you already know how these things work, your time is better spent on the "edge of the circle": http://matt.might.net/articles/phd-school-in-pictures/ which appl…

> If you're in the latter group -- you've already got the skills to build real shit. Don't waste your time on homework problems.

I disagree. Not everything is about business and money. Many people already build "real shit" for a living and want to simply have fun building other things, and focus on the cool parts, and not all the boring parts involved in a commercial project.

Also CS is constantly evolving. Nobody knows the "fundamentals" once for all. A ray tracer is still a ray tracer, but languages and technologies have changed immensely in just a few years. Git didn't exist 15 years ago. A langage like Rust is 10 years old. React is 7 years old. We need these homework problems simply to keep up to date.

Re: More challenging projects every programmer should try

#190

Post by profquail »

chubot wrote: post #155
profquail wrote: post #104

Earlier quoted context omitted.

I wrote an implementation of this several years back. If you’re interested in the code: https://github.com/jack-pappas/facio/tree/master/Reggie The derivatives approach makes Unicode support easier since its able to keep the symbols sets for each transition edge (in the DFA) more compact by virtue of supporting negation. If you add in aggressive term-normalization, hash-consing, and an efficient dense-set implementat…

Very cool! And thanks for the reminder about Unicode. I think supporting union and intersection is also somewhat unique to the derivatives method, and also related? (Although I think there are really 2 derivatives methods: Brzozowski and Antimirov) What happens if you don't do the optimizations? Does the DFA blow up in size, meaning the compile time is large? Or does it make for a slower runtime? I would expect most…

The optimizations get you the following:

* Normalization: this is where "smart constructors" come in handy; having a normal form for the terms allows the caching to work better. This also impacts the compactness of the generated DFA. * Hash-consing: this turns structural equality (in this case) to a simple pointer equality; applied recursively, this makes it much faster to compare two terms for equality, and overall speeds up the DFA generation by a non-trivial amount (I forget the exact numbers, but it was significant). * Dense set implementation: The AVL tree-based data structure in the facio/Reggie code is an implementation of the Discrete Interval Encoding Tree (DIET) data structure from "Diets for fat sets" and "More on Balanced Diets" papers.

Note the optimizations I've mentioned here impact the performance of generating the DFA. Once you have the DFA, it'll run at the same speed as one generated in any other way. Part of the motiviation for my writing this library was to learn about regex/DFAs/grammars, but also to try to improve on the performance of fslex/fsyacc at the time. Using this library, the FSharpLex tool can generate the DFA for the full F# language grammar in well under 1 sec; the code generation takes a bit longer, largely due to having to convert the DFA into a different form for backwards-compatibility with fslex.

Overall, I feel like the derivatives technique is generally better and simpler, and I'm not aware of any real downsides. The only one that comes to mind is if you're wanting to implement things like backreferences and capture groups -- those obviously make the implementation (of the DFA) more complicated, and there's a lot less literature on it (last I saw, maybe only one or two papers on implementing those features on top of a derivatives-based regex engine).

Post reply on HN