More challenging projects every programmer should try • Hacker News Board

Re: More challenging projects every programmer should try

#211

Post by na85 » Mon, Dec 21, 2020, 10:38 PM UTC

burnthrow wrote: post #193

na85 wrote: post #122

Earlier quoted context omitted.

Not only that but adventofcode seems to leverage purposely-obtuse input formatting as an artificial difficulty enhancement. I stopped doing advent this year when I realized I was spending more time debugging my parsing than I was actually solving the problems. It's just not that fun for me to sometimes spend 2+ hours finding parse bugs before moving on to the actual puzzle.

The puzzles aren't fun at all. Like you said, they're basically "write a parser + complete a tedious word problem." They require substantial time investment for little creative reward, it's basically masochism. Perl Advent Calendar is my jam.

I dunno, some of them are fun. The ship navigation one this year was fun to solve, as was the gameboy emulator one.

Re: More challenging projects every programmer should try

#212

Post by eneveu » Mon, Dec 21, 2020, 11:27 PM UTC

twic wrote: post #185

tijsvd wrote: post #135

Earlier quoted context omitted.

This is not unique, it's standard practice. Many exchanges send large UDP packets with the most valuable information at the front. Or the packet is structured such that you can make an informed bet based on size and first sub-message. Failing the checksum was sort of common as well, but exchanges don't like it. These days most tricks have to do with avoiding as much serialisation time as possible, e.g. by sending ahe…

Exchanges have cracked down on this sort of thing a lot this year. From a letter from the CME to the CFTC dated July 24, 2020: > On July 26, 2020, an enhancement to the Market Segment Gateway (“MSGW”) will be introduced to further safeguard the CME Globex electronic trading platform (“CME Globex”) infrastructure by introducing a delay of at least three microseconds if the MSGW receives a partial order message as a me…

Here is the link to the PDF: https://www.cftc.gov/sites/default/files/filings/orgrules/20...

Re: More challenging projects every programmer should try

#213

Post by IgorPartola » Tue, Dec 22, 2020, 1:26 AM UTC

caffeine wrote: post #180

IgorPartola wrote: post #106

Earlier quoted context omitted.

Would you mind if I asked you a question since I have never worked in this industry but did play with crypto trading a while back. Before Mt Gox was shut down I was trading on a couple of tertiary small exchanges and at the time there was a lot of talk of arbitrage between different exchanges and how transaction latency and fees made it very risky at best and a losing proposition in most cases. But what I was wonderi…

Your idea is right but the timescale is much shorter. Essentially it's a race. And in 2014 you could make good money doing exactly what you described, the timescale was millis. In 2016-2017 even more money, but different venues. What you are describing is the most basic, canonical form of latency arbitrage. That idea is implemented widely in the HFT industry across all sorts of products, not just crypto, with billion…

Thank you. That makes a lot of sense. The problem was that I was doing this on my residential cable connection from the US and the exchange where I was executing trades would be in Eastern Europe. The ironic thing is that even back then I had a good idea of how to set up communications with a server like theirs to be significantly faster but it didn’t occur to me that milliseconds mattered. A typical trade if they executed it almost instantly still took 300ms. And they didn’t execute instantly so it was closer to 2 seconds more likely than not. Out of curiosity, what are the fancy examples of this vs this simplistic one?

Re: More challenging projects every programmer should try

#214

Post by IgorPartola » Tue, Dec 22, 2020, 1:27 AM UTC

lordnacho wrote: post #124

IgorPartola wrote: post #106

Earlier quoted context omitted.

Would you mind if I asked you a question since I have never worked in this industry but did play with crypto trading a while back. Before Mt Gox was shut down I was trading on a couple of tertiary small exchanges and at the time there was a lot of talk of arbitrage between different exchanges and how transaction latency and fees made it very risky at best and a losing proposition in most cases. But what I was wonderi…

Quant trading guy here. Most likely your simulation didn't account for market impact. It's maybe the hardest bit. See liquidity based bullet points above. Other place to look is whether the data was recorded with timestamps of where the trading happened, but you probably thought of that one. Idea makes sense though.

I was moving very small amounts. I think my Max trade was set at like $50 equivalent so it wasn’t anything crazy. It seems my main issue was latency and not realizing that my time scales should have been significantly shorter.

Re: More challenging projects every programmer should try

#215

Post by whatever_dude » Tue, Dec 22, 2020, 3:44 AM UTC

My own favorite is an equation parser.

Before attempting to do so I thought it was implemented as a simple seek over the string, maybe a bunch of regex stuff. I guess it can be done that way, at the cost of growing complexity; but the proper solution (with a stack, etc) is so elegant (makeing it easy to add functions, operators, parenthesis, variables, etc) that it really makes one appreciate the value of good, thoughtful engineering.

Re: More challenging projects every programmer should try

#216

Post by logarhythmic » Tue, Dec 22, 2020, 5:42 AM UTC

watwut wrote: post #172

mukwenhac wrote: post #149

Earlier quoted context omitted.

Being a programmer doesn't mean one is not creative, those two are not mutually exclusive. Not to mention that one can observe a problem that can be fixed by a way they already know too.

Sure but plenty of people are attracted to programming because they are not creative. Pretending that they don't exist or pretending that everyone is great at everything is not accurate.

Are engineers creative? Physicists? Mathematicians? I don't know if you have a very narrow definition of creativity or if you seriously think that the people who created all the software that you use everyday are not creative

Re: More challenging projects every programmer should try

#217

Post by caffeine » Tue, Dec 22, 2020, 9:24 AM UTC

andrepd wrote: post #194

caffeine wrote: post #180

Earlier quoted context omitted.

Your idea is right but the timescale is much shorter. Essentially it's a race. And in 2014 you could make good money doing exactly what you described, the timescale was millis. In 2016-2017 even more money, but different venues. What you are describing is the most basic, canonical form of latency arbitrage. That idea is implemented widely in the HFT industry across all sorts of products, not just crypto, with billion…

Honest question: do you feel comfortable knowing that your job is doing something which is essentially net zero value to society? Negative in fact, you're burning resources which are needed elsewhere, and contributing to concentration of wealth. Not being rude or judgemental, just want to know how you feel about this.

Short answer: I am comfortable with my job.

I think your question makes some incorrect factual assumptions, but also it incorporates a world view that I don't subscribe to (not wrong, but also not what I believe).

I think that if I had your belief system and your set of facts, I would probably feel uncomfortable about it.

Re: More challenging projects every programmer should try

#218

Post by caffeine » Tue, Dec 22, 2020, 9:35 AM UTC

IgorPartola wrote: post #213

caffeine wrote: post #180

Earlier quoted context omitted.

Your idea is right but the timescale is much shorter. Essentially it's a race. And in 2014 you could make good money doing exactly what you described, the timescale was millis. In 2016-2017 even more money, but different venues. What you are describing is the most basic, canonical form of latency arbitrage. That idea is implemented widely in the HFT industry across all sorts of products, not just crypto, with billion…

Thank you. That makes a lot of sense. The problem was that I was doing this on my residential cable connection from the US and the exchange where I was executing trades would be in Eastern Europe. The ironic thing is that even back then I had a good idea of how to set up communications with a server like theirs to be significantly faster but it didn’t occur to me that milliseconds mattered. A typical trade if they ex…

Trading the same product on two exchanges is the most basic one because you know it's the same product, so correlation should be 1, assuming the cost to transfer between exchanges is zero. Example is arbing US equities.

Crypto is one level more complicated, because you don't share inventory across exchanges and transaction costs are high, which means a coin on Exchange1 isn't perfectly fungible with a coin on Exchange2.

One level more indirect than that might be basis trades, where you trade a derivative ( like SP500 futures) vs it's underlying (the SP500 stocks, although in practice it's SP500 ETFs). So here the correlation is very high but there is a difference between futures, stocks, and ETFs fundamentally, and those play into the pricing.

Going even further might be trading correlated products that don't have the same underlying, example is Nasdaq futures vs SP500 futures.

To simplify: It's basically about the level of correlation between the products. The strategies used to trade different correlations look qualitatively different.

Re: More challenging projects every programmer should try

#219

Post by IgorPartola » Tue, Dec 22, 2020, 12:58 PM UTC

caffeine wrote: post #218

IgorPartola wrote: post #213

Earlier quoted context omitted.

Thank you. That makes a lot of sense. The problem was that I was doing this on my residential cable connection from the US and the exchange where I was executing trades would be in Eastern Europe. The ironic thing is that even back then I had a good idea of how to set up communications with a server like theirs to be significantly faster but it didn’t occur to me that milliseconds mattered. A typical trade if they ex…

Trading the same product on two exchanges is the most basic one because you know it's the same product, so correlation should be 1, assuming the cost to transfer between exchanges is zero. Example is arbing US equities. Crypto is one level more complicated, because you don't share inventory across exchanges and transaction costs are high, which means a coin on Exchange1 isn't perfectly fungible with a coin on Exchang…

That makes sense and sounds pretty interesting. How “exciting” is this field in terms of innovation currently? Is it mostly about getting high level access to exchanges to speed up trading (which presumably requires paying your way in), or is it more about finding novel signals and ways to use them?

Re: More challenging projects every programmer should try

#220

Post by ddls » Tue, Dec 22, 2020, 1:34 PM UTC

henning wrote: post #9

As someone who has played with writing trading bots but never traded them with real money, some advice: if your results seem too good to be true, they probably are. Your trading bot may be doing unrealistic things or its results may not be reliable if the following are true: - You are trading in a market with low liquidity or one that is controlled by a small number of market participants. I'm not an expert but I thi…

Because

- management fees, margins, and available capital can easily be modelled properly

- you can easily set up constraints (like no fractional trading for your S&P example)

- you can set up a proper point-in-time database to avoid snooping (especially if you're using earnings reports or other fundamental data which is often actually released _after_ it's published release date...)

- you can set up regime-shifting simulation environments (various market conditions)

- you can avoid over-fitting if you're back-testing (with dozens of techniques, most notably : test once and forget about parameter optimization)

I would say that with paper trading and back-testing the serious problems are that :

- your orders don't show up on the book so no one sees and reacts to your limit orders

- your "filled" orders don't affect the book, so you're not affecting liquidity, so the market doesn't change in response to your trading

- your bot has no access to market micro-structure strategies and conditional orders (and if you want to trade fast or are placing big trades you need them)

These are the problems that make any simulation unrealistic, and they are fundamental. It's shadowboxing, which is not entirely devoid of value, but which is certainly insufficient on its own.

(I've worked as a quant developing strategies for several funds these past 15 years)