High-Speed Trading Firm Deleted Some Code by Accident • Hacker News Board

Re: High-Speed Trading Firm Deleted Some Code by Accident

#41

Post by tbrownaw » Fri, Oct 02, 2015, 12:38 PM UTC

smoothgrammer wrote: post #11

In finance you'll find low accountability for developers. Basically finance is where great developers go to get paid a lot to become horrible developers. Very few companies in finance are pushing the bar on quality code since they culturally cannot accept the cost of that. It doesn't make sense until it is too late and by then their code base is too entrenched. I've seen this at countless finance companies, and from…

But is it enough to put a company that doesn't do that at a competitive advantage? Is this really just company culture and assumptions, or are there actual market forces behind it?

Re: High-Speed Trading Firm Deleted Some Code by Accident

#42

Post by osi » Fri, Oct 02, 2015, 12:48 PM UTC

x0054 wrote: post #16

deckar01 wrote: post #3

Earlier quoted context omitted.

Some unit tests would be nice. Maybe even a special folder full of tests that know when laws have been broken.

Or how about at least some code documentation. In this case the programer deleted the code because he didn't understand why it was there. Well a simple "# Needed to Comply with regulation X" would probably have prevented this problem.

There's a specific incantation required to send the routed ISO orders via BATS as mentioned in the article. I can see how a refactoring might accidentally change that.

With that said, libraries that send orders to exchanges are required by the exchanges to be certified against their expected behaviors. If you want to send routed ISO orders, you have to test that with them. When making changes to said libraries, it can often be good to re-certify to ensure things are still working as expected.

Re: High-Speed Trading Firm Deleted Some Code by Accident

#43

Post by smoothgrammer » Fri, Oct 02, 2015, 1:30 PM UTC

mdpopescu wrote: post #33

smoothgrammer wrote: post #11

In finance you'll find low accountability for developers. Basically finance is where great developers go to get paid a lot to become horrible developers. Very few companies in finance are pushing the bar on quality code since they culturally cannot accept the cost of that. It doesn't make sense until it is too late and by then their code base is too entrenched. I've seen this at countless finance companies, and from…

I can confirm this. I just wish I had known about it before I accepted my current contract...

Get out while you can.

Re: High-Speed Trading Firm Deleted Some Code by Accident

#44

Post by smoothgrammer » Fri, Oct 02, 2015, 1:32 PM UTC

tbrownaw wrote: post #39

smoothgrammer wrote: post #15

Earlier quoted context omitted.

It is not about confidence. It is about ownership and being a good engineer. A good engineer would have cared about the code base and ensured that a test would fail, a use case would be linked to that failure, and someone would see that they have now violated a regulation. Then if the regulation changed, someone would grep through the use cases, and see what behavior is occuring/ how to modify the code base to cope w…

Real engineers are like true Scotsmen, because you can never have enough process. In this case, of course tests really do prove the absence of bugs rather than just (not) proving their presence. I really don't like the whole "unit tests lets you not worry about breaking things" view. That's not confidence, it's overconfidence . And it doesn't even mostly work for things where "correct" is more fuzzy than "eventually…

It is confidence. It is the right tool for the job.

What would you propose as an alternative?

Re: High-Speed Trading Firm Deleted Some Code by Accident

#45

Post by smoothgrammer » Fri, Oct 02, 2015, 1:33 PM UTC

vitalyd wrote: post #37

smoothgrammer wrote: post #15

Earlier quoted context omitted.

It is not about confidence. It is about ownership and being a good engineer. A good engineer would have cared about the code base and ensured that a test would fail, a use case would be linked to that failure, and someone would see that they have now violated a regulation. Then if the regulation changed, someone would grep through the use cases, and see what behavior is occuring/ how to modify the code base to cope w…

I'd be surprised if they didn't make postmortem changes to prevent such incidents in the future. Your posts in this thread imply that no mistake should ever happen, but let's be real - mistakes will always happen, the key is to do proper postmortem analysis and learn from them (and obviously prevent recurrences).

Some mistakes, yes. But this was a case of such a basic and clear cut use case that should have been tested.

My understanding is that companies such as this rely purely on manual testing. Thus it is easy for things to slip through the cracks. Usually they never get hit by penalties or get caught so the pressure to do better is not there.

Re: High-Speed Trading Firm Deleted Some Code by Accident

#46

Post by smoothgrammer » Fri, Oct 02, 2015, 1:36 PM UTC

tbrownaw wrote: post #41

smoothgrammer wrote: post #11

In finance you'll find low accountability for developers. Basically finance is where great developers go to get paid a lot to become horrible developers. Very few companies in finance are pushing the bar on quality code since they culturally cannot accept the cost of that. It doesn't make sense until it is too late and by then their code base is too entrenched. I've seen this at countless finance companies, and from…

But is it enough to put a company that doesn't do that at a competitive advantage? Is this really just company culture and assumptions, or are there actual market forces behind it?

It is a mixed bag.

In finance technology is seen as a cost center, not a profit center. They want the minimum needed to ship. Early on this results in a lot of tech debt. Later on, fixing the system is too hard and thus they need to hire super smart people to figure out the mess they are in for any code change.

Enforcement actions are INCREDIBLY rare. The penalties are shockingly low. Thus the cost of failure to comply is low, and the businesses are not prioritizing compliance.

Re: High-Speed Trading Firm Deleted Some Code by Accident

#47

Post by mrchicity » Fri, Oct 02, 2015, 1:45 PM UTC

kylebrown wrote: post #35

Sander_Marechal wrote: post #25

Can someone explain why the SEC rules insist you must buy from the exchange with the lowest price first? It appears to me to be a rule just to keep the small exchanges alive.

Otherwise a broker would just route mom n' pop's orders to the exchange that pays the biggest bribe to the broker (and screw mom n' pop).

Yes but for proprietary traders using their own capital, it doesn't make a lot of sense to force them to trade in a particular way since executing sub-optimally can only impact their own performance. There are legitimate reasons not to trade the lowest price market (maybe you believe the quote is slow, want to trade bigger size all at once, don't want to risk signaling to the market): http://www.hudson-trading.com/static/files/RegNMShudsonriver...

And retail brokers already do what you suggest. All their marketable flow gets sold to off-exchange market makers, while their limit order flow gets routed to exchanges that pay the highest liquidity rebates, Reg NMS just mandates what price it can trade at: http://news.indiana.edu/releases/iu/2014/02/study-of-potenti...

Re: High-Speed Trading Firm Deleted Some Code by Accident

#48

Post by mrchicity » Fri, Oct 02, 2015, 1:54 PM UTC

smoothgrammer wrote: post #46

tbrownaw wrote: post #41

Earlier quoted context omitted.

But is it enough to put a company that doesn't do that at a competitive advantage? Is this really just company culture and assumptions, or are there actual market forces behind it?

It is a mixed bag. In finance technology is seen as a cost center, not a profit center. They want the minimum needed to ship. Early on this results in a lot of tech debt. Later on, fixing the system is too hard and thus they need to hire super smart people to figure out the mess they are in for any code change. Enforcement actions are INCREDIBLY rare. The penalties are shockingly low. Thus the cost of failure to comp…

At a company like Tower, technology is their edge and definitely a competitive advantage. Having a system that is fast, reliable and scales is key to running a profitable automated trading operation. I worked for a similar trading company and technology was definitely not a cost center.

If enforcement actions are rare, it's probably because these firms are playing clean for the most part, contrary to popular belief. If you trade 5-10% of the market, regulators are going to audit you every year. They are looking for even the smallest issues to give a fine, even minor things like not having the right Head Trader's name on your supervisory forms after someone quits.

Re: High-Speed Trading Firm Deleted Some Code by Accident

#49

Post by osullivj » Fri, Oct 02, 2015, 1:57 PM UTC

A change like this making it through to production is indicative of a test failure, either automated or human. I have had the luxury of substantial human test teams on trading systems before 2008, and it was great. They wouldn't let something like this through...

Re: High-Speed Trading Firm Deleted Some Code by Accident

#50

Post by mrchicity » Fri, Oct 02, 2015, 2:04 PM UTC

waqf wrote: post #23

bdonlan wrote: post #19

Earlier quoted context omitted.

Why does the SEC care that you bought shares at a price higher than the cheapest? How does that hurt anyone other than yourself?

And relatedly, what is the point in the existence of multiple markets, if not that you can choose which market to trade on for your own technical–logistic reasons?

You can, but only at the same price. It's a weird system, especially when markets are located different places, so from each trader or exchange's vantage point, things are in different states. Working around this friction led to an explosion of special order types that some people argue are unfair or overly complex.

In Europe and Asia, there are competitive exchanges, but no rule to force people to route there. There are some guidelines for brokers with customer orders to have a duty of "best execution", but for proprietary traders or professionals, they can do whatever they feel is ideal.

And guess what, there are very few "trade throughs" or cases where markets invert with each other despite lacking such a rule. People usually act in their economic best interest without a law mandating they do so. If it happens, it's an arbitrage that someone will eliminate very quickly.