Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineering principles for building financial systems

substack.wasteman.codes

Post reply on HN
Print view View original

51–60 of 112 posts

Re: Engineering principles for building financial systems

#51

Post by ibejoeb »

throwaway2037 wrote: post #48
ibejoeb wrote: post #29

We're basically talking about bookkeeping systems here, rather than high finance. Use a good relational database. 1. ACID, so you don't have to invent it. 2. Arbitrary precision numeric data types with vetted operations and rounding modes. 3. It does time as well as anyone can 4. Your computations and reporting can be done entirely in SQL. 5. When you get good at SQL or hire someone who is, the reporting is elegant.…

What is the meaning of "high finance"?

Specialty stuff like high frequency trading, risk models, bespoke instruments, and other non-retail products. Like if you're doing HFT, you're probably rolling your own loops.

But the broader point is that this piece is about accounting-related systems that don't typically deal with hard volume, speed, or latency constraints. The term "finance" can mean the corporate "finance & accounting department" but you want to differentiate that from "finance" the industry. The headline does really tell you what actual material is.

Re: Engineering principles for building financial systems

#52

Post by [deleted] »

throwaway2037 wrote: post #48
ibejoeb wrote: post #29

We're basically talking about bookkeeping systems here, rather than high finance. Use a good relational database. 1. ACID, so you don't have to invent it. 2. Arbitrary precision numeric data types with vetted operations and rounding modes. 3. It does time as well as anyone can 4. Your computations and reporting can be done entirely in SQL. 5. When you get good at SQL or hire someone who is, the reporting is elegant.…

What is the meaning of "high finance"?

[deleted]

Re: Engineering principles for building financial systems

#53

Post by DarkContinent »

> Batch is just a special case of streaming

No. Designing a system that is always up and running and can process small amounts of data constantly is a completely different problem from designing a system that runs occasionally with a lot of data. For one thing, your output formats are usually different in the latter case (maybe you're creating a PDF for example). Also the high availability requirement just makes things different at the design level.

Finally, the author claims it's not hard to switch between batch and streaming. With a large volume of preexisting data, this is just not true. For example, if you make a REST API call for each document in a DB, it can take days or months to load that. If batching together documents isn't a possibility, how do you move data between stores easily? (This data movement is often required when switching between batch and streaming.)

Re: Engineering principles for building financial systems

#54

Post by dmoy »

ibejoeb wrote: post #51
throwaway2037 wrote: post #48

Earlier quoted context omitted.

What is the meaning of "high finance"?

Specialty stuff like high frequency trading, risk models, bespoke instruments, and other non-retail products. Like if you're doing HFT, you're probably rolling your own loops. But the broader point is that this piece is about accounting-related systems that don't typically deal with hard volume, speed, or latency constraints. The term "finance" can mean the corporate "finance & accounting department" but you want to…

Yea this article really should have used "accounting" instead of "finance" (or more specifically, operational accounting).

Right, like, if you're a programmer going into a project-for-finance/accounting-team at your large megacorp, and don't know if this article is the right subject area, just ask the finance/accounting person about "year end close" and see if they get a thousand yard stare or not. If yes, this is the right article lol.

Re: Engineering principles for building financial systems

#55

Post by simpaticoder »

amluto wrote: post #27
Galanwe wrote: post #4

I wouldn't endorse many of the things stated there. A lot of the points are inaccurate or straight up misleading. > The three main goals of your accounting system are to be (1) Accurate, (2) Auditable and (3) Timely. You forgot "consistent", which is very different from "accurate". Most financial transactions have multiple time dimensions: a trade is requested, booked, filled, matched, reconciliated, paid, settled at…

> Most financial transactions have multiple time dimensions: a trade is requested, booked, filled, matched, reconciliated, paid, settled at different times. These times often don't even follow a perfectly logical order (you can request and fill a trade and book it an hour later). Similar issues exist with transaction identifiers. Every party involved with any aspect of a transaction may assign that aspect of the tran…

There is an interesting tension in software between measurement and validation. The two concerns are often conflated, resulting in software systems that cannot measure the world as-it-is. The world (more precisely, measurements of the world) may be in (and often is in) an 'invalid' or 'inconsistent' state. Software can only help you detect and resolve such states if it is capable of representing them.

Re: Engineering principles for building financial systems

#56

Post by infecto »

Havoc wrote: post #50

Mostly agree though the concept of materiality seems out of place here. Decision making factors that in sure but system design absolutely shouldn’t. There is no reason why accounting systems can’t be accurate down to the penny. That’s what computers are good at after all. Only place where I could see system like relevance is in presentation- showing revenue on millions etc

Agree. I mostly ignored the rest of the post because this threw red flags that the poster does not build real financial systems. Financial systems should be capturing everything, now on the reporting end there is a question on materiality but most definitely not on the system design.

Re: Engineering principles for building financial systems

#57

Post by seanhunter »

ibejoeb wrote: post #51
throwaway2037 wrote: post #48

Earlier quoted context omitted.

What is the meaning of "high finance"?

Specialty stuff like high frequency trading, risk models, bespoke instruments, and other non-retail products. Like if you're doing HFT, you're probably rolling your own loops. But the broader point is that this piece is about accounting-related systems that don't typically deal with hard volume, speed, or latency constraints. The term "finance" can mean the corporate "finance & accounting department" but you want to…

For HFT, risk models, exotic products etc you still are almost certainly better off doing your bookkeeping using a normal database. In my experience[1] you want to make sure the financial records are consistent and accurate and able at all times to be reconciled to third parties no matter what lifecycle events happen on your trades and you want that record to be out of band with your actual decisionmaking stuff which is the bit that needs to be fast and/or complex. For exotics specifically, the contracts often specify exactly how things like rounding etc should be handled so you need to be able to implement custom logic to do that accurately or you'll get constant annoying price breaks of a few cents and your ops people will hate you because they'll keep having to reconcile things with counterparties.

[1] Fast equities trading and slow complex exotics and bonds trading/pricing/risk. Strictly speaking the fast thing was algo trading[2] rather than actual HFT but we were counting microseconds etc so it's similar in the time constraint.

[2] The distinction I would draw is algo trading is algorithmic execution of client orders in a brokerage setting vs HFT is proprietary trading like you would do in a hedge fund. In algo trading a lot of thought goes into how you can ensure your clients are not able to be ripped off by HFTs in the market, so they are sort of the adversary even though a lot of what you're doing is similar from an execution pov.

Re: Engineering principles for building financial systems

#59

Post by HWR_14 »

throwaway2037 wrote: post #44
amluto wrote: post #25

Earlier quoted context omitted.

> Unix timestamp, or even integer based UTC datetimes work perfectly fine. For serious work, it’s worth noting that leap seconds are not representable in this format. Many financial applications can get away without representing leap seconds, but this is fundamentally a kludge. If you actually need to represent any point in time (according to UTC or any particular timezone), use a representation that actually represe…

Is there any financial system, regime, or product that depends upon leap seconds? I have never heard of any. If so, it is safe to ignore them. When storing date/time stamps, I prefer to persist into two separate fields: (a) date/time using UTC time zone and (b) original time zone offset, e.g., UTC+8. When debugging, it is useful to know the original time zone offset. And when querying and processing data, it is easie…

> Is there any financial system, regime, or product that depends upon leap seconds?

According to the standard for increasing leap seconds, they interact with the opening auction of the Tokyo Stock Exchange by delaying it one second.

Re: Engineering principles for building financial systems

#60

Post by HWR_14 »

bigiain wrote: post #8
Terr_ wrote: post #3

> Use consistent rounding methodologies This may also mean promoting them to a named part of the business-domain in code, with their own functions, unit-tests, stuff like "fetch Rounding Strategy Suite by country code", etc. > Use integer representations of time. This one is a little controversial but I stand by it. There are so many libraries in different technologies that parse timestamps into objects, and they all…

Also: > Granularity of your financial amounts should ... ... comply with relevant regulations. I worked on a gambling webapp a long time back - we were bound by government regulations on gambling licences to perform all calculations at a precision of one ten thousandth of a cent, and were only permitted to round to cents once at the very end just before displaying a dollar amount to a person. (This suited me down to…

> This suited me down to the ground because I pointed out to everybody that Javascript couldn't be guaranteed to treat numbers and calculations in a way that would keep the regulators auditors happy,

Javascript can represent dollar values up to $2.25 billion at a resolution of ten-thousandths of a cent without any loss of precision.

I would want all money calculations done by the backend team as a matter of policy. But it's not a technical limitation.

Post reply on HN