Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineering principles for building financial systems

substack.wasteman.codes

Post reply on HN
Print view View original

61–70 of 112 posts

Re: Engineering principles for building financial systems

#61

Post by HWR_14 »

woah wrote: post #58

Cryptocurrencies tend to use very large integers (128 or 256 bit) for everything throughout the system. Rounding doesn't happen until a number hits the UI. Why would you design a system that intentionally destroys data several times internally?

Sometimes precision (and thus data) needs to be destroyed. Say you owe me $1,000.211 and write me a check for $1,000.21

Do you still owe me a tenth-of-a-cent? Even if you do, does anyone (me, you, the taxing authorities) care? In fact, does anyone want to go through the effort of maintaining that debt? Even if I repeat this with 100,000 customers, I'm out $100. Again, does anyone care?

Re: Engineering principles for building financial systems

#62

Post by gradschool »

throwaway2037 wrote: post #44
amluto wrote: post #25

Earlier quoted context omitted.

> Unix timestamp, or even integer based UTC datetimes work perfectly fine. For serious work, it’s worth noting that leap seconds are not representable in this format. Many financial applications can get away without representing leap seconds, but this is fundamentally a kludge. If you actually need to represent any point in time (according to UTC or any particular timezone), use a representation that actually represe…

Is there any financial system, regime, or product that depends upon leap seconds? I have never heard of any. If so, it is safe to ignore them. When storing date/time stamps, I prefer to persist into two separate fields: (a) date/time using UTC time zone and (b) original time zone offset, e.g., UTC+8. When debugging, it is useful to know the original time zone offset. And when querying and processing data, it is easie…

> If so, it is safe to ignore them.

if you say so, but don't whinge if someone pulls off a major heist during a leap second by counting on you to ignore it

https://www.imdb.com/title/tt0137494/

Re: Engineering principles for building financial systems

#63

Post by lkdfjlkdfjlg »

DarkContinent wrote: post #53

> Batch is just a special case of streaming No. Designing a system that is always up and running and can process small amounts of data constantly is a completely different problem from designing a system that runs occasionally with a lot of data. For one thing, your output formats are usually different in the latter case (maybe you're creating a PDF for example). Also the high availability requirement just makes thin…

I'm seconding this, and I have first hand experience in exactly this problem, in finance. My first boss also had the view that "batching is a special case of streaming where you stream N and streaming is also a special case of batching where the batch size is 1 and so it doesn't matter which one you implement". This was never performant enough and he was eventually asked to leave.

Re: Engineering principles for building financial systems

#64

Post by davidw »

Terr_ wrote: post #11
bigiain wrote: post #8

Earlier quoted context omitted.

Also: > Granularity of your financial amounts should ... ... comply with relevant regulations. I worked on a gambling webapp a long time back - we were bound by government regulations on gambling licences to perform all calculations at a precision of one ten thousandth of a cent, and were only permitted to round to cents once at the very end just before displaying a dollar amount to a person. (This suited me down to…

> We asked for and got a ruling from the regulators that the time that mattered was the time the API call was processed by the backend I haven't been in that situation, but I imagine I would reach for a "what if it was snail-mail" analogy. "It's dangerous to honor whatever time the bettor wrote by hand, the postmark-time may not be fair if the mailman got delayed because of a half-illegible address or random USPS del…

I wonder if that's the sort of ruling that just got thrown out by the Supreme Court in favor of having courts decide.

Re: Engineering principles for building financial systems

#65

Post by cvdub »

Managing rounding and ensuring each set of entries balance can be tricky, especially if you have to share data with a system that can only handle currencies with two decimal places. There are scenarios where it’s actually not possible to have every set of entries balance, and have the total sum of all entries equal the correct balance.

For example, if you had three monthly payments of $5, $10, and $10, you might book something like:

Cash (5) Expense 8.33 Deferred (3.33)

Cash (10) Expense 8.33 Deferred 1.67

Cash (10) Expense 8.33 Deferred 1.67

All three of those blocks of entries balance, but the sum of expenses is 24.99 instead of 25.

I’m not sure there’s a way around this issue if you’re forced to use two decimal places. Luckily the discrepancy is immaterial. I’d love to know if anyone else has encountered this problem.

Re: Engineering principles for building financial systems

#66

Post by nostrademons »

seanhunter wrote: post #57
ibejoeb wrote: post #51

Earlier quoted context omitted.

Specialty stuff like high frequency trading, risk models, bespoke instruments, and other non-retail products. Like if you're doing HFT, you're probably rolling your own loops. But the broader point is that this piece is about accounting-related systems that don't typically deal with hard volume, speed, or latency constraints. The term "finance" can mean the corporate "finance & accounting department" but you want to…

For HFT, risk models, exotic products etc you still are almost certainly better off doing your bookkeeping using a normal database. In my experience[1] you want to make sure the financial records are consistent and accurate and able at all times to be reconciled to third parties no matter what lifecycle events happen on your trades and you want that record to be out of band with your actual decisionmaking stuff which…

You should do your bookkeeping in a normal database, but the models themselves usually need something specialized. Ideally keep them in cache/RAM, because if you have to hit the disk you'll probably get beaten to execution by another HFT. If the data set is too large to keep in RAM (and you can't afford to just buy more computers), then page out portions to flash and mmap them back in when you need to. (Ironically, this is sort of how programs were constructed in the 1970s before the days of virtual memory, filesystem abstractions, and file formats.)

Re: Engineering principles for building financial systems

#67

Post by seanhunter »

nostrademons wrote: post #66
seanhunter wrote: post #57

Earlier quoted context omitted.

For HFT, risk models, exotic products etc you still are almost certainly better off doing your bookkeeping using a normal database. In my experience[1] you want to make sure the financial records are consistent and accurate and able at all times to be reconciled to third parties no matter what lifecycle events happen on your trades and you want that record to be out of band with your actual decisionmaking stuff which…

You should do your bookkeeping in a normal database, but the models themselves usually need something specialized. Ideally keep them in cache/RAM, because if you have to hit the disk you'll probably get beaten to execution by another HFT. If the data set is too large to keep in RAM (and you can't afford to just buy more computers), then page out portions to flash and mmap them back in when you need to. (Ironically, t…

Absolutely. Our models didn't look anything like most programs I have worked on elsewhere in terms of architecture etc. But the execution/bookkeeping records just went into a conventional sql db out of band from the main routing/strategy logic.

Re: Engineering principles for building financial systems

#68

Post by caseysoftware »

Years ago, I worked on APIs for a top 10 bank in the US. In our discovery/delving into their logic, the most fascinating thing was interest rate.

Some of their systems stored - and therefore the APIs - presented interest rate as a simple number (5% was 5) where other used a decimal (5% was 0.05) and still others used basis points (5% was 500).

One of their VPs told me they once saw an unexpected uptick in loan applications and dug in to find out the quoted rate was 0.05%

Consistency is key.

Re: Engineering principles for building financial systems

#69

Post by mgaunard »

eru wrote: post #23
mgaunard wrote: post #19

Representing prices is difficult because a lot of financial instruments have widely different values and price increments. Decimal floating-point gives you the range to cover everything but is generally very inefficient to process. What you usually want is fixed-point decimal, with the caveat that the scaling factor will be different per asset.

Depending on how many numbers you process, inefficiency might not matter. You could probably use arbitrary precision rational numbers (eg as implemented in Python https://docs.python.org/3/library/fractions.html ) for all your accounting needs, and it would still be fast enough.

There are millions of financial instruments and their price change 100 million times per day.

That's a lot to process, so yes, it matters.

Re: Engineering principles for building financial systems

#70

Post by [deleted] »

caseysoftware wrote: post #68

Years ago, I worked on APIs for a top 10 bank in the US. In our discovery/delving into their logic, the most fascinating thing was interest rate. Some of their systems stored - and therefore the APIs - presented interest rate as a simple number (5% was 5) where other used a decimal (5% was 0.05) and still others used basis points (5% was 500). One of their VPs told me they once saw an unexpected uptick in loan applic…

[deleted]
Post reply on HN