Pro-rated billing is one of the many places outside of database query planners where Dynamic Programming is useful! Maybe there's a better way to do it, but well, if you give a man a hammer.. I wrote up how we did this at Equipmentshare back in 2016: https://tech.davis-hansson.com/p/price-wars.html
Billing systems are a nightmare for engineers
31–40 of 372 posts
Re: Billing systems are a nightmare for engineers
#32I've worked at two companies in their payment systems. This post does a good job of mentioning many of the billing system challenges. I think it understates how difficult and important idempotency is to maintain at scale with multiple teams though.
Additionally, it didn't mention financial regulation changes. India had changes sometime in the last few years which required whole new systems built that were specific to India customers. One example of complexity is this: does the system apply to customers who are in India or does it apply to businesses who are in India (the owners might not be)?
He also did not mention fraud, which is basically uncollected spend that results in charge backs typically. If you get too many chargebacks, you'll get fined. If you get even more chargebacks, you'll get kicked off the card network and will no longer be able to accept cards within that network.
Post pay (customer gets the product and pays afterwards) models are subject to more fraud, uncollected charges, etc.
There's also the fun of dealing with downstream payment processors. I haven't worked with a processor or bank yet that was very reliable. Sometimes they'd return 500 http codes and then process a payment, requiring manual intervention. Sometimes that happens with files of payments because most banks deal with files for transactions.
Re: Billing systems are a nightmare for engineers
#33I've worked at two companies in their payment systems. This post does a good job of mentioning many of the billing system challenges. I think it understates how difficult and important idempotency is to maintain at scale with multiple teams though. Additionally, it didn't mention financial regulation changes. India had changes sometime in the last few years which required whole new systems built that were specific to…
> Additionally, it didn't mention financial regulation changes. India had changes sometime in the last few years which required whole new systems built that were specific to India customers. One example of complexity is this: does the system apply to customers who are in India or does it apply to businesses who are in India (the owners might not be)?
IIRC, I've read this is one of the reasons that systems like SAP are so popular. They might be an ugly monster from a technical perspective and seem overly complex, but they provide thoroughly tested implementations that handle all that stuff.
Re: Billing systems are a nightmare for engineers
#34Random idea: Don't build a billing system. Or at least not a precise one. Have a price list as normal, but just guestimate the user's bills based on any easily accessible information. Eg. For a hosting company I might just count how many VM's they have active when the billing script runs . If the user isn't happy with the total, give them a button to correct the total and pay that. Spot check what the user's pay, and…
This is a great way to be bankrupted by crypto currency miners.
Re: Billing systems are a nightmare for engineers
#35Once upon a time I was the first product person at a now-decacorn, and my first task was fixing the billing system. It was quite the monster, and we ended up implementing a combination of Zuora and an internal system, as there were some parts of the billing model Zuora couldn't handle.
I came away from this with one big lesson - if you're considering a complex billing model, consider the engineering implications first. With most products, engineering feedback gets taken into account - often product proposes something, engineering breaks it down, product realizes that feature x is vastly more complicated than they thought and not worth the effort, and the requirements are changed to simplify or remove it.
The one thing that never seems to be true of is pricing models - that decision gets made at the very top and handed down with no chance for feedback. I think that if it the folks designing the billing system realized the costs, they might simplify things. If the complexity of your billing system means that 3% of your engineering team (plus additional folks in support and finance) is going to be working on it forever, but if you simplify it a bit you could keep 90% of it and only have 1% of engineering working on it, that might be a good tradeoff - after all, that leaves you more engineers to build features, which should drive additional sales. Unfortunately, that analysis never seems to get done up front and the cost is only understood after the billing system is deeply integrated into everything and would take an unpalatable amount of effort to change.
Re: Billing systems are a nightmare for engineers
#36I've worked at two companies in their payment systems. This post does a good job of mentioning many of the billing system challenges. I think it understates how difficult and important idempotency is to maintain at scale with multiple teams though. Additionally, it didn't mention financial regulation changes. India had changes sometime in the last few years which required whole new systems built that were specific to…
Hey @Brystefor, I'm @Rafsark's co-founder.
We could have spent more time on idempotency, indeed.
We have not deep dived into the 3 other aspects you mentioned indeed, we're actually planning to
1/ open-source the articles and add additional challenges as they are mentioned
2/ open our roadmap based on these inputs as well
Thanks for sharing! Will ping you back here when it's live
PS: we had not thought of financial regulation changes indeed!
Re: Billing systems are a nightmare for engineers
#37I'm getting PTSD reading this article
Re: Billing systems are a nightmare for engineers
#38Just wait until you meet billing's angry roommate: invoicing. In the US, an invoice is just a weird PDF that you might glance at before sending off to your accounts payable team. But in other countries, especially those that use VAT style taxing systems, an invoice can be a legal document that expresses costs and taxes in a legally prescribed way for accounting purposes. Many countries have prescriptive rules over ho…
Many years ago I led a billing team at Yahoo! responsible for billing for premium services in European markets.
My team existed for the sole reason that the European business did not trust that the US payment services team understood European needs, and the "invoice is a legal document" thing was one of them. I spent so many meetings repeating to the US team that no, we could not switch to their invoicing as long as a new software release might retroactively change the template used to show already issued invoices (e.g. the registered office might change, or the VAT number).
We didn't have to deal with printed invoices thankfully, but we did ensure we produced each invoice once and stored it, so that the European finance team could sleep at night.
At the time Yahoo! had 8 different payment platforms. Some of those were due to weirdness around acquisitions and Japan (which was a joint venture), but apart from mine I believe two others also existed because of local weirdness that the local businesses decided it was safest not to let Yahoo! US mess with.
At the time I left, after 3 years of that, we'd managed to get agreements to migrate a few bits and pieces to the US after they'd shown the understood requirements for specific features, but it was like pulling teeth.
Re: Billing systems are a nightmare for engineers
#39Billing is a nightmare, and if I had one piece of advice for people building a pay-for-what-you-use system like most SaaS, it's this: DO NOT bill against your business logic entities. They change, and doing a COUNT at the end of the month won't catch all things which changed or which cost you money during the month. Instead, figure out what you bill for and and when you do that thing, record a row in a DB or into a s…
I think event sourcing / logging is the way to solve this kind of thing.
You have an obvious start/stop checkpoint in the log, processing can be done in batch after hours, no information ever gets destroyed, etc.
Mutable database rows that are used for accounting purposes should sound dangerous to anyone trusted with these systems.
Re: Billing systems are a nightmare for engineers
#40Always has been. Marketing and sales create complexity on the front end, devising all kinds of products, plans, discount schemes and related complexity; jurisdiction, taxes, and accounting rules add complexity on the backend. In the middle, measuring what you're billing for is the hook around which all that complexity chaotically orbits.
Decades ago - the computer involved was an honest to goodness System 360 (which is to say, not a 370, Z, or any of the the code compatible 360 successors) - I spent an evening trying to figure out the service charge on my bank account. I couldn't get at all close to the figure on the statement by applying the bank's schedule of charges. I took it into the bank, and got bumped by the teller to a department manager, and thence to a VP (small town; small bank), whom I told, "I will pay the service charge if you can explain to me what I'm being charged for." 2 hours later he gave up, having not gotten any where near an explanation, grinned, and said, "I can fix this." He set some flag on my account, and I never again paid service charges at that bank. Yep - that flag remained set on my account for nearly 20 years, through multiple software and hardware upgrades, until the bank was acquired by a big, expanding regional bank, and I left for a different, entirely local bank.