I have rolled out two transactional databases of my own. In both cases I had to provide very specific properties and for some reason I could not find an existing product that would meet all requirements. For example, one of them was an embedded device that was very restricted in memory, all operations needed to run with hard bounds on time and memory and the storage for the data was a flash chip without wear levelling which required the database itself to manage writes to prolong the chip's life.
The key is to notice how your database system is going to be different from others and what properties are not essential.
Also, making general purpose DBMS tends to be much more complex vs making more niche solutions where you know a bit more about what the uses are going to be and what kinds of loads you can expect.
Creating a custom engine for a given application can be very simple task because you can easily cross out requirements you don't care about and you only care that it works well for the loads that this particular application can generate.
Also, it is unlikely you are going to beat fierce competition in general purpose "and a kitchen sink" database management system market, but much easier to find a niche that is underserved and create a usable, competitive product with relatively little effort. That's how SQLite started.