https://en.m.wikipedia.org/wiki/Conservation_of_energyFood (calories in) is just a form of energy your body is capable of utilizing. Fat is just a physical medium to store energy for use in your body. For your body to move and perform actions (live), you require energy. All things considered at this scale, if you expend more energy than you consume, you'll ultimately either die (due to lack of energy) or use other forms of energy in your body (e.g. fat).
This is of course the 30,000 ft view and there's a lot of complex biology and chemistry going on but it's pretty straightforward if you believe the law of conservation of energy is correct and our bodies aren't doing some sort of nuclear physics to process energy and instead relying on various forms of chemical and physical energy.
As you point out, it's not so cut and dry though in practice. There's a time component here and our bodies get quite clever. We build up efficiency of certain tasks over time requiring less energy usage. We resist building muscle because it requires more energy to maintain. Some food's energy density isn't perceived very well by our bodies and we remain more or less hungry irrelevant of potential calories making it more difficult to maintain a calorie deficit/diet based solely on our bodies senses. Many nutritional information products don't accurately measure calories of food, sometimes making it difficult to use as a base metric (theres a lot of errors in these measurements, especially by small food producers/providers). We also have incredible difficulty trying to estimate how many calories our body is actually expending because it's a fairly complex system (the "calories out" part of the equation) although there are some figures out there to help you estimate these within some general ranges depending on factors like age, diet, fitness, activity and so on--it's not perfect but it's something to work with quantitatively.
The benefit of calorie counting though is that there are fairly basic physical and chemical principles at play that make it a useful tool in weight loss and management. Even if a bit variable (both the in and out) and sometimes inaccurately represented, it gives us something fairly quantitative to work with, measure, and change around to control our weight. We can measure it and change other variables in our lifestyles to, over time, manage weight. It works and the statement is valid but it's not simple to use by any means, there's a lot one needs to learn over time to better learn how to measure their in/out and overall body feeling based on these factors.