"This is confusing...if you only eat X calories per day and exercise, you will lose weight, and permanently"
Not really. "Calories" is just a physical quantity. It measures the energy that the combustion of something produces after putting it on a calorimeter. This needs a lot of energy.
So for example, you put sugar on a calorimeter, you give it (lots of) energy(making temp go up hundreds of Celsius degrees or pressure of several atmospheres) that you measure with precision to initialize the combustion and then you measure the energy the reaction produces.
So, according to the formula:
"calories" of sugar = energy out - energy in [of burning].
Note that in a calorimeter you don't care about the residue.
This is completely different to what metabolism does, in fact the living organism do not raise metabolites hundred of Celsius degrees, they use enzimes, and the byproducts and waste are completely different because living organisms have to get rid of them, so the byproducts have to be something the blood could carry.
So calories are just the UPPER LIMIT of what a metabolic reaction could give the body if the byproduct were to be the same, that are not.
Living organism break molecules link one by one in a series of chained reactions, which is itself called "methabolic reaction".
Every methabolic reaction is DIFFERENT. The human beings do not process pentose sugar the same as glucose sugar or methanol or an oil.
Some process require double or triple the steps of others, some are poisonous to your body(very acidic, or neuro transmitters, like alcohol or whatever) so the body tries to get rid of them as fast as he can.
Industrial processes completely disrupted the food we eat. Omega 6 oil replaced natural Omega 3 intake(as the animals we eat eat fodder), palm and hidrogenated oils anywhere and corn syrup sugar with no fiber(difficult to preserve in the fridge).