Fascinating question - and not one whose answer is known.
Fiber is good for microbes merely by the virtue that humans lack the glycan-degrading enzymes that are necessary to break it down, so it reaches the colon intact where the microbes can eat it. Humans can digest starch (though it can be physically and chemically modified to be harder to digest) but not the hundreds of other types of fibers that are found in a traditional diet.
I think there is good research that shows that higher fiber diets are associated with lower risk of developing a range of metabolic and immune pathologies, but the particular mechanistic linkages are so subtle that it will require absolutely massive studies to identify them. In general, we know that humans used to eat a much higher fiber diet (e.g. the Hadza people eat 70-150 grams of fiber a day), and we believe that produces a much healthier microbial composition.
The paper you cited is really interesting! I haven't read it - but the overall idea that the mucus lining of the gut can be degraded by microbes who are sourcing carbon, energy, and nitrogen from it is well established.
I think there is consensus that some amount of gut barrier integrity is due to microbial signals. This occurs in two ways - 1) our epithelial cells sense microbial products (proteins, carbohydrates, etc.) and respond by tightening the junctions between them. The overall idea being that you want to keep the bacteria in the colon, but you must balance some level of nutrient flow. 2) The goblet cells which produce mucus in the gut, respond to microbial signals to increase or decrease their mucus production.
There is a lot of research going on trying to understand how certain diets cause defective mucus production and in turn how that can allow microbes to get to close to the epithelial lining (usually the mucus is ~100 microns thick) which results in inflammation.
Recent evidence of another function of microbial activity in the gut of hibernating animals. In short, it appears that they help the host supply enough nitrogen for maintaining muscle mass during hibernation: https://www.science.org/doi/10.1126/science.abh2950