Technology changes that game.
Grains are the core of the human diet, and have been since the dawn of civilization. But they are only edible because of civilization - the grinding, fermentation, and cooking breaking them down into something our bodies can use.
In the US and other wealthy countries, we might feed grain to livestock, but it's also often parts of grain that we can't really eat (hulls, etc). And in poorer countries, livestock are often open-grazed on grasses that are inedible for us. And non-grazers like hogs often eat garbage, food waste that we don't consume.
When farming, the questions are even more complex. The livestock generates manure that fertilizes the fields, so much of that "lost" energy is recovered.