500 calories is not an unreasonable amount of calories at all for a lunch type meal.
Even if you had 500 for breakfast, then lunch, then 1000 for dinner, that's only 2k which pretty close to a typical BMR for an adult male.
The problem is the quality of nutrients. If all those 2000 calories were packed in with lots of fibre, and had a good ratio of protein/healthy fats/carbs that's likely to be enough to keep you from snacking a bunch in the evenings and between meals. If it's empty calories, you're likely to end up eating even more.
Also, bread is not as bad as you portray it. Proper brown whole grain bread can have as much as 20% protein as a percentage of total calorie content. That's a very high relative protein content compared to most foods really, other than lean meats and eggs.