It's important to point out why games use immediate mode GUIs: 1. The GUI needs to be overlaid on the game image (OpenGL/DirectX). This is difficult with traditional GUIs like QT. 2. The GUI needs to be updated in sync with the game, again, it's difficult to integrate traditional GUIs event loops into the game loop, especially with stuff like double/triple buffering. 3. The GUI needs to be as fast as possible, games…
Every game I've worked on didn't actually use an IMGUI for anything you'd ever show a player; (unless it's a style of game that really doesn't need much UI), it's just custom gui implementations. A lot of modern games actually put the UI in with the scene graph of the game itself (IE: Unity's new GUI system)
Internal tools tend to be IMGUI, Winforms, or WPF from what I've seen. Essentially whatever the original author is most productive in. No need to be super performant because your only users are other devs with really high end systems and rarely an actual need for a constant 30/60fps.
Your engine likely already has a graph structure, event system, and input handling. The core pieces of a retained mode UI system are already there.