the atmospheric drag at perigee happens perpendicular to the object speed, which is tangential to the gravity field, so the object doesn't get shifts on it's vertical speed, only it's energy, having less to convert in potential energy (i.e. apogee height)
in the thereabout of perigee you have two arms, one inbound and one outbound. the horizontal component of the drag for both is all in the direction against the travel and slow the craft without changing the vertical component of its speed
the vertical component of the drag (for as long as the craft has the energy to get's outside of the dense atmosphere) kind of cancels out, as the drag force inbound pushes against your down velocity vector and the drag force outbound pushes against your up velocity vector.
there is however a small energy loss due to the fact that the journey trough the atmosphere isn't perfectly simmetrical, since you're losing speed along all the way, the inbound leg is shorter than the outbound leg, causing the vertical component of the drag going outside to slightly exceed the component of the vertical drag coming in - so your perigee does change little by litte due this (and other more nuanced effects, but we're painting with a large brush here) - this asymmetry gets stronger the more you stay in the atmosphere and the furthest you dip in the dense area, until you ultimately don't have the energy to escape anymore.
the fact that the atmosphere isn't on/off like that but has a long boundary layer stretching far from the dense parts kinda complicates things a little more, because drag is applied the whole circle, but not by that much.