Most riding trousers are graded from a straight block: as the size goes up, the waist and hips grow together in fixed proportion. If your hips are fuller relative to your waist, you size up until the hips fit and inherit a waistband that now stands away at the back. On a bike that gap is more than cosmetic, because wind funnels straight up your back and the armor shifts out of position when you lean forward.
Sizing down to close the waist just trades one problem for another: now the hips and thighs bind, the knee armor rides off the joint, and you are uncomfortable in a different place. The waist gap is a shape problem, and shape problems need shape solutions.