Quantitative microwave holography is a recently developed direct inversion imaging algorithm. This method shows promise in medical imaging due to its low computational cost and ability to quantitatively estimate the tissue permittivity. However, it faces challenges regarding nonphysical permittivity values (i.e., real relative permittivity less than 1, and imaginary relative permittivity greater than 0) in the reconstructed images. Here, we propose an optimization method capable of removing nonphysical values. This approach is demonstrated in a tissue imaging experiment. Additionally, we discuss the impact of phase wrapping on the Rytov's approximation-based version of quantitative microwave holography.