The coherent robust control problem for a class of linear quantum passive systems with model uncertainties is considered in this study, where both the plant and the controller are described by quantum stochastic differential equations (QSDEs). In the framework of language, the model uncertainties are translated into the uncertainties in the Hamiltonian, the coupling operators, and the scattering matrices. Based on the quantum bounded real lemma, the robust controller design of the original uncertain plant is transformed into the controller design of a new scaled system without model uncertainties. Further, the latter is transformed into the solving problem of a couple of Riccati equations and therefore the numerical solutions of the coefficient matrices of QSDEs associated with the controller dynamics are obtained. Finally, numerical simulations on an optical system coupled to three optical channels are performed to verify the effectiveness of the method proposed in this study.