As an extension of the High Efficiency Video Coding standard, Scalable High-efficiency Video Coding (SHVC) has been developed to satisfy the challenges posed by heterogeneous environments. A layer-based scheme is employed to support various forms of scalability, but this results in significantly increased computational complexity. A fast Coding Unit (CU) size decision algorithm is proposed to alleviate the additional computational requirement of the Enhancement Layers (ELs). The inter-layer dependency and the spatial correlation are jointly used to skip and early terminate the CU size decision process. In addition, the quantitive relationship between the texture homogeneity and the AC coefficient energy is investigated to further narrow the depth range to be evaluated. Experimental results show that the proposed scheme saves encoding time by up to 72% compared with the SHM 12.0 implementation, with negligible degradation in Rate-Distortion performance.