Saliency map estimation in computer vision aims to estimate the locations where people gaze in images. Since people tend to look at objects in images, the parameters of the model pre-trained on ImageNet for image classification are useful for the saliency map estimation. However, there is no research on the relationship between the image classification accuracy and the performance of the saliency map estimation. In this study, it is shown that there is a strong correlation between image classification accuracy and saliency map estimation accuracy. The authors also investigated the effective architecture based on multi-scale images and the up-sampling layers to refine the saliency-map resolution. The model achieved the state-of-the-art accuracy on the PASCAL-S, OSIE, and MIT1003 datasets. In the MIT saliency benchmark, the model achieved the best performance in some metrics and competitive results in the other metrics.