Background: A number of approaches leverage design diversity to tolerate software design faults in service-oriented applications. The use of design diversity depends on the assumption that functionally equivalent services, i.e., variant services, rarely fail on the same input case. However, there are no directives to assess whether variant services are actually diverse and fail on disjoint subsets of the input space. Aim: To provide proper assessment of service diversity in order to achieve a high level of reliability by employing either a diversity-based solution with the variant services or a single service that exhibits higher reliability than would be the case if design diversity was adopted. Method: We propose an experimental setup that encompasses (i) a set of directives to organize the preparation and execution of the experiment to investigate service diversity; (ii) investigation of whether variant services are actually diverse by using statistical tests; and (iii) an analysis of if and by how much the reliability of a diversity-based solution that leverages voters is an improvement over one that uses a single service. We evaluated the applicability and usefulness of the proposed experimental setup by employing it to assess diversity of variant services adhering to four different requirements specifications. For each specification, we analysed three different services. Results: We found that the proposed directives were effective for the purposes of this assessment. Assessment results demonstrated that services implementing the four requirements specifications are actually diverse at a 0.05 significance level. For two of the specifications, coincident failures of two or more services are infrequent enough to promote gains in overall system reliability. Conclusions: Our findings reveal threats to the effectiveness of service-oriented applications that relies on design diversity to tolerate faults. This in turn reinforces the usefulness of our experimental setup, which supports researchers' decision making when selecting variants for building fault-tolerant service applications.