ways to improve reliability of an experiment

Few questionnaires measure test-retest reliability (mostly because of the logistics), but with the proliferation of online research, we should encourage more of this type of measure. When you correlate the two sets of measures, look for very high correlations ( r > 0.7) to establish retest reliability.Īs you can see, there’s some effort and planning involved: you need for participants to agree to answer the same questions twice. Later (by at least a few days, typically), have them answer the same questions again. Have a set of participants answer a set of questions (or perform a set of tasks). Test-Retest Reliabilityĭo customers provide the same set of responses when nothing about their experience or their attitudes has changed? You don’t want your measurement system to fluctuate when all other things are static. In that same study, we found that the average intra-rater reliability when judging problem severity was r =. You can also measure intra-rater reliability, whereby you correlate multiple scores from one observer.

Experts rating the severity of a problemįor example, we found that the average inter-rater reliability of usability experts rating the severity of usability problems was r =.

Evaluators identifying interface problems.

Kappa and the correlation coefficient are two common measures of inter-rater reliability.

Where there’s judgment there’s disagreement.Įven highly trained experts disagree among themselves when observing the same phenomenon. The extent to which raters or observers respond the same way to a given phenomenon is one measure of reliability. These four methods are the most common ways of measuring reliability for any empirical method or metric. But don’t let bad memories of testing allow you to dismiss their relevance to measuring the customer experience. Here are the four most common ways of measuring reliability for any empirical method or metric:īecause reliability comes from a history in educational measurement (think standardized tests), many of the terms we use to assess reliability come from the testing lexicon. In fact, before you can establish validity, you need to establish reliability.

Reliability is a measure of the consistency of a metric or a method.Įvery metric or method we use, including things like methods for uncovering usability problems in an interface and expert judgment, must be assessed for reliability.

Ways to improve reliability of an experiment