Reliability in surveys means that a measurement gives consistent results; validity means that it actually measures what you intend to measure. The most common way to assess reliability is Cronbach's alpha, a coefficient that shows how well the items measuring the same concept agree with each other. By common convention, an alpha of around 0.70 is considered acceptable and around 0.80 good. A high alpha, however, does not prove that the scale measures the right thing; validity has to be examined separately. In this article we explain both concepts with a dartboard analogy, show how to interpret Cronbach's alpha and what it all means for an engagement survey.
- Reliability shows whether a measurement gives consistent results; validity shows whether it actually measures what you intend to measure. An unreliable scale cannot be valid, but a reliable scale can be invalid.
- Cronbach's alpha is an internal consistency coefficient that shows how consistent the items measuring the same concept are with each other. By common convention around 0.70 is acceptable and around 0.80 is good; above 0.95 may point to redundant items.
- Alpha rises automatically as the number of items increases, so a high alpha on its own does not mean a good scale.
- In an in-house engagement survey, the reliability of the index items should be checked at every survey, and validated items should not be edited unless there is a strong reason.
The Dartboard Analogy: Reliability vs Validity
The easiest way to explain the two concepts is a dartboard. The bullseye is the concept you want to measure (for example employee engagement), and each throw is a measurement.
| What the throws look like | Reliability | Validity | Survey equivalent |
|---|---|---|---|
| Darts clustered at the center | High | High | Items are consistent and genuinely measure engagement |
| Darts clustered together, but in a corner far from the center | High | Low | Items are consistent but measure something else (for example general satisfaction) |
| Darts scattered across the board, averaging near the center | Low | Weak | Items are disconnected; the average looks reasonable by chance |
| Darts scattered and far from the center | Low | Low | The scale is neither consistent nor aimed at the right concept |
The key rule from the table: reliability is a precondition for validity, but it is not enough on its own. Darts landing close together does not mean they were thrown at the right spot. In survey terms: items receiving consistent responses does not mean they measure engagement.
Types of Reliability
Reliability is not a single thing. The types most often encountered in survey research are:
- Internal consistency: Whether items measuring the same concept receive consistent responses. We expect someone who agrees with "I am proud to work for this company" to also agree with "I would recommend this company as a place to work." Its most common indicator is Cronbach's alpha.
- Test-retest reliability: The same scale is given to the same people twice, a short interval apart (for example two weeks) during which the concept is not expected to change, and the relationship between the two measurements is examined. It is used when a scale is developed academically; it is rarely applied in company surveys, because employees' views change over time anyway.
- Split-half reliability: Items are split into two groups and the results of the two halves are compared. Because Cronbach's alpha in a sense summarizes the average of all possible split halves, alpha is generally preferred today.
- Inter-rater reliability: Whether two people assign the same open-ended comment to the same theme when coding. It matters in comment analysis; it does not apply to closed-ended items.
In company surveys, internal consistency is the type checked most often in practice, because it can be calculated from each survey's own data.
What Is Cronbach's Alpha?
Cronbach's alpha is a coefficient that shows the internal consistency of a scale made up of several items. It is named after Lee J. Cronbach, who popularized it in a paper published in 1951. It usually takes a value between 0 and 1; the closer to 1, the more consistent the items are with each other.
The Formula in Words
Alpha is calculated with this logic:
α = (k / (k − 1)) × (1 − sum of item variances / variance of the total score)
- k: The number of items in the scale.
- Sum of item variances: How spread out the responses to each item are, added up item by item.
- Variance of the total score: How spread out each employee's total score across all items is across employees.
The intuition: if the items measure the same concept, an employee who scores high on one item scores high on the others too. Total scores then differ clearly between employees, and the variance of the total score becomes much larger than the sum of the item variances. The ratio shrinks and alpha rises. If the items are unrelated, total scores bunch around the average and alpha falls.
You don't need to calculate alpha by hand; SPSS, R, Python and most survey analysis tools report it directly. What matters is interpreting the result correctly.
How to Interpret Cronbach's Alpha
Below are practical thresholds frequently used in the literature and in practice. The 0.70 threshold is usually attributed to Nunnally's work on psychometrics; the other ranges appear in various forms in textbooks.
| Alpha value | Common interpretation | What to do |
|---|---|---|
| Below 0.60 | Poor | The items may not measure the same concept; review the scale |
| 0.60–0.70 | Borderline | Usable for exploratory purposes; be cautious when making decisions |
| 0.70–0.80 | Acceptable | Sufficient in most cases for group comparisons |
| 0.80–0.90 | Good | The target range for indices in company surveys |
| 0.90–0.95 | Very good | The level needed for decisions about individuals |
| Above 0.95 | Possibly too high | Items may repeat each other; consider removing some |
These thresholds are not statistical laws but established conventions. In a short sub-dimension with few items, 0.68 can be reasonable, while in a 20-item scale 0.85 is unremarkable.
How Does the Number of Items Affect Alpha?
The most overlooked feature of alpha is that it is sensitive to the number of items. Even when the average relationship between items stays the same, alpha rises as items are added. The table below shows standardized alpha for two cases, with an average inter-item correlation of 0.30 and 0.50:
| Number of items | Alpha with average correlation 0.30 | Alpha with average correlation 0.50 |
|---|---|---|
| 3 | 0.56 | 0.75 |
| 5 | 0.68 | 0.83 |
| 10 | 0.81 | 0.91 |
| 20 | 0.90 | 0.95 |
Calculated with the standardized alpha formula: α = k × r / (1 + (k − 1) × r), where k is the number of items and r is the average inter-item correlation.
This table has two practical consequences. First, an alpha slightly below 0.70 in a short sub-dimension of 3–4 items is not a reason for alarm on its own. Second, seeing 0.90 in a 20-item scale does not prove the scale is good; the same alpha can come from many weakly related items. That is why, alongside alpha, you should also look at the average inter-item correlation and each item's relationship with the total score (the corrected item-total correlation).
Another limitation of alpha is that it assumes all items measure the concept equally strongly. When this assumption does not hold, alpha can understate reliability. For this reason, McDonald's omega is increasingly reported alongside alpha in academic work.
Types of Validity
Reliability answers "is it consistent?", validity answers "does it measure the right thing?". Validity is not captured by a single coefficient; it is assessed by bringing different kinds of evidence together.
Content Validity
Whether the items cover all the important aspects of the concept being measured. An index that measures engagement only through "intent to stay" items leaves out dimensions such as pride and discretionary effort. Content validity is usually established by subject-matter experts reviewing the items and comparing them with definitions in the literature. Face validity (whether the items look relevant to respondents) is a more superficial version of it.
Construct Validity and Factor Analysis
Whether the scale actually reflects the theoretical concept and its dimensions. Its most common tool is factor analysis:
- Exploratory factor analysis: Shows which groups the items naturally fall into. For example, "manager" items are expected to load on one factor and "career" items on another.
- Confirmatory factor analysis: Tests whether a predefined structure (for example a three-dimensional engagement model) fits the data.
Construct validity has two more complementary aspects. Convergent validity means the scale correlates strongly with other scales measuring similar concepts; discriminant validity means it is not more related than it should be to different concepts. If your engagement scale gives almost the same result as a job satisfaction scale, it becomes hard to claim you are measuring two different things.
Criterion and Predictive Validity
Whether the scale is actually related to an external criterion it is expected to relate to. A relationship with a criterion measured at the same time is called concurrent validity; a relationship with a future outcome is called predictive validity. For example, if units with low engagement scores show higher voluntary turnover over the following 12 months, that is strong evidence of the index's predictive validity. Organizations can run this analysis with their own HR data; the only condition is to work at unit level so that anonymity is protected.
What Does This Mean for an In-House Engagement Survey?
Academic scale development and an in-house survey are not the same thing, but the same principles apply.
Evaluate the Index Items Separately
An engagement survey contains two types of items: the small number of items that make up the engagement index, and items that measure the areas affecting engagement (manager, career, recognition, etc.). Reliability matters most for the index, because year-over-year comparisons, unit rankings and key driver analysis all rely on that single number. Calculate the alpha of the index items at every survey; if alpha drops noticeably, investigate why (a translation issue, a newly added employee group, a changed item).
Don't Edit Validated Items
Scientific scales such as the UWES were developed by testing items one by one. Changing an item's wording "to make it clearer" can invalidate the scale's reliability and validity evidence for your use. It also removes the possibility of comparing your results with other studies that use the same scale. If you use a scientific scale, use the version that has been adapted to your language and tested for reliability, as is and in line with its terms of use. We describe the scale's dimensions in our Utrecht Work Engagement Scale (UWES) article and the different models in Gallup Q12 and employee engagement models.
Keep the Scale Consistent Across the Survey
Using different response scales within the same index, or phrasing some items negatively and others positively, can artificially lower internal consistency. We explain why the index items should use the same 5-point agreement scale in what is a Likert scale.
Run a Pilot
Newly written items, or items translated into a new language, should be tested with a small group before being sent to the whole organization. A practical pilot includes these steps:
- Give the items to a group of 30–50 people from different units and education levels.
- Talk one-to-one with a few of them and ask how they understood each item.
- Calculate the alpha and item-total correlations of the index items.
- Rewrite or remove items that relate very weakly to the total, attract too many "Neutral" responses or are understood differently.
We explain how to write items and structure the survey step by step in our guide how to create an employee engagement survey.
Reliability Does Not Reveal Response Bias
A group of employees who answer "Agree" to every item without thinking raises alpha but distorts the accuracy of the measurement. Biases such as social desirability or acquiescence produce consistent errors, so they do not show up in a reliability coefficient. We cover these biases in survey response bias.
Common Mistakes
- Treating a high alpha as proof of validity: Alpha only shows consistency. Very similar items produce a high alpha but may measure only a narrow part of the concept.
- Calculating a single alpha for the whole survey: The combined alpha of 50 items on different topics such as pay, manager and workload will be high but meaningless. Alpha should be calculated separately for each scale or sub-dimension that measures one concept.
- Calculating without reversing negatively worded items: If negative items such as "I often think about leaving my job" are not reverse-coded before the calculation, alpha comes out low or even negative.
- Counting an alpha above 0.95 as a success: This level often points to items asking the same thing in different words. It lengthens the survey needlessly and tires employees.
- Adding items to raise alpha: Alpha increases automatically with the number of items. A longer survey is not a better measurement.
- Freely editing the items of scientific scales: Wording changes, shortening and dropping items can invalidate the scale's established properties.
- Interpreting alpha in small groups: An alpha calculated for a 15-person team is very unstable. Run reliability analysis at the organization level or for large units.
- Testing reliability once and forgetting it: Reliability can change as the organization grows, as new employee groups (for example field employees) are added or as the survey is translated into a new language. Check it at every survey.
A reliable and valid scale is the foundation of trust in survey results. When your scale is sound, you can answer managers' question "Do these results reflect reality?" with evidence. The other key factor affecting the statistical precision of results is the number of responses; we cover it in sample size and margin of error.
Frequently asked questions
What is the difference between reliability and validity in surveys?
Reliability is consistency: items measuring the same concept give similar results, and the result does not change much when the measurement is repeated. Validity is whether the measurement actually captures the intended concept. A scale can be consistent but measure the wrong thing, which is why the two are checked separately.
What is Cronbach's alpha?
Cronbach's alpha is a coefficient that shows the internal consistency of a scale made up of several items, and it usually takes a value between 0 and 1. If the items measure the same concept, they receive related responses and alpha rises. It is named after Lee J. Cronbach, who popularized the coefficient in 1951.
What is a good Cronbach's alpha value?
Under commonly used rules of thumb, around 0.70 is acceptable, around 0.80 is good and around 0.90 is very good. Values above 0.95 may indicate that items repeat each other. These thresholds are not laws but conventions that should be interpreted in light of the scale's purpose and number of items.
Does a high Cronbach's alpha mean the scale is valid?
No. Alpha only shows that the items are consistent with each other; it does not show that they measure the right concept. Alpha also rises automatically as more items are added. Validity requires separate evidence such as content review, factor analysis and relationships with external criteria.
