Skip to content

Quality Metrics Reference

Askalot evaluates survey data quality across two complementary dimensions grounded in the Total Survey Error framework (Groves et al., 2009):

  • Sample Representativeness — Do respondent demographics match the target population? Quantifies coverage and nonresponse error.
  • Response Quality — Are survey questions producing informative, diverse answers? Detects measurement error.

Both dimensions must be acceptable for data to be fit for analysis.

Data quality vs. instrument design quality

This reference covers the data quality of responses you have already collected. To assess a questionnaire's design quality before fielding — economy, gate density, verification coverage, path diversity, and respondent burden — see the Instrument Health scorecard, which grades the QML itself rather than any collected data.

Sample Representativeness Metrics

These metrics compare the actual demographic distribution of survey respondents against target distributions defined in a sampling strategy.

RMSE (Root Mean Square Error)

Overall deviation between sample and target distributions across all demographic categories.

\[ \text{RMSE} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (a_i - t_i)^2} \]

Where \(a_i\) is the actual proportion and \(t_i\) is the target proportion for category \(i\).

RMSE penalizes large deviations more than many small ones due to the squared term. A single demographic category that is 10 percentage points off-target contributes more to RMSE than five categories each off by 2 points.

RMSE Interpretation
< 0.02 Excellent — sample closely matches targets
0.02–0.05 Acceptable — minor deviations, weighting should correct
0.05–0.10 Concerning — significant deviations, investigate causes
> 0.10 Problematic — structural sampling issues, weighting may not suffice

MAE (Mean Absolute Error)

Average per-category deviation. More robust to outlier categories than RMSE.

\[ \text{MAE} = \frac{1}{N} \sum_{i=1}^{N} |a_i - t_i| \]

When MAE is much smaller than RMSE, one or two categories are severely off while others are fine. When MAE ≈ RMSE, deviations are uniformly distributed.

Chi-Square Test

Tests whether observed deviations from targets are statistically significant versus random sampling noise.

\[ \chi^2 = \sum_{i=1}^{k} \frac{(O_i - E_i)^2}{E_i} \]

Where \(O_i\) is the observed count and \(E_i\) is the expected count based on target proportions.

Sample Size Sensitivity

With large samples (N > 500), chi-square will flag almost any deviation as statistically significant, even if the practical magnitude is trivial. Use RMSE/MAE to assess practical significance, and chi-square for statistical significance. A significant chi-square with RMSE < 0.02 may not be actionable.

Max Deviation

The largest single-category gap between actual and target proportions. Identifies the most problematic demographic category.

Max Deviation Interpretation
< 0.05 All categories within acceptable range
0.05–0.10 One category moderately off — likely correctable by weighting
> 0.10 Structural gap — at least one group is substantially misrepresented

Composite Quality Score

The overall quality score (0–1) is derived from the composite error via score = 1 / (1 + 5 × error), where the error is the weighted average of each factor's RMSE (or the worst-case factor under max aggregation). Higher is better. The Balansor quality UI colors the score at exactly these two thresholds:

Score Rating Interpretation
> 0.8 Good (green) Sample adequately represents target population
0.6–0.8 Acceptable (yellow) Some factors off-target, weighting recommended
< 0.6 Poor (red) Significant representativeness issues — investigate root causes before relying on estimates

Per-Factor Review

A high overall score can mask one poorly represented factor. Always review the per-factor breakdown — the factor with the lowest score often drives the most bias in survey estimates.

Weighting Diagnostics

After raking (iterative proportional fitting) produces a Silver dataset, these metrics evaluate whether weighting improved representativeness without introducing excessive variance.

Design Effect (DEFF)

Variance inflation from unequal weighting (Kish, 1965):

\[ \text{DEFF} = 1 + \text{CV}^2_w \]

Where \(\text{CV}_w\) is the coefficient of variation of the weights.

DEFF Interpretation
< 1.5 Minimal impact on precision
1.5–2.0 Moderate precision loss — acceptable for most studies
2.0–3.0 Substantial — review weight distribution
> 3.0 Severe — weighting compensating for structural sample problems

Effective Sample Size

The equivalent unweighted sample size after accounting for the design effect:

\[ n_{\text{eff}} = \frac{n}{\text{DEFF}} = \frac{(\sum w_i)^2}{\sum w_i^2} \]
\(n_{\text{eff}} / n\) Interpretation
> 0.80 Minimal efficiency loss from weighting
0.50–0.80 Moderate loss — some groups heavily weighted
< 0.50 Severe loss — weighting compensating too aggressively

Weight Quality Benchmarks

Metric Good Acceptable Problematic
Weight CV < 0.5 0.5–1.0 > 1.0
Max/min weight ratio < 4:1 4:1–6:1 > 6:1

Interpreting Design Effect

A design effect of 1.5 means your effective sample size is 67% of your actual sample size. Weights exceeding a 4:1 ratio (max/min) signal structural problems that weighting alone cannot fix — consider adjusting recruitment strategy instead.

Completion Rate

Proportion of a campaign's surveys that are complete, out of all surveys created for it, invited or not. Complete means the respondent answered more than 80% of the questions that applied to them, the same rule Completion by Mode uses. A respondent who pressed Finish after a couple of questions has a closed survey but is not counted as complete. Provides a high-level measure of fieldwork success and potential nonresponse bias.

Completion by Mode

The campaign completion rate divides by every survey created, invited or not, so it cannot compare channels: a respondent only chooses the web page or their AI assistant once they start. Completion by mode therefore divides by starters, surveys whose first question was shown, separately for each survey mode (direct, ai_assisted, field_interview). A consent screen is not a question, so a respondent who consented in the AI assistant and went no further is not a starter.

Each starter is classified by the AAPOR item thresholds on its share of applicable questions answered:

Disposition Answered share
Complete above 80%
Partial 50% to 80%
Break-off below 50%

Applicable questions are those the respondent reached, plus those still ahead on their path whose display condition is true or not yet decidable from the answers given. A survey that ended normally is therefore measured by its own completeness, while a break-off is measured against what still lay ahead of it. Comments carry no answer and never count.

The completion rate is complete starters divided by all starters, reported with a 95% Wilson confidence interval. A mode with fewer than 30 starters reports its counts without a rate, because a rate over a handful of cases reads as a finding when it is noise.

Response Quality Metrics

These metrics evaluate survey responses themselves — independent of sampling strategy.

Per-Question Metrics

Normalized Entropy (Categorical Questions)

Measures answer diversity for radio and dropdown questions. Based on Shannon entropy (Shannon, 1948), normalized to a 0–1 scale:

\[ H_{\text{norm}} = \frac{H}{\log k} = \frac{-\sum_{i=1}^{k} p_i \log p_i}{\log k} \]

Where \(p_i\) is the proportion of responses selecting option \(i\) and \(k\) is the number of available options.

\(H_{\text{norm}}\) Interpretation
> 0.7 Good diversity — responses spread across options
0.3–0.7 Moderate — some options dominate
< 0.3 Low diversity — responses concentrated on few options (ceiling/floor effect)

A value of 1.0 means perfectly uniform distribution. A value approaching 0.0 means nearly all respondents chose the same option.

Effective Number of Answers

The Hill number of order 1 (Hill, 1973) — how many options carry meaningful weight:

\[ N_{\text{eff}} = \exp(H) = \exp\!\left(-\sum_{i=1}^{k} p_i \log p_i\right) \]

If a question has 5 options but \(N_{\text{eff}} = 2.1\), only about 2 options are receiving substantial responses. More intuitive than raw entropy for identifying underutilized response scales.

Option Usage Ratio

Fraction of available options that received at least one response:

\[ \text{OUR} = \frac{|\{i : c_i > 0\}|}{k} \]

A ratio below 0.5 suggests some options are irrelevant and could be merged or removed.

Item Variance (Numeric Questions)

Response spread for editbox, slider and range questions; a range answer is measured at each end (lowest and highest value) separately. For weighted (Silver) datasets, uses weighted variance:

\[ \sigma^2_w = \frac{\sum_{i=1}^{n} w_i (x_i - \bar{x}_w)^2}{\sum_{i=1}^{n} w_i} \]

A variance of zero means all respondents gave the same answer — a strong signal of a poorly calibrated scale or ceiling/floor effect.

Balansor's per-question table shows this measure as the standard deviation (σ, the square root of the variance), so it reads in the question's own unit — a property-value question's spread shows as 787K, not as a variance in dollars squared. The raw variance stays in the API payload as item_variance.

Item Non-Response Rate

Proportion of respondents who skipped the question:

\[ \text{NRR} = \frac{n_{\text{missing}}}{n_{\text{total}}} \]
Non-Response Rate Interpretation
< 5% Normal
5–10% Monitor — may indicate sensitivity or confusion
10–20% Investigate wording, placement, or sensitivity
> 20% Redesign the question

Increasing non-response rates toward the end of the survey typically indicate respondent fatigue.

Group Metrics

For question groups and matrix questions with multiple sub-items, and, for straightlining, for standalone questions that share one answer scale.

Straightlining Score

Proportion of respondents giving identical answers to every item of a set (Krosnick, 1991). A set is either:

  • the sub-items of one question group or matrix question, or
  • three or more standalone questions with the same control and exactly the same answer options (codes and labels), when those options number at least three. These are listed as Shared scale, with their question IDs. A two-option answer set such as Yes/No is left out: unrelated questions share it all the time, so answering "No" to each of them says nothing about attention.

Answers only count as identical within one set: picking option 1 on "department" and option 1 on "reason for leaving" is not straightlining.

\[ S = \frac{n_{\text{straightliners}}}{n_{\text{respondents}}} \]
Straightlining Interpretation
< 5% Normal — no action needed
5–15% Investigate — may indicate confusing question design
> 15% Structural problem — redesign the matrix or add attention checks

Weight Independence

Straightlining is computed from unweighted respondent counts, since weighting adjusts demographic balance but does not affect individual response patterns.

Not assessed

A questionnaire with no question group, no matrix question and no three questions sharing one scale has nothing to compare. Straightlining is then shown as not assessed, not as a clean result. The same applies to a single set that no respondent answered in full, which happens when routing skips some of its questions: it shows a dash rather than 0%.

Cronbach's Alpha (Internal Consistency)

Measures whether sub-items in a group measure the same underlying construct (Cronbach, 1951). Requires 3 or more sub-items:

\[ \alpha = \frac{k}{k-1} \left(1 - \frac{\sum_{i=1}^{k} \sigma^2_i}{\sigma^2_{\text{total}}}\right) \]

Where \(k\) is the number of sub-items, \(\sigma^2_i\) is the variance of sub-item \(i\), and \(\sigma^2_{\text{total}}\) is the variance of the total score.

\(\alpha\) Interpretation
> 0.8 Good internal consistency
0.6–0.8 Acceptable
< 0.6 Poor — sub-items may not measure the same construct

Dataset-Level Aggregates

Metric Formula Concern Threshold
Mean Normalized Entropy Average \(H_{\text{norm}}\) across categorical questions < 0.3
Acquiescence Bias Index Proportion of "agree" responses across Likert items > 0.6
Overall Non-Response Rate Mean skip rate across all questions > 0.15

Acquiescence Bias Index

Detects systematic tendency to agree with statements regardless of content. Computed only for Likert-type agree/disagree scales:

\[ \text{ABI} = \frac{\sum \text{agree responses}}{\sum \text{total Likert responses}} \]

A value of 0.5 indicates no bias (balanced agreement/disagreement). Values above 0.6 suggest respondents are agreeing regardless of question content — consider adding reverse-coded items.

Weight Awareness

For Silver (weighted) datasets, entropy, variance, and acquiescence metrics automatically use the weight column. Straightlining and speeder detection use unweighted respondent counts since weighting doesn't affect response patterns.

Speeder Detection

Identifies respondents who answered unusually fast, suggesting inattentive or automated responding (Kunz & Hadler, 2020):

\[ \text{threshold} = \frac{\text{median answering time}}{3} \]

Respondents below the threshold are flagged as speeders.

Answering time, not elapsed time. Askalot times each question individually — from the moment it is presented to the moment the answer comes back — and sums those spans. This is not the wall-clock gap between a survey being created and being submitted, and the difference matters: a respondent invited on Monday who sits down on Thursday spans three days while answering at a perfectly ordinary pace. Because the threshold is a median over the whole cohort, a handful of such respondents would lift it until the people who simply got on with it are the ones flagged.

Time spent on a question the respondent then navigated back from still counts — it was genuinely spent — even though it does not count as an answer attempt.

Minimum Sample Requirement

Speeder detection requires at least 3 surveys with usable timing to compute a meaningful median. With fewer, the check is skipped.

Synthetic responses are excluded

Answers produced by synthetic response generation are tagged as machine-generated and left out of the median entirely. A generated survey answers in milliseconds; counted in, it would either flag nothing at all or — in a campaign mixing real and generated responses — make the generated surveys the speeders and hide the human ones. Values prefilled from a respondent's own record stay in: they take no time by construction, and that is an honest part of the answering effort.

The speeder rate is a share of the timed cohort, not of your dataset

Because of the exclusions above, some responses carry no usable answering time — machine-filled surveys, and any response that recorded no completed question. Those are reported separately and are neither speeders nor cleared: nobody could measure them either way. The percentage is therefore taken over the respondents whose time could be measured, and that count is shown beside it.

This matters when the two differ. In a dataset of 1,000 responses where only 400 were timed, 20 speeders is 5% of the timed cohort, not 2% of the dataset — and 5% is the point at which the guidance below suggests acting. Reading it against the larger number would understate the problem by exactly the responses that were never assessed.

Multi-Flag Aggregation (Baseball Rule)

No single quality check should be used alone to exclude respondents. The baseball rule (CloudResearch, 2024) aggregates multiple independent quality flags per respondent and recommends exclusion only when 3 or more flags co-occur:

Flag Trigger
Speeder Answering time below ⅓ of median
Straightliner Identical responses across every item of any straightlining set (a question group, or three or more questions sharing one scale)
High non-response More than 30% of questions skipped

This multi-indicator approach reduces false positives — a respondent who happens to be fast but answers thoughtfully is not penalized.

Interpreting Results

Common Patterns and Actions

Observation Possible Cause Action
Low entropy (< 30%) on a question Ceiling/floor effect or poorly balanced options Review option wording; consider splitting dominant category
High non-response (> 15%) on a question Confusing or sensitive question Simplify wording or make optionality explicit
High straightlining (> 15%) in a group Inattentive responding or undifferentiated sub-items Add attention checks; vary scale anchors; shorten battery
Low Cronbach's alpha (< 0.5) Sub-items don't measure the same construct Review item selection; consider removing weak items
High acquiescence (> 60%) Yea-saying bias Add reverse-coded items; vary scale direction
Low option usage (< 50%) Unused options are irrelevant Merge or remove rarely-chosen options
High RMSE (> 0.05) Systematic sampling bias Adjust quotas, diversify recruitment channels
Large design effect (> 2.0) Extreme weight adjustments Improve sample balance before relying on weighting
Speeders flagged (> 5% of the timed cohort) Inattentive or incentive-driven responding Review incentive structure; add attention checks
Multi-flag respondents Multiple quality issues co-occur Exclude from analysis; document exclusion criteria

Two Dimensions, One Picture

Sample representativeness and response quality are independent dimensions. A dataset can have excellent representativeness but poor response quality, or vice versa.

Representativeness Response Quality Assessment
Good Good Data is fit for analysis
Good Poor Sample matches targets but responses are unreliable — investigate measurement issues
Poor Good Responses are genuine but sample is biased — apply weighting or adjust recruitment
Poor Poor Both structural and measurement problems — address recruitment and questionnaire design

Methodological Foundation

The quality analysis framework is grounded in established survey research methodology:

Framework Application
Total Survey Error (Groves et al., 2009) Organizes metrics by error source: coverage, sampling, nonresponse, measurement
AAPOR Standards Response rate calculations, disposition codes, quality reporting
Kish (1965) Design effects, effective sample size under complex designs
Krosnick (1991) Satisficing theory — straightlining, acquiescence as response quality indicators
Shannon (1948) Information entropy as a measure of response diversity
Cronbach (1951) Internal consistency reliability for multi-item scales
Hill (1973) Effective number of species (answers) — Hill numbers for diversity measurement
Kalton & Flores-Cervantes (2003) Weighting methods — raking, calibration, weight trimming
Kunz & Hadler (2020) Web paradata — response time analysis, speeder detection thresholds
CloudResearch (2024) Multi-flag quality aggregation — baseball rule for respondent exclusion