Skip to content

AI-Assisted Result Analysis

Balansor's AI Analyst evaluates a completed campaign along two fronts and combines them into a single verdict:

  • Statistical data quality — sample representativeness (does the sample match the target population?) and response quality (are questions producing informative, diverse answers?).
  • Research-goal answerability — does the collected data actually answer the research questions, KPIs, and success criteria recorded in the project's Research Brief? This closes the goal → KPI → question chain established when the questionnaire was designed: every research goal declares a KPI and how it is measured, and the Analyst checks whether the data collected against each one is sufficient to conclude it.

The Analyst delegates the first front to a Quality Analyst sub-agent and the second to a Research Evaluator sub-agent, then synthesizes both into a structured report with actionable recommendations.

Overview

After collecting survey responses, researchers need to answer two fundamental questions before trusting their data:

  1. Is the sample representative? — Do respondent demographics match the target population defined in the sampling strategy?
  2. Are the responses reliable? — Do answer patterns indicate genuine engagement, or are there signs of satisficing, inattention, or bias?

Balansor addresses both questions through computed metrics and an AI analyst agent that interprets results in methodological context.

flowchart LR
    subgraph input["Input"]
        b[Bronze Dataset]
        s[Sampling Strategy]
    end

    subgraph metrics["Quality Metrics"]
        r[Sample Representativeness]
        q[Response Quality]
    end

    subgraph output["Output"]
        w[Silver Dataset]
        rpt[Analyst Report]
    end

    b --> r
    s --> r
    b --> q
    r --> rpt
    q --> rpt
    b -->|"code open-ends, then weight"| w
    w --> r

Quality Metrics

The platform computes quality metrics across two dimensions, grounded in established survey methodology (Groves, Kish, Krosnick, Shannon, Cronbach):

Dimension Key Metrics
Sample Representativeness RMSE, MAE, Chi-Square, Max Deviation, Composite Quality Score
Weighting Diagnostics Design Effect (DEFF), Effective Sample Size, Weight CV, Completion Rate
Response Quality Normalized Entropy, Straightlining Score, Cronbach's Alpha, Acquiescence Bias, Speeder Detection, Multi-Flag Aggregation

See the Quality Metrics Reference for complete definitions, mathematical formulas, interpretation thresholds, and recommended actions.

The AI Analyst Agent

The AI Analyst interprets quality metrics in methodological context and checks the results against the Research Brief, producing a structured report. It runs as a background task in Balansor, powered by the askalot_ai agent framework: the top-level analyst orchestrates a Quality Analyst sub-agent (statistical data quality) and a Research Evaluator sub-agent (research-goal answerability, via the read-only answerability chain over the Brief), then a Summarizer stitches their findings into the final report.

How It Works

sequenceDiagram
    participant U as Researcher
    participant B as Balansor
    participant R as Redis
    participant A as Analyst Agent

    U->>B: Start Quality Analysis
    B->>R: Create progress queue
    B->>B: Pre-fetch all quality data
    Note over B: Bundle Representativeness story,<br/>current Strategy, the excluded set,<br/>recovery + design effect
    B->>A: Send pre-computed metrics + brief context
    A->>A: Quality Analyst — statistical quality
    A->>A: Research Evaluator — answerability vs Brief
    A->>R: Emit progress events

    A->>A: Summarizer stitches the findings
    A->>R: Emit completed report

    U->>B: Poll for status
    B->>R: Read progress
    B-->>U: Display report
  1. Credential resolution: The agent resolves AI provider credentials from user settings, organization configuration, or environment variables (in that priority order)
  2. Data pre-fetch: All quality metrics are gathered in Python via direct repository and analysis calls before the agent starts, so the sub-agents interpret ready data instead of re-fetching it
  3. Multi-turn analysis: The analyst runs the analyze-quality flow, delegating to the Quality Analyst and Research Evaluator sub-agents; the run is bounded by a wall-clock timeout rather than a fixed turn count, using provider-native streaming to avoid timeouts
  4. Progress tracking: Redis Streams provide cross-worker progress reporting — safe across Gunicorn's multiple worker processes
  5. Report generation: A separate summarizer agent synthesizes the sub-agents' findings into a structured markdown report

Agent Profile

Property Value
Flow analyze-quality
Sub-agents Quality Analyst, Research Evaluator
Model tier High (Claude Opus)
Run bound Wall-clock timeout (multi-turn — headroom for both sub-agent delegations plus synthesis)
Knowledge base Data quality assessment, weighting methodology, response quality metrics, research-brief answerability

The analyst's skill body encodes survey methodology expertise from AAPOR standards, Kish (1965), Groves et al. (2009), Kalton & Flores-Cervantes (2003), Krosnick (1991), and ESOMAR guidelines — including specific formulas, thresholds, and decision frameworks for interpreting design effects, straightlining scores, speeder flags, and multi-flag aggregation.

Report Structure

The analyst produces a report with five sections:

1. Executive Summary 2–3 sentences: overall quality assessment, fitness for purpose, and the single most important recommendation.

2. Sample Representativeness

  • Overall quality score with interpretation
  • RMSE and MAE values with context
  • Per-factor analysis identifying which demographics match targets and which deviate
  • Specific numbers: "Age 18–24 is over-represented by 8pp (32% actual vs 24% target)"

3. Weighting Assessment (if Silver dataset exists)

  • Quality improvement percentages from raking
  • Per-factor improvement breakdown
  • Weighting diagnostics: design effect (DEFF), effective sample size (ESS), weight CV, and weight ratio interpretation
  • Flags for any factors that worsened after weighting
  • Assessment of whether weighting was effective or structural changes are needed

4. Response Quality

  • Speeder detection: number flagged, percentage of sample, median answering time
  • Straightlining: groups with high scores (> 0.05)
  • Item non-response: questions exceeding 5% missing rates
  • Acquiescence bias index (if Likert scales present)
  • Cronbach's alpha for group reliability
  • Low entropy questions with interpretation
  • Multi-flag respondents (baseball rule) — exclusion recommendation if any

5. Key Findings 3–5 specific, data-driven findings. Each references actual numbers from quality metrics.

6. Recommendations 2–4 prioritized, actionable recommendations. Each includes what to change, why (linked to specific finding), and expected impact.

Grounding in Survey Methodology

When the Analyst discusses total survey error, nonresponse bias, raking convergence, design effects, or response-quality frameworks, it grounds the narrative in the peer-reviewed literature: Groves et al. on Total Survey Error, Bethlehem on nonresponse, Heeringa on weighting and DEFF, Schouten on adaptive survey design, Krosnick on satisficing. Citations include the paper and year so a researcher can verify the claim directly. See AI-Assisted Features Overview → Grounding in Peer-Reviewed Methodology.

Using Quality Analysis in Balansor

Viewing Quality Metrics

Quality lives on the Bundle Overview page — click a Bundle's title on the pipeline board. The page renders the Representativeness story, one sample traced from design intent to deliverable:

  1. Strategy → Pool — did sampling achieve the design? One reading per Pool, against the Strategy that generated it
  2. Strategy → Actual — how far off was the realized sample before weighting? Against the Bundle's current Strategy, with a per-campaign breakdown. Reports no strategy until one is assigned
  3. Strategy → Weighted — did weighting recover the design? Against the same Strategy, plus the recovery figure and the design effect it cost
  4. Pool → Actual — what did fielding itself contribute? Pool-referenced, so it answers a different question from the three above
  5. Response Quality — answer diversity, straightlining, consistency

Beside them, the Analysis Base card carries the completeness threshold, and the excluded-set block describes who that threshold removed relative to who it kept. The representativeness measures show per-factor breakdowns with actual vs target distributions. If the Bundle has no current Strategy, the page header offers a strategy selector so you can assign one.

See Data Analysis: Quality Checks for the full walkthrough, including the two bases.

Response Quality Section

Displays four aggregate summary cards:

Card Metric What It Shows
Diversity Mean Normalized Entropy Average answer diversity across categorical questions
Acquiescence Bias Index Agreement tendency in Likert scales
Non-Response Overall Rate Average skip rate across questions
Coverage Question Count Number of questions analyzed by type

Below the summary: a per-question metrics table, straightlining detection panel (for groups), internal consistency panel (Cronbach's alpha for groups with 3+ sub-items), speeder detection results, and multi-flag respondent aggregation.

Running AI Analysis

  1. Open the Bundle's Overview page (click its title on the pipeline board), then click "Run AI Analysis" in the AI Analysis panel
  2. The analyst agent starts in the background — progress updates appear in real time
  3. When complete, the structured report appears with methodology-grounded interpretation

The analysis reads the Bundle's full Representativeness story — every measure, the excluded set, the recovery figure and the design effect — so the report interprets exactly what you see on the page. It requires a derived Silver dataset (the report is stored on the Silver row); a current Strategy is not required, since the story carries explicit no strategy states the analyst reads honestly rather than scoring around.

The AI analysis requires:

  • An AI provider API key (configured in user profile, organization settings, or environment)
  • A Portor MCP endpoint (for accessing quality tools and campaign context)
  • Redis (for cross-worker progress tracking)

Dataset Detail Page

Each dataset's detail page names its owning Bundle and links to the Bundle's Overview page, where the full Representativeness story and the AI analysis live.

Interpreting Results

For detailed interpretation guidance — common patterns, recommended actions, and the two-dimensional quality matrix — see the Quality Metrics Reference: Interpreting Results.

MCP Tools

Quality analysis tools are available through the MCP interface for programmatic or AI-assisted access:

Tool Purpose
get_bundle_quality The Bundle's Representativeness story: selection, fielded and weighted (all against the current Strategy), the Pool-referenced fielding shift, response quality, and what the completeness threshold excluded
compare_bundle_quality Bronze vs Silver against the Bundle's current Strategy, carrying the recovery figure — how much distance from the design weighting closed

See the Bundle Tools Reference for complete parameter documentation.