Personalized AI Dialogues and Parents' HPV Vaccine Intent: A Randomized Trial

Contributions

• Demonstrates that an unmodified, out-of-the-box GPT-4 Turbo model can effectively address vaccine hesitancy through personalized dialogue, without domain-specific fine-tuning or medical training.
• Provides a large-scale randomized trial (N=1,409) evaluating AI dialogue against both standard-of-care (CDC brochure) and control conditions, with pre-registered outcome measures.
• Shows that AI dialogue achieves comparable effects to traditional health communication while generating 15,559 automatically fact-checkable claims, 99.6% of which were medically accurate.
• Identifies substantial heterogeneity in parental HPV vaccine concerns (23 distinct categories) and documents how the model tailors responses across these varied concerns.
• Demonstrates that the dialogue approach produces larger gains in vaccination intentions at higher thresholds (e.g., 20.1% of participants shifted ≥30 points in the AI arm vs. 12.6% in control).

Method

Study design: Pre-registered randomized clinical trial with 1,409 eligible participants (N=7,368 screened) aged 18–50 who had unvaccinated children aged 11–26 or had not yet decided on vaccination. Participants were quota-matched to national distributions on gender, race, and ethnicity via the Cint Exchange recruitment platform.

Randomization: Participants were randomized into three arms via Qualtrics using a 2:1:1 ratio favoring the AI-dialogue condition:

• AI-dialogue (n=706): Four-round back-and-forth conversation with GPT-4 Turbo prompted to address their stated HPV vaccine concerns
• Brochure (n=352): Display of a CDC informational brochure on HPV vaccination, held on-screen for 1 minute
• Control (n=351): Four-round conversation with an AI about an unrelated topic (cat vs. dog preferences) to isolate effects of interaction itself

Exclusion criteria: Participants already planning to vaccinate (≥75 vaccination intentions for both sons and daughters), youngest child >26 years old, or failure of attention checks.

Outcome measure: Primary outcome was change in HPV vaccination intentions (0–100 scale) for sons and daughters separately, measured before and after intervention assignment.

AI dialogue procedure: Participants first reported their vaccination concerns as free text (e.g., "I haven't got my children vaccinated... someone also said the vaccine exacerbated an autoimmune disease"). This concern was automatically summarized by GPT-4 into a single sentence, presented back to the participant for confirmation, and then forwarded to GPT-4 for a targeted response addressing that specific concern. The conversation followed a multi-round format with system prompts informing the model of the participant's demographic characteristics (age, gender, current children status) and vaccination context. Conversations lasted a median of 2.79 minutes.

Fact-checking procedure: All 15,559 AI-generated claims were extracted via automated claim extraction (GPT-4.1 with a structured extraction prompt) and independently fact-checked using two approaches: (1) Perplexity-Sonar-Pro with an online search model rated on 0–100 accuracy scale, with low-accuracy claims (<40) re-evaluated via Perplexity-Sonar-Pro-Search (multi-step reasoning), and (2) manual quality review of remaining inaccurate claims classified into error types (factual inaccuracy, misrepresentation, mechanism misrepresentation, misleading framing, unsupported claims).

Results

Sample: Final analytical sample comprised 622 AI-dialogue, 302 brochure, and 333 control participants (49.5%, 24.0%, 25.5% respectively). Attrition rates differed across arms (AI: 11.9%, brochure: 14.2%, control: 5.1%), but statistical modeling controlling for baseline intentions showed attrition was not differential by treatment arm.

Primary outcome—AI-dialogue vs. control:

• Sons: 9.73 points increase (95% CI [6.89, 12.56], p<.001)
• Daughters: 7.72 points increase (95% CI [4.98, 10.46], p<.001)

Brochure vs. control:

• Sons: 8.04 points increase (95% CI [4.83, 11.25], p<.001)
• Daughters: 5.51 points increase (95% CI [2.52, 8.50], p<.001)

AI-dialogue vs. brochure: No statistically significant difference (sons: 1.69 points, 95% CI [−1.30, 4.67], p=.268; daughters: 2.21 points, 95% CI [−0.57, 5.00], p=.120).

Distribution of effects: Post-hoc analyses revealed the AI dialogue was more effective at producing large intention increases. For example, 20.1% of AI-dialogue participants increased intentions by ≥30 points for sons, compared to 12.6% in control (p<.006); for daughters, 15.6% vs. 9.4% (p=.010). This demonstrates the AI dialogue was not just matching the average effect of the brochure but was producing more extreme positive shifts in a subset of participants.

Heterogeneity analysis: No significant moderation of treatment effects across age, gender, education, race, political orientation, baseline vaccination intentions, or concern type (p>.05 for all interactions after Holm-Bonferroni correction). The dialogue's advantage generalized across diverse parental concerns about HPV vaccines.

Concern classification: Analysis of 1,409 free-text baseline responses identified 23 distinct concern categories, with the most prevalent being:

• Safety and side effects (23.5%)
• Positive perceptions / confidence (17.6%)
• Lack of information and misunderstanding (14.2%)
• Doubts about vaccine efficacy and development (9.8%)
• Misinformation and conspiracy theories (4.8%)

Fact-checking results: Of 15,559 AI-generated claims:

• Overall accuracy: Median=100, mean=95.4 (SD=14.6), range 0–100
• High accuracy (≥80): 95.4% of claims
• Genuinely inaccurate after re-evaluation: 0.4% (65 claims) classified into three error types:
• Factual inaccuracy (n=35, 53.8%) — outdated statistics, overstated infection risks
• Misrepresentation (n=11, 16.9%) — e.g., attributing FDA approval to CDC
• Mechanism misrepresentation (n=7, 10.8%) — e.g., mischaracterizing transmission routes

The AI's accuracy was achieved without domain-specific training, fine-tuning, or access to vetted medical information—it relied on its general knowledge.

Notes

Strengths:

• Large, nationally representative sample with pre-registered analysis and balanced randomization
• Novel contribution: demonstrates unmodified LLMs can tackle vaccine hesitancy without specialized training or fine-tuning
• Comprehensive fact-checking of 15,559 AI-generated claims using multiple independent methods
• Addresses heterogeneous parental concerns through personalized dialogue, more flexible than one-size-fits-all messaging
• Clear comparison against standard-of-care (CDC brochure) and appropriate control (unrelated dialogue)

Limitations:

• Single interaction only; no follow-up on whether intention changes translate to vaccination behavior
• Recruitment via online panel (Cint Exchange) limits generalizability to digital-native populations
• Study design does not isolate which dialogue features (personalization, evidence quality, rapport) drive effectiveness
• No measurement of how participants evaluate AI source credibility or whether perceived-AI identity affects persuasion
• Attrition differed across conditions (11.9% vs. 5.1%), though statistical modeling suggests this was not treatment-driven differential dropout

Open questions:

• Do intention changes sustain and translate to actual vaccination behavior in follow-up?
• How does the method perform for other health behaviors or belief domains beyond vaccines?
• What dialogue elements are critical: is personalization necessary, or do targeted facts alone suffice?
• How do results generalize beyond English-speaking, online-recruited U.S. parents?