Turning Thousands of Survey Responses into Ranked Insights with AI

A repeatable process for analyzing open-ended feedback, with human checks built in

Marketing Analyst (Contract) | CVS Health / Aetna | September to December 2025 | Tools: Custom-built GPT, Excel

The Challenge

Member surveys produced thousands of open-ended comments each, far more than anyone could read manually, and the marketing team needed to know which problems mattered most.

My Process

1. Prepare the Data: Survey responses were provided in CSV format and reviewed in Excel before being submitted to the organization's internally developed AI tool. I cleaned the data by removing irrelevant or inaccurate responses and personally removed identifying information to protect member privacy.

2. Prompt for Themes: Using an internally approved, custom-built GPT, I tested and refined prompts to identify recurring complaints and categorize common themes. I also asked the tool to quantify findings, including the percentage of responses associated with particular issues, to help establish which pain points appeared most frequently.

3. Review and Verify: AI-generated findings were reviewed multiple times against the original survey responses. I found that the tool occasionally struggled to distinguish between similar complaint categories, so I refined my instructions and clarified category definitions when needed. I also experimented with the tool's sensitivity settings to improve the consistency and accuracy of its results.

4. Rank and Quantify: Once themes were sufficiently refined, I used the AI-generated analysis to identify the most frequently reported issues and organize them by prevalence. This helped turn thousands of individual comments into a more digestible picture of where members were experiencing the most difficulty.

5. Report and Share: I compiled the findings into written reports highlighting ranked pain points, relevant percentages, and representative member comments. Reports were shared with my manager and relevant internal stakeholders, including teams responsible for evaluating and addressing the member experience.

Results

• Analyzed approximately 15–25 individual surveys, each containing roughly 2,500–10,000 responses.

• Developed a repeatable process for using AI to identify, categorize, and quantify common member complaints.

• Produced written reports that translated large volumes of open-ended feedback into prioritized insights.

• Provided internal teams with representative member comments and ranked findings to help guide decisions about areas needing improvement.

Survey topics included the customer service experience, website navigation, and the enrollment process.

Lessons

Prompt refinement, clear category definitions, data preparation, and repeated human review were all essential to producing reliable findings.

The biggest takeaway was that AI worked best as an analytical starting point, but not a replacement for human judgment. It helped see patterns at scale, while my review helped ensure those patterns actually reflected what members were saying.