Test: Quick Add Exit Poll

Does a templatized question increase usage of Interactive Questions?

 

Team: PM, Product Design Manager, Jr Product Designer, Engineering

Hypothesis: Creating a way to add an interactive question in one-click will increase usage.

 

Background

Problem Space: Interactive questions is not performing as expected.

OKR: Increase the number of teachers who have gotten at least 15 responses to a video.

What we know:

Teachers said they wanted interactive questions and that they would use them for instructional video.

Only 2% of users have actually used this feature since we launched.

Why aren’t teachers adding interactive questions to their videos?

User Survey Findings:

  • 10% are using another tool

  • 15% didn’t know this existed

  • 30% don’t have enough time

  • 30% don’t think they need it

Biggest opportunity area:

Focus on the 60% who said they don’t have enough time or don’t think they need it.

  • 10% using another tool

  • 15% didn’t know it existed - push guides and notifications through Pendo

Squad Brainstorm

 

Knowing these findings, we ran a larger squad brainstorm to ideate potential solutions.

Goals:

  • Understand what solutions have the most energy around them

  • Get engineering buy in

  • Graph impact and effort score

Next Steps:

  • Align on solution

  • Validate effort score (Eng)

  • Validate impact score (UX)

 

Proposed Solution Validation

Create a Quick-Add Exit Poll that allows users to add a question to the end of their video in one click.

  • Effort Score Findings:

    • Dev can duplicate Interactive questions functionality and create a one-off template

    • Added scope (minimal) to show what results look like in Interactive Questions responses section

  • Impact Score Findings:

    • 65.7% of teachers said they would use the quick add exit ticket if available to them.

      • When adding in those who said “might use” this number jumps to 97.1%

      • 0% of users said they confidently wouldn’t use it

    • 85.8% of teachers say the addition of poll questions would change their behavior positively towards Interactive Questions.

  • Users are excited about the idea!

“I love having a check in after videos!”

“I am required to check for student understanding throughout lessons. This feature would be a marvelous way to evaluate student misunderstandings as they move through lessons.”

“Checking for understanding is vital for my lesson planning.  I need to know what my students know and don't know, so I can plan accordingly or re-teach as needed.”

“I don't use it as much because we're back in person but for homework it could be great!”

Design Process

 

User Flow

Key takeaways:

  • For the purpose of the test, we need to find ways to reduce scope

  • We believe the success of this feature will be:

    1) Not adding any additional clicks - Can we still copy share link when a user sees the Quick Add Modal? Can this still copy the share link?

    2) Be very clear to users what they are adding and where

User Stories

As a Teacher I want to…

  • See whether students have submitted their response (on all 3 response views)

  • See which students have selected what option 

  • See a group of students that have selected the same option

  • See the relative popularity of the given options amongst the students

  • Understand how a poll question affects student score (if at all)

As a Student I want to…

  • Understand that poll questions are not graded, so I feel comfortable providing my opinion

  • Understand that my poll response has been successfully submitted

Poll Question Verbiage

Internal Feedback via former Educators

We asked our group of internal former educators to help:

“If we were to create one non-graded poll question that was universal for teachers to quick add to their video - what would be a helpful/good question?”

Final version:

After watching this video…

  • I got everything! 😊

  • I got most of it 🙂

  • I got some of it 😕

  • I need help 😰

 

Hi-Fi Mocks

 
 

Results and Findings

🧪 Test Plan

Participants:

  • 20 total average-usage domains randomly selected (10 Test group domains and 10 control group domains)

  • Domains were chosen rather than a % of user base to ensure these users fit our target demographic (active Teachers) and to avoided confusion of one-off experiences among various school districts

  • Average: 289 teachers per district, 383.52 videos created monthly

Success Metrics:

  • Increase the % of teachers who have added at least one question to their videos

  • Increase clicks on Interactive Questions CTA

📈 Results

  • Metrics were evaluated WOW on a 10 week period (11/18 - 1/19)

  • Test group did not increase the % of who had added at least one question to their video or increase clicks to interactive questions CTA

    • Test group had a 2% increase in adding at least one interactive question vs Control group’s 3%

  • <1 % of users added an Exit Poll, the majority of users either selected “Not Now” or “Do not show this again”

🔎 Additional Insights

  • 3% of users selected “See how it works” suggesting they had interest in learning about the feature

  • Low usage patterns due to time of year (Holiday Break) suggest increased Bias to this test data

Next Steps

  • Brainstorm next experiment with Squad

  • Discuss with Stakeholders risk of rolling out “Add Exit Poll” to GA Release

    • Include necessary GTM teams

  • Evaluate Editable Poll Question Release ROI

  • Create Pendo Guides to increase visibility of feature

    • (Shown Left) Resulted in a 4.4% conversion rate