Hero and Video
Based on testing, remove background video on Student Life.
BACKGROUND ISSUE
Second level pages, such as Student Life and Academics, have heroes with background video and a CTA to watch a video inside a modal. This design is based on reasonable, yet not thoroughly tested assumptions. Can we do further research on our site?
- Is auto-playing background video more compelling than a static image?
- Is a CTA leading to a modal better than an in-content video embed?
- Is “Watch Video” stronger than a goal oriented CTA, such as “Visit”?
Student Life Test 1
Test variations and combinations on the Student Life page.
FORM VARIATIONS
We tested five variations against each other.
page with video background and link to modal (0)
2. background image and CTA to modal (3)
background image with a CTA and video in text (4)
Background image without CTA and video in text (2)
Video in background and video in text (1)
TESTING SOLUTION
We ran a split test where users were randomly served one of the five versions. This test was run for three months and had 4,359 experiment sessions.
Results
| Version | Video Placement | Hero | Video Plays | Video Play to End | Page Value |
|---|---|---|---|---|---|
| 0 | Modal | Video | 19% | 4% | .02 |
| 1 | Text | Video | 22% | 8% | <.01 |
| 2 | Text | Image | 15% | 5% | .04 |
| 3 | Modal | Image | 16% | 5% | .04 |
| 4 | Text | Image | 22% | 3% | .05 |
There is not a clear pattern between video placement and video engagement or video
placement and page value. However, there is evidence that heroes with an image perform
better than heroes with a background video. Each of the three heroes with static images
performed better than the heroes with auto-playing videos. The version with the CTA,
Version 4, has the best page value. Oddly, there is a reverse correlation between
video complete and page value.
Student Life Test 2
Since the previous test had 3 strong contenders with similar results, we retested those versions along with a new version. We ran a split test where users were randomly served one of four versions - all with background images. This test was run for three months and had 6,295 experiment sessions.
background image with visit cta and video in text (0)
background image with no cta and video in text (2)
background image and video in modal (3)
background image and Ask CTA video in text (4)
Results
| Version | Video Placement | CTA | Video Plays | Video Play to End | Page Value |
|---|---|---|---|---|---|
| 0 | Text | Visit | 20% | 3% | .05 |
| 2 | Text | None | 22% | 5% | .05 |
| 3 | Modal | Watch | 12% | 4% | .05 |
| 4 | Text | Ask | 27% | 5% | .06 |
The results were fairly similar. Overall, the version with the Ask a Student CTA performed
the best. There is not a clear pattern between video placement and page value. Contrary
to the previous test, there is no reverse correlation between video complete and page
value. Placing the video in a modal seems to get less video plays but the equivalent amount
of video completes.
Science Test
Testing the lessons from the previous test, we swapped in an image for the video on the Science page. We ran an A/B split test where users were randomly served the two versions. This test was run for 2 months and had 2,578 experiment sessions.
Control - background Video
Alt - background Image
Results
| Version | Hero | Video Plays | Video Play to End | Page Value |
|---|---|---|---|---|
| Control | Video | 3.6% | 2.7% | .04 |
| Alt | Image | 3.8% | 2.6% | .03 |
The results were fairly similar. Overall, the version with a video background performed
better than the image, this is contrary to the finding above with student life.
Arts Test
Testing the lessons from the previous tests, we swapped in an image for the video on the Arts page. We ran an A/B split test where users were randomly served the two versions. This test was run for 3 months and had 1,120 experiment sessions.
Results
| Version | Hero | Page Value |
|---|---|---|
| Control | Video | .04 |
| Alt | Image | .03 |
Like Science and unlike Student Life, the version with a video background performed
better than the image.
Recommendations
- On Student Life, use image hero with CTA, and video in text.
- On Science and Arts, use video hero and video in modal.
- Use of video is case by base and we did not identify clear layout rules.
- When there is a clear CTA and strong video, consider using CTA in hero and video in text.