Articles in this section

Headline Testing export options

  • Updated
There are two exportable CSV reports available in the Headline Testing dashboard— Tests.csv and Headlines.csv. You can find these options by clicking the "export options" link, which can be found below the Average CTR Increase section. (Note: These exportable files currently don’t include image information for tests involving images.)
 
Screen_Shot_2020-08-03_at_6_07_20_PM.png


Report Types

Summary Report (CSV): For comparisons across tests.
Variant Breakdown Report (CSV): For comparisons across variants, regardless of test.


Report Metrics 

Status

  • cancelled: The test was stopped manually before a winner was found.
  • winner_found: A winning headline has been found and Chartbeat is automatically displaying that winning headline for all visits on your homepage.
  • stopped_with_winner: A winning headline has been found but a user has stopped Chartbeat from automatically displaying the winning headline to all users on your homepage. If you've stopped a headline that Chartbeat was auto-playing and want that headline to be permanently played to all readers, you must change the headline directly in your CMS.
  • no_winner_found: A winning headline has not been found. This most often occurs because the headline options are too similar to reach 95% statistical confidence that one is meaningfully better than the others. 
If headline tests are frequently ending at 15 minutes with no winner found, check out our separate article on troubleshooting issues with your Headline Tests.
 
Trials: Every time that a test headline is loaded in a page and served to a visitor. For example a headline that ran a total of 2,000 trials was served to 2,000 unique cookies. On the other hand, if an experiment had 5,000 trials it means that headlines were served a total of 5,000 times.
 
Views: Only those trials where the reader saw, but did not click, the headline variant.
 
Clicks: Only those trials where the reader clicked the headline variants but did not engage with the destination page for more than 15 seconds.
 
Engaged Clicks: Only those trials where the reader clicks the headline variant, then engages with the destination page for more at least 15 seconds. 



Summary Report (CSV) fields:

test_id

Unique ID assigned to each headline test.
host

The Chartbeat Site where the test was initiated.
domain

The domain name where the test was initiated.
path

The Chartbeat page path where the test was initiated.
created_at

Time when the test was started.
winner_found_at

Time when a winning headline was found. This field will not contain values for tests where no winner was found.
stopped_at

Time when the test was ended, manually or automatically.
test_duration_sec

Test duration in seconds. This field will not contain values for tests that resulted in 'no winner found' and 'canceled' statuses.
test_duration_min

Test duration in minutes.
status

Final status of the completed test.
num_variants

The number of headline variants used in the test. The original headline (variant A) counts as one variant.
winning_variant

Indicates the variant ID of the winning headline (A, B, C...).
ctr_increase

The increase in Click Through Rate comparing the winning headline variant to the original headline (variant A), expressed as a decimal. This field will not contain a value for tests where no winner was found, or where the original headline was found the winner.
winning_headline

Contains the headline text of the winning headline variant.
destination_url

The target URL of the headline where the test was initiated.
headlines

Lists each headline variant, including both the variant ID and the variant headline text.
sections

Chartbeat section tags associated with the destination page URL.
created_by

The email of the Chartbeat user who created the test.



Variant Breakdown Report (CSV) fields:

headline_id

Unique ID assigned to each headline variant (A, B, C...).
is_winner

TRUE for winning headlines, FALSE for non-winning headlines, empty for tests where no winner was found.
headline

Headline text for each variant.
views*

Pageviews where the variant was displayed, but not clicked.
clicks*

Pageviews where the variant was clicked and the user spent less that 15 seconds of engaged time on the subsequent page (destination URL).
engaged_clicks*

Pageviews where the variant was clicked and the user spent more that 15 seconds of engaged time on the subsequent page (destination URL).
ctr_mean_percent

The Click Through Rate for the headline, expressed as a percentage.
*Total trials = views + clicks + engaged_clicks.

Formulas

Trials = Views + Clicks + Engaged Clicks
CTR % = (Clicks + Engaged Clicks)/(Clicks + Engaged Clicks + Views) * 100 (*see note below)
CTR Lift = (Winning CTR - Original CTR)/(Original CTR)
 
(*) You may find a small percentage difference between this calculated value and the value listed under ctr_mean_percent. This difference is due to an assumption about the distribution of views, clicks, and engaged clicks at the start of the experiment, prior to any observed click-thru behaviors. For all experiments, we assume values of [100, 2, 3] for views, clicks, and engaged clicks as a starting point for the sampling algorithms. The mean CTR % shown in the report is then (2+ Clicks + 3+ Engaged Clicks)/(2+ Clicks + 3+ Engaged Clicks + 100+ Views) * 100 

Related articles