This article describes the Metrics for News "baseline" which Engagement Scores are measured against.
The baseline is a snapshot of how your newsroom's content performed for a given period in time.
In 2022, Metrics for News was updated to make the baseline calculations more sophisticated.
The baseline anchors your data to help you understand if you are performing above or below what’s considered “normal engagement” for your content.
Previously in Metrics for News, the baseline was calculated based on a single average value. The performance of individual metrics were then set based on points away from that average. The current system allows for more nuance in indicating story performance, and helps mitigate the effect of outlier values in the full metric dataset.
This documentation will help you understand the changes and what you may want to do, if anything, to ensure the most accurate and updated data. Many thanks to our long-time partners at Tribune Publishing, Albany Times Union, Buffalo News, Chattanooga Times Free Press, and our friends at Crain Communications for their feedback, questions and newsroom-tested explanations that you will find below.
How will this change our scores?
The Metrics for News engagement scores are a blend of several metrics, weighted for the importance placed on each. When you first set up Metrics for News, you or your newsroom set a baseline period to anchor your data against (typically a 3-6 month time frame for what your newsroom would have considered average engagement). Engagement Scores have been based on that average since then. Until now.
After nearly a year of testing, refining and user feedback, the Metrics for News team made a change to how scores are calculated, using percentiles instead of averages. For some clients the change was barely noticeable, but for others, scores may have increased.
Why are we using percentiles now?
Averages are pretty easy to understand, but can paint an inaccurate picture. The average assumes that scores are distributed roughly along a bell curve, with most scores in the middle, a few outliers to the left (below average), and a few outliers to the right (above average).
Reality is a bit different, though, and that's where the percentile can be much more informative. A percentile shows which part of the curve the score falls at, and how many scores are represented by that metric.
What does that mean for us? How do we know where the scores fall?
As admins in Metrics for News you will be able to see the percentile ranges for every metric used in your blended engagement scores. To find this breakdown, go to Settings > Engagement Scores and then click “View or Edit Your Baseline Data”.
There you will see your current baseline date range, such as the example below.
You will also be able to see the breakdown of each metric, such as the example below.
This chart shows that the 50th percentile (or median) for views per story is 1,330. That means 50% of stories have more views, and 50% have fewer views. (If we were looking at averages alone, the average views per story may have been higher in your newsroom than using the percentile approach). Using the percentile gives a much better sense of performance and creates more equity across teams and authors.
You now have percentiles for each of the metrics used for your blended engagement scores, and for each of the categories and characteristics you track in Metrics for News. That means you now know the median referrals per story from Twitter and from Facebook; the median words per story; the median time per story; and much more. All of this is factored into the engagement scores for stories, topics, authors, teams, etc.
Do we need to update our Metrics for News settings?
That depends. For some of our long-time partners, the baseline may be anchored to older datasets that are no longer relevant. Some partners had baselines going back to 2018, for example. For evaluating content in 2022, you will want a more updated period to measure against. We would suggest a 3-6 month time range in 2021. Many of our partners are choosing a date range for the last six months of 2021.
The best way to update those settings will be to refresh your baseline to a more recent period that is reflective of a “normalish” time. You can update your baseline on your own or ask for help from the MFN team at support@metricsfornews.com.
You can review the baseline period you initially set in your MFN account settings. You can go to Settings > Engagement Scores to review or edit the baseline. When you do that, you will now also see percentiles for each component metric included in the current baseline.
Why did my engagement scores change?
The Metrics for News team has updated how we calculate the baseline and has gone through a process of cleaning up the data to filter out any outliers that may have been inflating the scores unnecessarily. This could mean that scores for individual authors or teams or topics may go down. That’s normal. It’s something everyone will experience. It also means that your scores will be more fair across the board and more reflective of how content is performing for staff (less influenced by wire content, infrequent authors or other outliers).
If you want to nerd out a little more on percentiles, you can think about this: If anyone on your staff has children, they will most likely be familiar with understanding percentiles in growth charts often seen at the pediatrician’s office. Percentiles are measurements that show where a child is compared with other kids. The higher the percentile number, the bigger a child is compared with other kids of the same age and gender, whether it's for height or weight. The lower the percentile number, the smaller the child is.
You can consider the same thing for your stories. Percentiles will show you where any story ranks in engagement compared with other stories. High percentiles will indicate a story earned better engagement compared to other stories. Lower percentiles will indicate a story earned less engagement compared to other stories. This will paint a more accurate picture of engagement instead of relying only on averages.
Do I need to do anything about this change?
The only thing we would recommend considering is if your baseline is anchored to data that’s a few years old, we recommend refreshing it to a more current period. We would suggest a 3-6 month timeframe in 2021.
What should I do if I need help?
If you need help from the Metrics for News team, send your questions to support@metricsfornews.com.
Updated as of January 2022.