Read more is a feature that is currently stable on almost all Wikipedia mobile websites. It aims to drive page views by engaging users by directing them to related content. The rationale is, if readers are offered suggestions that are similar to the topic they are reading about, this will further engage their reading session time, it will further educate them about the topic they are looking for, and supports a richer reading experience for those who are just randomly browsing topics. The concept already exists on apps, where you can check theperformance report.

If a reader has reached to the bottom of the article, they might be looking to read more about the topic and surfacing articles that are similar might be exactly what they are looking for.

Using theExtension:RelatedArticles extension will show suggested articles at the bottom of pages encouraging the user to read another page.

A user who gets to the bottom of an article on either mobile or desktop web is shown a list of other articles that are related to the one they just finished. The notion is that if the reader has read to the end of the article, they might be looking to read more about the topic and surfacing articles that are similar might be exactly what they are looking for.

This has been released on apps and resulted in a 16% click-through rate (For users who saw it). Additionally,25% of the users who read more results, clicked through at least 1x in a 1-day period (data).

After 4 months of running in beta on both mobile and desktop, a proposal wasmoved to stable on mobile.

Where do article suggestions come from?

Amore like query which will programmatically choose related pages. Here is an overview of the algorithm the function is based on:https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html.

The more-like query service ranks articles in order to show the top 3 and uses a score that computes the similarity between documents, this can be fine-tuned as documented on the wiki page for it:Help:CirrusSearch#Morelike. The defaults in code can be seen on githubhere.

Given the concerns expressed by community membershere and elsewhere about popular results making little room for relevance or serendipity, changes were made to the algorithm in August aftertesting and communication. As of August 30th, 2016, boostlinks (a measure of importance) and boosttemplates (usually a measure of quality) were removed from the scoring, as they caused popular, less relevant pages to appear with high frequency. The ticket for this work is capturedhere.

Results can be overridden by editors using the{{#relatedarticle:}} magic word.

Do editors have any control or are they given any preview of the suggested articles?

Though this feature is not considered part of the article, editors can change the suggested articles given by adding up to 3 manually curated examples to this part of the page navigation.

{{#related:new page title1}}

{{#related:new page title2}}

{{#related:new page title3}}

For example:

OnKorur language the related pages have been over-ridden to:

{{#related:Western Oceanic languages}}

{{#related: New Guinea}}

{{#related: Mbula language}}

User:Jkatz (WMF) thinks that making them editable is a problem because:

• the results are not automatically updated
• it means that improvements we make to the algorithm/selection will be lost to pages where an editor has overridden the automated selection.
• right now the manual selection option only applies to the web (not the apps), which is misleading
• some editors have requested that we do automated refreshes every time you load the page--this would not be possible if the above keywords are used.

How is related articles different from the See also section, navigation boxes and the category system?

They have pretty pictures, are limited to 3, for greater simplicity (unlike see also, are only intended for users who have gotten to the bottom of the article without finding another link more appealing to visit). It is thought that the images and simplicity increase the click-through rate. Which is the metric of success for this feature.

The following is an attempt to summarize the feedback collected on thetalk page for this feature (as of Feb 25, 2016), along with responses to each one.

Disambiguation pages are pretty bad. See the holocaust disambiguation page as an example:w:Holocaust (disambiguation)

Response:

Luckily, the feature has no real utility on those pages. There is now a ticket to remove this:T127068 prior to any further roll-out.

FIXEDOn English Wikipedia, the use of non-free (fair use) article images for navigation goes against En Wikipedia policy.

Response:

• This is being remedied by this task:T124225 and is considered a blocker

The click-through-rate metric is not air-tight and reader value is not certain.

Response:

• It is true that click-through rates can be gamed. I (JKatz) submit that sustained click-through rates are indicative that readers are finding value, and placement at the bottom of the page, as well as some analysis of raw pagelogs above, suggest that cannibalization of blue links is not occurring.However, enough others disagree that some additional work is necessary, at least prior desktop roll-out. Here are some ideas:
• Roll-out on some small % of anon-traffic. It is not entirely fair to test only with logged-in users a feature that is intended primarily for users who are not logged in.
• Then, a/b test and look at overall pageviews (for those of you who find session depth to be a bad metric: please suggest a better one)
• The, use the quick-survey feature to ask users who click on 'related pages' if they are satisfied with the results or to get feedback

This is the same as see-also and does not add additional value

Response:

• The sustained high click through rate on mobile suggests that users are finding it valuable enough to warrant promotion. On desktop this is not yet the case--though the click-rate still exceeds any given link. (see above about obtaining better measures of reader value).
• The best thing to do would be to combine with see also, but this is thorny and a lot of technical work. I think we should be assessing whether this is a net benefit to Wikipedia or net negative, not whether or not is a perfect solution.

Sometimes pages return anywhere from suboptimal to damaging results

Response:

• for 'damaging' results, which are rare, editors can over-ride results. I was against mixing algorithm and editing, but the value here is now clear.
• for 'suboptimal results', for now I think we should live with it because the readers seem to bederiving value (see below on underlined portion)

Algorithms are non-NPOV and go against our ethos OR algorithms should be adapted to community norms

Response:

• In theFAQ, we have pasted put a link to the code somewhere and describe the variables/logic? Ultimately, a solution might be to allow each community to tweak the weighting to their needs, but I think that is not warranted by the value of this feature at this point. How do you feel about transparency with regard to the algorithm logic as a solution?
• we might want to denote that they are machine-driven somewhere to clarify. Is this a blocker?

In German and maybe Russian, the tool may violate a principle against 'Themenring' policy

Response:

• I see no reason why we should push on these wikis if they don't want it, but have asked community members to poll their wikis and find out if the concerns are representative

The community was not consulted first and we would have helped you avoid major pitfalls.

Response:

• Unfortunately, yes. We thought since this worked on apps it was a no-brainer, but we clearly misjudged. Given that we obviously made a mistake here, and are making an attempt to improve moving forward both with this feature andothers, we hope that you can help us move forward together.

• %CTR (click through rate, clicks/views) is higher than 10%.
• Ideally, we will be able to ensure that these clicks are not-cannibalistic (increasing overall clicks as opposed to taking clicks from blue links)

This was tested in beta first.

A user reaching the bottom of an article is shown the title and lead image for 3 articles that relate to the article they just finished.

We were able to measure the engagement clicks/impressions with this feature.

A user reaching the bottom of an article is shown the title and lead image for 3 articles that relate to the article they just finished so that they can continue reading about that topic. Someone editing a Wikipedia article can manually change the article suggestions using wikitext so that they can correct any erroneous or sub-optimal suggestions. A project stakeholder (such as PM or data analysis) is able to measure the engagement clicks/impressions with this feature so they can determine if it is adding user value.

We wanted to track:

• Impressions of read more suggestions (the lump--not each item suggested)
• Clicks to read a suggested article
• The position of article (1,2,3). Ideally: whether or not article was edited manually
• Ideally: a)overall referrals from a page with read more, b)overall referrals from same page without read more

(this has been updated on 2/24/16 to reflect the actual order to-date)

1. build read more according to mobile web specsDone
2. launch on mobile web betaDone
3. launch on desktop web betaDone
4. measure impact using event logs CTR (referral data won't be helpful at these numbers)Done
5. discuss with community
6. launch on mobile
7. launch on desktop (open discussion about whether we launch in beta first and/or do progressive rollout)
8. Delivery Estimate: end of Q3 on mobile across wikis...desktop is too much of a question mark as of February 24, 2016.

The related pages feature was deployed to 90% of all users on the mobile version of English Wikipedia on March 30, 2017. We collected data in the timespan between March 30 and May 15, 2017. The purpose of this test was to determine the impact of the feature over time. The main metrics we focused on throughout this analysis were clickthrough rate and variations in total events recorded between the test group and the control group.

On enwiki, clickthrough rates for related pages links were especially high (6%) when users see the related pages section at the bottom of the article. The overall clickthrough rate on related pages links for mobile was 1.9%. No significant changes in these rates were observed over time, indicating that users will continue using related pages with a similar frequency. When compared to the control group, no significant impact on pageviews was observed hinting at the conclusion that while the feature's content curation is preferred among users, it does not significantly affect user behavior - if a user was interested in continuing to read, she will select an article suggested by related pages with a higher probability than another article on a page. However, the feature has little to no effect on users who were not interested in accessing more articles.

On mobile enwiki, clickthrough rates (ratio of pages clicked/pages seen) for the feature average d 1.91% (ranging between 1.85% and 1.97% with 95% convidence) over the period 03/30/2017 - 05/15/2017. When users view the related pages (by scrolling to the bottom of the article), the clickthrough rate is 6.0% (range 5.89% - 6.14% with 95% confidence). No significant changes in this trend were observed over time.

On mobile enwiki, we observed negligible differences between the expected pageviews for the test (related pages on) and the control (related pages off), with the ratio of expected events for the test group to expected events for the control group being 1.003 (range 0.998 to 1.010 with 95% confidence), suggesting that the change has little effect on total pageviews

When comparing the ratio of unique events per total events per session for the test and control groups, similar results were observed, with the control group having 0.9986 unique events per session, and the control group having 0.9994 unique events per session. No significant changes to these trends were observed over time.

SELECTcount(*)AStotal_Events,event_eventName,left(TIMESTAMP,8)ASmmddyyyyFROM`RelatedArticles_16352530`WHEREwiki='enwiki'andevent_skin='minerva-stable'GROUPBYmmddyyyy,event_eventName;

SELECTcount(DISTINCTevent_userSessionToken)AStotal_Events,event_eventName,LEFT(TIMESTAMP,8)ASddmmyyyyFROMRelatedArticles_16352530WHEREwiki='enwiki'andevent_skin='minerva-stable'GROUPBYddmmyyyy,event_eventName;

As measured for the first two weeks of 2016 on a sampled basis, related articles items were frequently tapped on mobile web beta Wikipedias (the skin "minerva-beta" in the table below) - 19.7% of the time when seen. The relatively high tap rate on the mobile web beta is believed to be attributable to the interesting content, as well as in part automatically collapsed sections in its phone form factor layout yielding a higher impact visually to the related articles at the end of the article.

Interestingly, while users on the desktop web Wikipedias (see "vector" in data below) were likely to see the related articles panel (35% ready-to-seen event ratio versus mobile web's 27.3% ratio), desktop users evidently were far less likely to click on a related article - they apparently clicked only 3.4% of the time when seeing the related articles panel.

selectevent_skin,event_eventName,count(*)fromRelatedArticles_14496900wheretimestamp>'2016'andtimestamp<'20160115'groupbyevent_skin,event_eventName;

event_skinevent_eventNamecount(*)cologneblueready3minerva-betaclicked217minerva-betaready4025minerva-betaseen1099modernready114modernseen62monobookclicked1monobookready270monobookseen82vectorclicked48vectorready4053vectorseen1429

The relatively lower click rate on the desktop web is believed to be attributable to the richly laid out expanded sections yielding a relatively lower visual impact for the related articles. It may also be related to the mechanics of beta opt-in: on the mobile web beta opt-in enables all beta features, whereas on the desktop web users are able to opt-in to specific beta features. It should be noted that some users on desktop web also enable the feature to auto-enroll on all beta features.

Tablet devices by default are presented the mobile website, which commonly expands sections, but still has a relatively lighter weight layout. But users have the ability to change to desktop mode, where again sections are expanded but there's a relatively heavier weight layout.

Note that for a confined set of tablet devices, although it's an imperfect analysis and there are a variety of potentially impacting factors that could muddle the analysis, the data suggest that we can at least say with some level of confidence that

• mobile web beta treatment dramatically outperformed the desktop treatment
• although users with a confined set of tablets on mobile web beta had not insignificant engagement with related articles, their rate of engagement was somewhat lower than the fuller mobile web beta population (confined tablet population's engagement at 16.7% versus full mobile web beta population's engagement of 19.7% for a two week period and 13.2% versus 18.4% for the full period since it was introduced on mobile web beta) - but this was still quite promising on mobile web beta

selectevent_skin,event_eventName,count(*)fromRelatedArticles_14496900wheretimestamp>'2016'andtimestamp<'20160115'and(userAgentlike'%Nexus 7%'oruserAgentlike'%Nexus 9%'oruserAgentlike'%iPad%')groupbyevent_skin,event_eventName;

minerva-betaclicked11minerva-betaready262minerva-betaseen66monobookready1monobookseen1vectorready37vectorseen9

All events and skins since related articles available in beta for confined set of tablets...

selectevent_skin,event_eventName,count(*)fromRelatedArticles_14496900whereuserAgentlike'%Nexus 7%'oruserAgentlike'%Nexus 9%'oruserAgentlike'%iPad%'groupbyevent_skin,event_eventName;

minerva-betaclicked39minerva-betaready1485minerva-betaseen295monobookready2monobookseen1vectorready70vectorseen21

All events and skins since related articles available in beta for all UAs...

selectevent_skin,event_eventName,count(*)fromRelatedArticles_14496900groupbyevent_skin,event_eventName;

cologneblueready9cologneblueseen2minerva-betaclicked768minerva-betaready19009minerva-betaseen4170modernready185modernseen105monobookclicked4monobookready535monobookseen171vectorclicked110vectorready8005vectorseen2848

Do note that a CTA experiment to heighten the general enrollment in mobile web beta was running in the latter part of 2015 but wasdisabled, hence relatively lower event counts for the two week period observed at the start of January 2016. Engagement with the related articles was relatively high with both the CTA in force and has continued to be relatively high (albeit slightly lower) since the CTA was removed. See the data above. Note also that asmall experiment with collapsed sections has been running, but should have a negligible impact on the analysis in this page.

One additional signal in the environment suggesting the efficacy of the mobile web beta related articles feature is the share of internally referred pageviews as a percentage of total pagevews. Again, caveats could apply, but it appears that prior to the related articles feature, mobile web beta internally referred pageviews were in the low 50s percentagewise, whereas with the feature they are now in the high 60s percentagewise (the stable channel where the feature was not implemented stayed relatively consistent around 25%). However, this is only a signal; partial rollout in the stable channel of the mobile web would be more telling.

What follows are relative internally referred pageviews rates on the mobile web for a set of Thursdays - two Thursdays in the first two weeks of January, and two Thursdays prior to the related articles feature being released in beta on the mobile web and desktop web. Dates coinciding with holidays or known high rate fundraising campaign banners are not included, as they would complicate the analysis.

Note on this query: feature running in mobile web beta Wikipedias.

selectreferer_class,x_analytics_map['mf-m'],count(1)fromwmf.webrequestwhereyear=2016andmonth=1andday=14andaccess_method='mobile web'andagent_type='user'andis_pageview=trueandnormalized_host.project_class='wikipedia'andpageview_info['page_title']<>'-'andpageview_info['page_title']<>'Special:Search'groupbyreferer_class,x_analytics_map['mf-m'];

beta: 48450/(48450+14958+8465) = 67.4%stable: 58400028/(58400028+56362895+113239899) = 25.6%externalNULL113239899externalb8465unknownNULL56362895unknownb14958internalNULL58400028internalb48450

Note on this query: feature running in mobile web beta Wikipedias.

selectreferer_class,x_analytics_map['mf-m'],count(1)fromwmf.webrequestwhereyear=2016andmonth=1andday=7andaccess_method='mobile web'andagent_type='user'andis_pageview=trueandnormalized_host.project_class='wikipedia'andpageview_info['page_title']<>'-'andpageview_info['page_title']<>'Special:Search'groupbyreferer_class,x_analytics_map['mf-m'];

beta: 51732/(51732+14540+8137) = 69.5%stable: 59881357/(59881357+55286651+109964317) = 26.6%externalNULL109964317externalb8137unknownNULL55286651unknownb14540internalNULL59881357internalb51732

Note on this query: feature was not yet running in mobile web beta Wikipedias.

selectreferer_class,x_analytics_map['mf-m'],count(1)fromwmf.webrequestwhereyear=2015andmonth=12andday=3andaccess_method='mobile web'andagent_type='user'andis_pageview=trueandnormalized_host.project_class='wikipedia'andpageview_info['page_title']<>'-'andpageview_info['page_title']<>'Special:Search'groupbyreferer_class,x_analytics_map['mf-m'];

beta: 184067/(184067+55814+119778) = 51.1%stable: 50273872/(50273872+51372508+102363512) = 24.6%externalNULL102363512externalb119778unknownNULL51372508unknownb55814internalNULL50273872internalb184067

Note on this query: feature was not yet running in mobile web beta Wikipedias.

selectreferer_class,x_analytics_map['mf-m'],count(1)fromwmf.webrequestwhereyear=2015andmonth=11andday=19andaccess_method='mobile web'andagent_type='user'andis_pageview=trueandnormalized_host.project_class='wikipedia'andpageview_info['page_title']<>'-'andpageview_info['page_title']<>'Special:Search'groupbyreferer_class,x_analytics_map['mf-m'];

beta = 189359/(189359+58549+121871) = 51.2%stable = 51119529/(51119529+54209700+105205731) = 24.3%externalNULL105205731externalb121871unknownNULL54209700unknownb58549internalNULL51119529internalb189359

In light of the feedback noted above, I would like to propose the following next steps for this feature:

1. Publish variables and code that impact article selection (seeFAQ)Done
2. Fix non-free image issue via:T124225Done
3. Work onT127068 to remove from disambiguation pages
4. Rollout on mobile web on select wikis (it seems like stakeholders are less concerned with the experience here, and the 'performance' is much higher)
5. Rollout small traffic a/b test on desktop on a few wikis and either measure session depth change or use quick survey (the latter is more difficult)
6. Use results of 3 to inform possible desktop rollout (Hear back from De and Ru Wikipedias as to whether or not a rollout is desired, given 'Themenring' policy)

Any thoughts, suggestions? I will post on the talk page as well here:Talk:Reading/Web/Projects/Related pages/2016#h-Response_to_feedback_so_far_and_proposal_for_moving_forward-2016-02-26T00:17:00.000Z

On June 15th 2016 21:00 local time the German speaking Wikipedia will most likely reject this feature with a large mayority. Sorry for all the money spend. --Eingangskontrolle (talk)16:12, 4 June 2016 (UTC)[reply]

• enable close button on individual items to alert editors (or tweak algorithm) when a particularly recommendation is not helpful
• merge with see-also
• Recast as a "Find Similar Articles" button with a long list of search-style results