VE as mobile default Making VisualEditor the default mobile editing interface
|
This page talks about theEditing Team's work experimenting with VisualEditor being the default mobile editor on aselectnumber of wikis.
This initiative is a part of our team'slarger effort to simplify contributing on mobile, described in the Foundation's2018-2019 annual plan. And more specifically, to increase the likelihood newer contributors will have success making quick edits on-the-go, by presenting them, by default, with a simpler and more visual editing interface.
Watching this page is a good way to be involved with and stay up-to-date about:
If there isanything about this project you want to know, ask and/or talk about, please let us know on thetalk page! Our team is eager to hear what you think...
As the results published last year have not raised any concerns, and as they aren't showing significant differences in the editing experience, new accounts and logged-out users will get the visual editor as their default editor on mobile. We expect to reduce the number of reverted edits by providingEdit check to more users.
The Editing Team finished analyzing data from an A/B test of showing visual editor as the default editing interface on mobile to newcomers.The results are summarized below.
We have made some changes to the A/B test's design. Those design changes are listed and explained below. We would value knowing what you think of them.
Before that, here is a summary of where we left off in November 2019...
Where we left off
In November 2019, we shared three updates about the status of the A/B test:
While this data quality issue has been fixed and the test has been re-run, we have not yet analyzed the test data.
Changes in test design
The objective of the A/B test has been, and continues to be, to determine what interface when shown as default causes newcomers to have a better editing experience.
In service of the above, we are making two main refinements to the A/B test's design:
The rationale for these refinements are detailed below.
Rationale for changes in test design
Next steps
We have not yet set dates for when this revised analysis will begin or when you can expect to to see results. When we do, we will post an update to this page.
In the meantime, we would value hearing what shortcomings you see in the revisions we are planning to make to the definition of "better" and to the audience of editors that are included in this analysis.
In ourlast update, we noted an oddity with the A/B test data: a statistically higher number of contributors in the test were being put into the "wikitext test bucket" than were being put into the "visual editor test bucket."
Since then, we:
...more details below.
Issue
The two test buckets –default-source anddefault-visual – had a statistically significantdifferent number of contributors in each one. This makes it difficult to accurately compare the behavior of contributors in one test bucket to the behavior of contributors in the other test bucket.
Cause
After some deeper analysis, we uncovered alikely explanation for the issue: there was a percentage of people who never had their bucket assignment recorded.
Said another way: some contributors involved in the test would attempt to make an edit, be "assigned" to a test bucket, but that bucket assignment was never recorded.
The reason for this issue has to do with when and how the software "records" which test bucket a contributor is assigned to. It turned out that this recording happensafter the editor code (mobile wikitext or mobile visual editor) finishes loading. This means in situations where the editor fails to load (e.g. poor network connection) or a contributor cancels their edit, no test bucket assignment is recorded. And because the mobile visual editor code is larger than the mobile wikitext editor code, we can assume a greater number of contributors who would have been placed in thedefault-visual test bucket, did not have their bucket assignment recorded.
Fix
To fix this issue, recording a contributor's bucket assignment now happens in the code that loads the editor, rather than the editor code itself. This way, the time it takes for the editor (mobile wikitext or mobile VE) to load, no longer will affect whether a contributor's bucket assignment will be recorded or not.
Next steps
In the coming weeks, we will be re-running the A/B test with the fix described in place. This way, we can ensure we have two equally balanced test buckets to compare.
This all means you can expect to see results from this test sometime in January, 2020 [or before].
We have a few updates to share about the A/B test...
| Scenario | Description | Action(s) |
| A | default-visual has ahigher* edit completion rate thandefault-source | Step 1: Create a proposal to make the VisualEditor the default mobile editing interface on all wikis Step 2: Consult wikis about the results of the A/B tests and request feedback on the proposal |
| B | default-visual has alower* edit completion rate thandefault-source | Step 1: Leave wikitext as default on all non-test wikis Step 2: Run additional analysis (qualitative and quantitative) to understand what might be the cause for contributors in the A/B test using wikitext being more likely to complete their edits than contributors in the test using the VisualEditor. |
| C | default-visual anddefault-source havethe same* edit completion rates | Step 1: Compare A/B test groups across other metrics (e.g. edit revert rate) Step 2: Assuming VE does not encourage lower quality edits than those in wikitext, create a proposal to make the VisualEditor the default mobile editing interface on all wikis Step 3: Consult wikis about the results of the A/B tests and request feedback on the proposal |
*Defining "higher" / "lower" / "the same": we will depend on our statistical tests to determine whether a change in the average contributor's ECR is large enough to be considered significantly higher or lower. Specifically, we plan to use aMann–WhitneyU test or, if time allows, amultilevel model incorporating per-user and per-wiki fixed effects.
Tomorrow, 9 July, the A/B test that will trial visual editor as the default editing interface onmobile will officially begin. This means, 50% of contributors to the20 participating wikis who meet the criteria listed below, will see the visual editor after tapping edit on mobile and 50% of contributors will see wikitext after tapping edit on mobile.
All contributors to the20 participating wikis, registered and unregistered, who have made less than 100 total edits and have never switched editing interfaces on mobile before will be included in the A/B test.*
*More information about how the A/B test will work can be found below:A/B test information.
After spending the past few weeks consulting with wikis about the prospect of being a part of the "VE as default" A/B test, we now have a list of 20 wikis that will be participating. These wikis are listed in the table below.
The A/B test itself will start within the next week. We will update this page once the test begins.
| Azerbaijani | Danish | Hungarian | Portugese | Swedish |
| Bulgarian | Estonian | Malay | Romanian | Tamil |
| Cantonese | Finnish | Malayalam | Santali | Thai |
| Croatian | Greek | Norwegian | Serbian | Urdu |
We are in the process of coming up with an initial list of wikis to invite to participate in this experiment.
We are striving tomake editing on mobile web simpler.Research leads us to think newer contributors will have more success editing on mobile using the VisualEditor given it presents a more structured, visual and easily understood interface than wikitext does, the editor that is currently presented to contributors by default on mobile. These are qualitiesthe Design Research team has found to be especially important on mobile, where screen space is more limited than on desktop.
Despite what we have come to think are VisualEditor's advantages, few contributors are discovering and using it. In May 2018 for example, just1% of mobile web edit sessions (or ~20,000 of the ~2 million total sessions recorded) happened in VE.
Defaults clearly impact how contributors experience editing Wikipedia. Defaults also expressunspoken assumptions. Assumptions in this context, about which editing experience will help the most number of contributors be successful editing using a mobile device. This project is about investigating this assumption. Our goal is to learn what mobile editing interface is more likely to cause newcomers to start and continue editing Wikipedia. The key metric we will use to evaluate and compare these editing interfaces is editor retention.
Currently, both editing interfaces –VE andwikitext – are available on mobile for contributors to use. Although, the mobile wikitext editor is shown by default to new contributors, while the visual editor is accessible via a dropdown menu in the editing toolbar.
As part of this project, we will be working with a select number of wikis to switch which editing interface is shown to new contributors, on mobile, by default. Meaning, on the subset of wikis we work with, the mobile VisualEditor will be shown to new contributors editing on when they tap "edit" and wikitext will be available via a dropdown menu in the editing toolbar.
This test willnotaffectyouraccount if any of the following are true:
To understand whether VE being the default mobile editing interface creates a "better" editing experience for new contributors, we will be running an A/B test.
Currently, all contributors are presented with the wikitext editor when they first tap "edit" on the mobile website. If a contributor then switches editors using the drop-down in the editing toolbar, their new choice is remembered when they tap "edit" in the future.
This analysis will include contributors who havenever published an edit to Wikipedia before. Half of the people included in the analysis will have been randomly selected to receive the mobile visual editor when they first tap "edit", while the remaining half will have received the mobile wikitext editor as before. Contributors who do not edit using the mobile website or who have not published an edit to Wikipedia before will not be included in this analysis.
Once the test is live, the question for us then becomes:which of these editing experiences is a "better" default for newer contributors?
To answer this question, we first need to define what "better" means so we can compare the two test groups and, ultimately, decide whether to explore making the mobile visual editor the default mobile editing interface for more contributors on more wikis.
Considering our priority is the long term health of Wikimedia projects and this "health" depends on new people starting and continuing to edit, we will useeditor retention to determine which editing interface is "better" in this context.
There are other metrics that can be helpful in assessing the experience these two editing interfaces are causing people to have. Here are some of the other measures we will consider using to compare the two test groups:
We are not sure if these are the right measures, so we are continuing to think about them. If there is a metric you think we should be considering that is not represented above or a metric you think we should be weighing more heavily than others, please let us know on thetalk page. This list is still evolving.
The A/B test ran at 20 Wikipedias from 1 November 2019 through 26 September 2022 and included new contributors who have not edited on any platform before and who were both logged-in and logged-out.
What follows are the key results from this analysis:
The full A/B test report can be found here:Mobile VE as Default AB Test Analysis Report