Copyedit structured task  A task for newcomers to complete copyedits based on machine suggestions
|
This page describes work on a "copyedit" structured task, which is a type ofstructured task that the Growth team may offer through thenewcomer homepage.This page contains major assets, designs, open questions, and decisions.Most incremental updates on progress will be posted on the generalGrowth team updates page, with some large or detailed updates posted here.
Structured tasks are meant to break down editing tasks into step-by-step workflows that make sense for newcomers and make sense on mobile devices.The Growth team believes that introducing these new kinds of editing workflows will allow more new people to begin participating on Wikipedia, some of whom will learn to do more substantial edits and get involved with their communities.Afterdiscussing the idea of structured tasks with communities, we decided to build the first structured task: "add a link".
Even as we built that first task, we have been thinking about what subsequent structured tasks could be; we want newcomers to have multiple task types to choose from so that they can find the ones that they like to do, and can increase in difficulty as they learn more.The second task we started working on was "add an image".But in our initial community discussions of the idea of structured tasks, the task type that communities desiredmost was a task aroundcopyediting -- something related to spelling, grammar, punctuation, tone, etc.Here are our initial notes from looking into this and discussing with community members.
We know that there are many open questions around how this would work, many potential reasons that it might not go right: what kind of copyediting are we talking about?Just spelling, or something more?Is there any sort of algorithm that will work well across all languages?These questions are why we are hoping to hear from lots of community members and have an ongoing discussion as we decide how to proceed.
A full summary of Research is available on MetaWiki:Research:Copyediting as a structured task.
Background research and literature review can be read in full.
Main findings:
We have identified LanguageTool as a candidate to surface possible copyedits in articles because:
We can get a very rough approximation of how well LanguageTool works for detecting copyedits in Wikipedia articles by comparing the amount of errors in featured articles with those in articles containing a copyedit-template.We find that the performance is reasonable in many languages after applying a post-processing step in which we filter some of the errors from LanguageTool (e.g. those overlapping with links or bold text).
We also compared the performance of simple spellcheckers which are available for more languages than supported by LanguageTool.They can also surface many meaningful errors for copyediting but suffer from a much higher rate of false positives.This can be partially addressed by post-processing steps to filter the errors. Another disadvantage is that spellcheckers perform considerably worse than LanguageTool in suggesting the correct improvement for the error.
One potentially substantial improvement could be to develop a model which assigns a confidence score to the errors surfaced by LanguageTool/spellchecker.This would allow us to prioritize those errors for the structured task copyediting task for which we have a high confidence that they are true copyedits. Some initial thoughts are inT299245.
Read here for more details:Research:Copyediting as a structured task/LanguageTool
We have completed an initial evaluation of sample copy edits utilizingLanguageTool andHunspell.To compare how each tool worked for Wikipedia articles, our research team created a list of sample copy edits for 5 languages: Arabic, Bengali, Czech, Spanish (Growth pilot wikis) and English (as a test-case for debugging).
The precision for Hunspell copy edits were judged less than 40% accurate across all wikis. Suggestions were accurate for 39% of English suggestions, 11% for Spanish, and 32% for Arabic, 16% for Czech, and 0% for Bengali.
LanguageTool first evaluation (V1 sample): LanguageTool currently supports ~30 languages, so only two of the Growth team pilot languages are supported: Spanish and Arabic. LanguageTool's copy edits were judged at 50% accurate or higher across all three wikis. Suggestions were accurate for 51% of English suggestions, 50% for Spanish, and 57% for Arabic.
LanguageTool second evaluation (V2 sample): We completed a second evaluation of LanguageTool as a way to surface copy edits in Wikipedia articles. We evaluated suggested errors in Arabic, English, and Spanish. In the previous evaluation we determined that certain rules often resulted in incorrect suggestions, so we added functionality to filtercertain rules. You can see that we ended with results with a higher level of accuracy than in the V1 sample.
For this evaluation we simply used a list of common misspellings curated by Growth pilot Ambassadors, and then checked for those misspellings in Wikipedia articles. Results looked promising, but we ended up with a fairly small sample in some languages. This might be a solution to help provide coverage to languages that aren't supported by LanguageTool, however, if we pursue this option further we will test again with a longer list of misspellings to see if we can get a more representative & significant results (and a better sense of what level of coverage this solution would provide).
Consider how to better handle highlyinflected andagglutinated languages, which likely won't benefit much from standard spell-checking approaches.
Further improving LanguageTool filters to decrease the number of false positives and thus further improve accuracy.
For languages not supported by an open source copy editing tool, we will consider a rule-based approach, i.e. only looking for very specific errors which could be based on alist of common misspellings. We will set up an additional test to estimate the accuracy and coverage of this type of approach.