Assessing the Accuracy of Google Translate to Allow Data Extraction From Trials Published in Non-English Languages

One of the strengths of systematic reviews is that they aim to include all relevant evidence. However, study eligibility is often restricted to the English language for practical reasons. Google Translate, a free Web-based resource for translation, has recently become available. However, it is unclear whether its translation accuracy is sufficient for systematic reviews. An earlier pilot study provided some evidence that data extraction from translated articles may be adequate but varies by language. To address several limitations of the pilot study, four collaborating Evidence-based Practice Centers conducted a more rigorous analysis of translations of articles from five languages. Systematic reviews conducted by the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Centers (EPCs) most commonly restrict literature searches to English language publications. In a sample of 10 recent Evidence Reports (numbers 189-198), 8 were restricted to English-language publications. One report included studies in languages for which the EPC had "available fluency" and only one reported not restricting by language. Among 28 other recent Comparative Effectiveness Reviews (CERs) with final or draft documents downloadable from the AHRQ Web site, 20 were restricted to English-language publications. Four explicitly did not impose any language restriction. Two did not report language restriction in their methods chapter and included one study each in Dutch and German. One placed no language restriction on comparative studies but included only English-language cohort studies. One included German- and French-language studies for nonoperative interventions (which were sparse), but only English-language publications for operative treatments "due to lack of translation resources." Three of the CERs wrote that the language restriction was due to lack of resources or prohibitive translation costs, despite the recognition in one CER "that requiring studies to be published in English could lead to bias." The current study was designed to form a collaboration of EPCs to better analyze the accuracy of the freely available, online, translation tool-Google Translate-for the purposes of data extraction of articles in selected non-English languages. The collaboration allowed for double data extraction and a better consensus determination of the important extraction items to assess; we also implemented an improved analytic technique. The research had the following aims: 1. Compare data extraction of trials done on original-language articles by native speakers with data extraction done on articles translated to English by Google Translate. 2. Track and enumerate the time and resources used for article translation and the extra time and resources required for data extraction related to use of translated articles.