How Language Quality Metrics Deliver Smart Localization

In a language quality review program, subject-matter experts examine your translation team’s work and flag problems for correction. But how do these reviewers ensure their feedback is objective, precise, and quantifiable? That’s the role of language quality metrics.

Such metrics provide a standardized rating of your vendor’s work, weighing multiple factors to provide a language quality score. As a result, you gain clear, consistent benchmarks to assess quality and improve performance. At the same time, a strong set of metrics allows for plenty of customization to your specific needs.

What’s the Best Way to Measure Language Quality?

For a flexible yet powerful way of measuring language quality, reviewers can turn to a varied set of frameworks such as Multidimensional Quality Framework (MQM) and Dynamic Quality Framework (DQF). Within these systems, a reviewer typically identifies and classifies errors by looking at three key dimensions:

• Accuracy: Does the translated text convey the same meaning as the source material?

• Fluency: Does the content obey the standard rules of the target language, such as grammar and spelling?

• Terminology: Does the vocabulary match the client’s preferred glossary or term base?

In addition to these core metrics, reviewers may consider other, optional measures of quality if these are relevant to the specific client or task. For example:

• Design: Is the physical representation of text (such as formatting) consistent with the source material?

• Local conventions: Does the translated content comply with the usual practice in a specific locale such as country or region?

• Style: Do the tone and voice reflect the client’s guidelines?

• Verity: Does the translation use appropriate language for the target locale or audience?

When reviewers flag a problem, they can note not just the kind of error, but also its severity—minor, major, or critical. The number, type, and severity of errors all contribute to the translation’s overall quality score.

Putting a Number on Language Quality

To calculate the quality score, a mathematical formula can be used to rate the translation on a scale of 1 to 100. This is a dynamic process, making use of quality assurance tools to score the text in real time as reviewers identify and flag errors.

In scoring a translation, every error discovered receives a numerical value that depends on a range of different variables. However, the importance the reviewer assigns to these variables isn’t set in stone. Instead, reviewers can tailor their weighting to meet different priorities and goals.

• Error type: By default, all flaws in a translation receive the same weight—but some dimensions of quality may count more than others. For instance, if accuracy is the most important characteristic for a given kind of translation, we can preset our values so that mistakes in accuracy receive a higher multiplier than others. Meanwhile, other kinds of error such as terminology, fluency, and style could receive a variety of lesser weights that reflect their relative importance.

• Severity: Each mistake also receives a score that reflects its level of severity. According to industry standards, a minor error has an impact of 1 on a scale of 1 to 100; a major issue has an impact of 5, and a critical issue has an impact of 10. However, the reviewer can modify those values to emphasize or deemphasize different dimensions of language quality—so a critical mistake in accuracy might count for 20 points instead of just 10, if accuracy is especially important for that particular task.

As reviewers detect and score the errors in a text, they divide the total value of all these quality issues by the number of words in the content under review. They then multiply this figure by 100 to produce the overall quality score.   

In assessing the importance of quality scores, the client sets the threshold for success. For instance, a company may consider 80 or more to be a passing grade for one type of content or goal—but anything below 99 might count as a failure for another. In this way, the company can gauge its translation team’s relative performance at different tasks.  

Why Metrics Matter—Now More Than Ever

How do you benefit when your language quality review team uses the right metrics and scoring? Such measurements enable you to achieve multiple goals that enable more successful localization.

Assess, calculate, and compare language quality across different types of content and tasks. Well-developed metrics provide you with a consistent, customizable standard for assessing the quality of all translated content. That gives you a more accurate view of your vendor’s performance, so you can make more informed decisions about resource allotment and strategy.

Track how your translation vendor’s work is trending over time. As the language review team processes more and more content, a QA software tool can graph the results and present them in a dashboard for your review—so, for example, you can see how quality has been improving or declining in the last six months. In this way, you can better understand whether your translation projects are heading in the right direction.

Improve the quality of machine translations. More and more, vendors are using software to translate their clients’ content—but the results of machine translation are only as good as the data behind them. By converting the judgments of human experts into quantitative metrics, your review team provides data that can be used to train and optimize a translation program’s algorithms.

As machine translation becomes the norm, companies may be able to produce greater amounts of localized content, aimed at more audiences than ever before. This increased output will only add to the challenge of quality assurance—and it will become all the more important to use metrics that work. 

If you are interested in finding out more about Language Quality Management or would like to learn how we can help your localization efforts, please contact us below.