How to Choose the Right Quality Metrics for Your Localization Project

In a previous post, we touched on how Language Quality Metrics Deliver Smart Localization. In this week’s entry we are taking a deeper dive into the subject as a continuing part of our Product Passport series.

By Cesar Matas Alsina

To assess the quality of translations, you need to know how to measure success—and that means using the right metrics.

The problem is, no single set of criteria fits every project. On the contrary, what counts as a successful translation will vary with the type of content, the goals of your localization program, and your specific business or industry.

Even if you outsource language quality reviews to a third-party provider, it helps to understand what goes into the process. Here’s a step-by-step look at how to select the best quality metrics for your localization projects.     

1. First, define your goals

What makes the “right” metric? It depends on your goals. So before deciding which metrics to use, your language quality review team needs to understand exactly what you want to achieve with a given project or type of content. 

For example, is the content intended to entertain and engage your audience? Or does it mainly need to convey accurate technical information? The more clearly you can define your objectives, the better you can know which dimensions of quality to emphasize.

2. Identify what type of assessment you need

Linguistic quality assessment can take two different forms. An analytical assessment identifies individual problems in a text, then generates a total score based on the number, type, and severity of errors. In a holistic approach, reviewers simply rate a text’s overall quality on a given dimension, such as its clarity or emotional tone.

Each type of assessment has different uses. An analytical assessment is well suited to IT and technical translations; a holistic assessment often makes sense for literature or some kinds of marketing materials.

Whatever you choose, some metrics may make more sense for one type of assessment than another. In some cases, a holistic approach might examine only one or two dimensions of quality, while an analytical approach might use a wider assortment of metrics to analyze the same text.

3. Start simple and build from there

All else being equal, a simple model of language quality is easier to use. As a result, it’s a good idea to begin with just a handful of metrics that apply to virtually any localization project.

The widely used TAUS MQM framework builds on three primary dimensions of language quality: accuracy, fluency, and terminology. By default, an analysis using this framework will attribute equal importance to errors of all three types. However, your language quality review may adjust this weighting to match the type of content and the goals of translation. 

Let’s say a company’s website includes long technical descriptions and lists of machine components, aimed at industrial buyers. In that case, consistent use of the right terms could matter far more than, say, grammar and syntax—so you may want a quality score that accords higher value to errors in terminology, and lower value to errors in fluency.

By contrast, precise terminology might not matter as much for entertainment-oriented content such as stories or game scripts. Errors in fluency, on the other hand, might carry more weight, as they could affect the audience’s ability to understand and enjoy the content.

In short, an effective linguistic quality assessment may only require a few key measurements of quality. Even so, your team still needs to make careful decisions about how much to emphasize each of those metrics.

4. Assess which metrics, if any, to add to your model

For some projects and types of content, your reviewers may need to measure more dimensions of language quality to provide a useful assessment.

If your content needs to follow certain local conventions, for instance, you can add a metric that accounts for that type of error. Or if your company has strict requirements for the tone and voice of its content, style could be an important dimension to measure.

As with the three primary metrics, your choice of secondary metrics will depend on the type of content and your localization goals. If some of these metrics are more important than others, your assessment will weight them accordingly.  

5. Track your results—and be ready to make changes

After identifying the relevant metrics, your language quality assurance team can begin auditing and scoring your content. If your company is localizing multiple types of content, your team will likely use a different set of metrics for each one.

By tracking scores over time, you can see how the quality of your translations is evolving over time and use that information to make better decisions. If your metrics don’t entirely align with your needs, however, these trends could present a misleading picture.

For this reason, it’s crucial to reassess your metrics as time goes on and make tweaks if need be. That could mean leaving some metrics out, or adjusting how much you weigh one kind of error over another. Even if you choose exactly the right metrics in the beginning, your content and your needs may change in ways that require adjustments to your framework. 

High-Quality Metrics Make High-Quality Localization

If you choose your metrics well, your language quality reviews will provide valuable insights to help you to improve your translations and allocate resources more wisely. However, it can be challenging to determine what metrics will work best for your project, and how precisely to weigh them against each other.

Fortunately, you don’t have to make these decisions on your own.

A language quality management provider can discuss your unique requirements and identify the metrics that will best meet your needs. Your provider can also help you track your results and fine-tune how you measure language quality over time, so your localization projects stay on track and on target. 

Can your next software localization project benefit from unbiased language quality management services? Contact Beyont to arrange a conversation about your needs.