How best to explain the uselessness of QA tools?
Autor wątku: Kay-Viktor Stegemann

Kay-Viktor Stegemann
Niemcy
Local time: 00:55
Członek ProZ.com
od 2016

angielski > niemiecki
Jan 30

Many big agencies and even end clients use QA tools (like XBench) on the translations they get and return a "QA report" to the translator with a list of inconsistencies, term/glossary violations and such. I cannot judge the usefulness of these reports in other languages, but in the pair English to German, 99.99% of the "errors" listed are false alarms, mostly due to the fact that a term can have a virtually unlimited number of correct spellings in German, due to changes of word forms for tense, ... See more
Many big agencies and even end clients use QA tools (like XBench) on the translations they get and return a "QA report" to the translator with a list of inconsistencies, term/glossary violations and such. I cannot judge the usefulness of these reports in other languages, but in the pair English to German, 99.99% of the "errors" listed are false alarms, mostly due to the fact that a term can have a virtually unlimited number of correct spellings in German, due to changes of word forms for tense, number, gender, case, and so on, and particularly to the fact that we can create compounds in German where the actual term "disappears" somewhere in the middle of the word. All that means that term and spelling "errors" from these reports can as well be trashed, it makes no sense even to look at them.

But some clients are particularly stubborn and demand explanations for every single deviation from the prescribed glossary terms, or at least an explicit "ignoring" of every single "error". This causes an enormous overhead and extra work without adding a micron of quality. Sometimes I'm at a loss how to deal with this when clients who have no grasp of the target language demand detailed explanations. Do you have similar problems sometimes? Do you explain grammar intricacies of the target language to the client in the source language?
Collapse


Elena Feriani
Matthias Brombach
Luca Colangelo
ahartje
Angie Garbarino
Kardi Kho
Stepan Konev
 

Jean Dimitriadis  Identity Verified
Członek ProZ.com
angielski > francuski
+ ...
False positive Jan 30

Would just adding the explanation "False positive" be enough to both 1. explain the reason for ignoring or otherwise not fixing the "error" ; and 2. highlight the ratio of useless false alarms that this tool produces for you, which could also strengthen your arguments for dropping its use?

The farthest I would go is do a line by line full explanation for a single report, and agree with the agency to use "False positive" going forward (unless a short, if possible reusable, and by no
... See more
Would just adding the explanation "False positive" be enough to both 1. explain the reason for ignoring or otherwise not fixing the "error" ; and 2. highlight the ratio of useless false alarms that this tool produces for you, which could also strengthen your arguments for dropping its use?

The farthest I would go is do a line by line full explanation for a single report, and agree with the agency to use "False positive" going forward (unless a short, if possible reusable, and by no means detailed explanation is needed, like "Terminology variation to account for different in-context meaning").

Finally, maybe the tool can be set in such a way as to avoid producing so many false positives, but I guess you are using a specific set of rules in XBench?

[Edited at 2021-01-30 21:21 GMT]
Collapse


Said Abdi Ahmed
María C Turri
Clement Cheung
 

Inga Petkelyte  Identity Verified
Portugalia
Local time: 23:55
litewski > portugalski
+ ...
Not anymore Jan 30

On several occasions, the time consumption for explanations (and finding references as an evidence) turned out to be way higher than the time for the translation itself. I ended up not doing that anymore. I tell the client to ask a specific question where they might have doubts and that's it. Normally, they don't. For preferencial options, I note that I disagree but if they/the agency are happy, the choice - and the responsability - is theirs. No more rummaging through books, internet, official ... See more
On several occasions, the time consumption for explanations (and finding references as an evidence) turned out to be way higher than the time for the translation itself. I ended up not doing that anymore. I tell the client to ask a specific question where they might have doubts and that's it. Normally, they don't. For preferencial options, I note that I disagree but if they/the agency are happy, the choice - and the responsability - is theirs. No more rummaging through books, internet, official guidelines to prove the obvious.Collapse


Victoria Britten
Sadek_A
neilmac
Philippe Etienne
Christine Andersen
Laura Kingdon
 

Kay-Viktor Stegemann
Niemcy
Local time: 00:55
Członek ProZ.com
od 2016

angielski > niemiecki
NOWY TEMAT
I don't use these tools, the client does Jan 30

Jean Dimitriadis wrote:

Would just adding the explanation "False positive" be enough to both 1. explain the reason for ignoring or otherwise not fixing the "error" ; and 2. highlight the ratio of useless false alarms that this tool produces for you, which could also strengthen your arguments for dropping its use?

The farthest I would go is do a line by line full explanation for a single report, and agree with the agency to use "False positive" going forward (unless a short, if possible reusable, and by no means detailed explanation is needed, like "Terminology variation to account for different in-context meaning").

Finally, maybe the tool can be set in such a way as to avoid producing so many false positives, but I guess you are using a specific set of rules in XBench?

[Edited at 2021-01-30 21:21 GMT]


I don't use these tools and I don't configure them, the client does. If I had any influence on this, I would tell them to switch off term checking and spell checking in these tools since these parts are (for my pair) completely useless (for spell checking, the Word spell checker is a hundred times better because it seems to be able to recognize compounds and even grammar issues). And writing "false positive" a thousand times is a lot of useless work even if I can copy and paste it 


Matthias Brombach
neilmac
Angie Garbarino
Christine Andersen
Laura Kingdon
 

Endre Both  Identity Verified
Niemcy
Local time: 00:55
Członek ProZ.com
od 2002

angielski > niemiecki
Charge them Jan 30

In one of my projects, the end client used to ask endless questions about minute details without having any target language knowledge beyond Google Translate. No amount of reasoning was enough to dissuade them. But shortly after I started billing for the extra time spent, the queries miraculously stopped, saving a lot of time for everyone involved.

Victoria Britten
Sadek_A
Susan Murphy Lamprecht
neilmac
Emanuele Vacca
Kay-Viktor Stegemann
Nikki Scott-Despaigne
 

Mario Cerutti  Identity Verified
Japonia
Local time: 07:55
Członek ProZ.com
włoski > japoński
+ ...
What I do Jan 31

Kay-Viktor Stegemann wrote:
And writing "false positive" a thousand times is a lot of useless work even if I can copy and paste it.

I diligently but quickly check line by line to see if I have actually made mistakes. If really necessary, I correct those mistakes only and give a final general statement in the reply email offering plenty of apologies, then I send the revised file. For the rest, in the same email I simply state that there are no problems.

This usually works. (PMs are generally quite busy themselves and I suspect that they are happier with just receiving the revised file, that is, without having to go through translators' comments one by one.)

[Edited at 2021-01-31 02:19 GMT]

[Edited at 2021-01-31 07:26 GMT]


Kevin Fulton
 

Sadek_A  Identity Verified
Local time: 18:55
angielski > arabski
+ ...
..... Jan 31

Your strategy should comprise the following:
- Making 100% sure in the first place that there were no mistakes, before delivery.
- Pointing out once and for all that the tool was developed by a human brain, and not the human brain by a tool. Which serves to shake their confidence in the tool, even the slightest bit!
- Indicating beyond any doubt that your base rate doesn't cover after-sales service, and that they would need to pay extra on an hourly basis for that added service
... See more
Your strategy should comprise the following:
- Making 100% sure in the first place that there were no mistakes, before delivery.
- Pointing out once and for all that the tool was developed by a human brain, and not the human brain by a tool. Which serves to shake their confidence in the tool, even the slightest bit!
- Indicating beyond any doubt that your base rate doesn't cover after-sales service, and that they would need to pay extra on an hourly basis for that added service, as per your sole discretion! Granted, you are 100% responsible for omissions, disorders and/or verifiable mistakes (verifiable only against the source! not against the whims of client and/or their personnel) in the part(s) you got paid/POed for; but, not for explaining choices and/or clearing mere suspicions!
- Ensuring that the work coming from them is big enough to justify all the hassle.

On a different note: by "micron of quality", did you mean "scrutiny of quality"? Just checking before possibly adding to glossary!
Collapse


Susan Murphy Lamprecht
 

Elena Feriani
Włochy
Local time: 00:55
Członek ProZ.com
francuski > włoski
+ ...
False positives Jan 31

Two of my clients, both from China, send me Xbench reports frequently and sometimes they want me to reply immediately. I have asked them to share their checklists so I run the report before delivering and when they send me the same report back I can just reply: "All false positives". It is expensive at 99 euro per year, but it's worth it in my opinion. I use it for every client now.
In Italian, I have the same issues with false positives due to gender and number or verb form, but at least
... See more
Two of my clients, both from China, send me Xbench reports frequently and sometimes they want me to reply immediately. I have asked them to share their checklists so I run the report before delivering and when they send me the same report back I can just reply: "All false positives". It is expensive at 99 euro per year, but it's worth it in my opinion. I use it for every client now.
In Italian, I have the same issues with false positives due to gender and number or verb form, but at least the spellcheck is very good.
Collapse


Susan Murphy Lamprecht
 

Adieu  Identity Verified
rosyjski > angielski
Wow that's obnoxious Jan 31

I thought having to copy false tags and putting a billion little check marks to ignore punctuation or confirm alphabet changes for transliteration of serial numbers and the like from Cyrillic in memoQ's QA was annoying... but written explanations? For software whose results overlook the rules of the target language? Damn.

Are these well-paying clients, at least? Have you tried billing them for QA time if not?


Chris S
 

Stepan Konev  Identity Verified
Rosja
Local time: 01:55
angielski > rosyjski
Picture is worth a thousand words Jan 31

I have a picture for this purpose.
It shows a list [still not exhaustive] of words that can be formed using the root 'бег' (equivalent of 'run' in English).


Elena Feriani
Elena Doroshenko
P.L.F.Persio
Adieu
WS McCallum
 

kd42
Estonia
Local time: 01:55
angielski > rosyjski
Switch off terminology check Jan 31

Kay-Viktor Stegemann wrote:
with a list of inconsistencies, term/glossary violations and such

I explained every X-th term "error" to the client, very clearly, with an example in their language.
Because the quality of their termbase is ridiculously low, all these were false alarms.

The present obsession with QA tools remind me of the early era of machine translation, when
a) the package was on two 5-inch floppies; and
b) most people were sure it would deliver flawless translations (so you can fire most of your translators).

Then I suggest solutions:
1) Pay me for the time I spend digging in the dirt (QA report);
2) Pay me for the time I spend digging in the dirt (QA report) and for trimming and optimizing their termbase;
3) Ditch the termbase and the terminology check.

All of my clients asked a couple of clever questions and chose item 3.


Adieu
Philippe Locquet
Laura Kingdon
 

Riccardo Schiaffino  Identity Verified
USA
Local time: 16:55
Członek ProZ.com
od 2003

angielski > włoski
+ ...
Xbench is a great tool, but... Jan 31

Xbench is a great tool, and I use it on almost every project I do.

But like any tool, it must be used properly, and the problem is that many users (especially PMs) don’t really know how to use it.


  • Spell check. The usefulness depends on the language - for some it is indeed very useful (English and Italian, among others), for other languages less so. The spell checker in Xbench is especially useful... See more
Xbench is a great tool, and I use it on almost every project I do.

But like any tool, it must be used properly, and the problem is that many users (especially PMs) don’t really know how to use it.


  • Spell check. The usefulness depends on the language - for some it is indeed very useful (English and Italian, among others), for other languages less so. The spell checker in Xbench is especially useful because, unlike the spell checker in other tools (e.g., MS Word, SDL Trados Studio, memoQ, etc.), it provides a list of misspelled words, with one occurrence only of each supposed misspelling. You can then sort the list, quickly select those that need to be further checked in context to add them to the rest of your QA, and just ignore the others. In my opinion, the spell checker should NOT in fact be used by the PMs, since they usually do not know the target language, and therefore are prone to add the entire list - this, indeed, would likely result in an enormous number of false positives.
  • Key terms. Here the usefulness depends both on the quality of the glossary used for the key terms (it should be as short as possible, with only actual technical terms included, and only one translation for each source term), and, again, also on the language. For certain languages it would in fact be better to use project or personal checklists (which allow, with some additional effort, to take into account the vagaries of most languages) instead of key term glossaries. Again, checklists and key terms glossaries are better left to the translator or to a language lead who knows the target language, rather than to a PM who doesn’t.


For a detailed introduction to the use of Xbench for terminology management and translation QA, see this page, with two online presentations on Xbench, from my blog About Translation
Collapse


texjax DDS PhD
Elena Feriani
Darius Sciuka
P.L.F.Persio
Oriana Bonan
 

Chris S  Identity Verified
Wielka Brytania
szwedzki > angielski
+ ...
Useless vs frustrating Feb 1

The trouble with declaring these tools “useless” is that I imagine, like all translators, you make mistakes, and sometimes they will pick them up, undermining your argument...

But I can see that they could be highly irritating, and it seems odd that the customer does not know this and improve them, as ultimately they are the ones who will be paying for all this wasted time.


P.L.F.Persio
Mervyn Henderson
Dr. Matthias Schauen
 

Riccardo Schiaffino  Identity Verified
USA
Local time: 16:55
Członek ProZ.com
od 2003

angielski > włoski
+ ...
Improvements to QA tools Feb 1

Chris S wrote:
But I can see that they could be highly irritating, and it seems odd that the customer does not know this and improve them, as ultimately they are the ones who will be paying for all this wasted time.


ApSIC (the company that develops Xbench) continues to improve the tool, and in my experience they are open to adding features based on customers’ suggestions.

The problem (and the source of frustration for the original poster here) is that many PMs either don’t really know how to use these tools, or don’t care, and send out to translators QA reports that are filled with false positives.

What PMs should do:

- Make sure that the key term glossaries they use are of high quality (containing only the terms that they should)
- Do not run the spell-check test themselves (the translators should, instead)
- Be open to translators not confirming each line of a QA report when there is an unavoidable reason for all false positives

For me there is a big difference between a translator writing something like, for example “I have carefully reviewed the terminology used in this translation, and confirm that it matches the required terminology. All remaining reported mismatches are due to [linguistic reason depending on the peculiarities of source and target language]” and a translator not bothering to look at the QA report. The former is acceptable, the latter not.

But PMs who insist on the translator justifying each and every instance of a QA item left open because the PM didn’t bother to use the QA tool properly should be asked to pay for the additional time they demand from their translators.

[Edited at 2021-02-01 16:58 GMT]


Chris S
Dan Lucas
Laura Kingdon
 

Philippe Locquet  Identity Verified
Portugalia
Local time: 23:55
Członek ProZ.com
od 2013

angielski > francuski
+ ...
1 2 3 Feb 5

kd42 wrote:

Then I suggest solutions:
1) Pay me for the time I spend digging in the dirt (QA report);
2) Pay me for the time I spend digging in the dirt (QA report) and for trimming and optimizing their termbase;
3) Ditch the termbase and the terminology check.

All of my clients asked a couple of clever questions and chose item 3.


Totally agree with this approach, makes total sense.
Many companies sometimes buy into advertisement promises and don't understand what these QA-tools do and what are their limitations. Not to mention language specific false positives, IMO, a proper QA should be performed by linguists fluent in the pair and knowing what they are doing.

Maybe there should be LSP-oriented training sessions, I haven't seen many of these online though.


kd42
 


To report site rules violations or get help, contact a site moderator:


You can also contact site staff by submitting a support request »

How best to explain the uselessness of QA tools?

Advanced search







Protemos translation business management system
Create your account in minutes, and start working! 3-month trial for agencies, and free for freelancers!

The system lets you keep client/vendor database, with contacts and rates, manage projects and assign jobs to vendors, issue invoices, track payments, store and manage project files, generate business reports on turnover profit per client/manager etc.

More info »
SDL Trados Studio 2021 Freelance
The leading translation software used by over 270,000 translators.

SDL Trados Studio 2021 has evolved to bring translators a brand new experience. Designed with user experience at its core, Studio 2021 transforms how new users get up and running and helps experienced users make the most of the powerful features.

More info »



Forums
  • All of ProZ.com
  • Szukaj terminu
  • Praca
  • Forum
  • Multiple search