Wordbee Beebox White paper

Add Translation to Your Content, Web Application or Software Written for Technical People June 22, 2015

http://www.wordbee.com http://www.beeboxlinks.com Copyright Wordbee SA

© 2015 Wordbee

1

1 Welcome This white paper provides insightful information about adding translation capabilities to content management systems, software applications, SaaS solutions, or any other type of product you might develop. Maybe you are developing a travel web site; selling books online, or working with cars... Whatever the precise nature of your business, you most likely manage a web site or application containing a sufficiently large number of articles, products, or content pages. In this scenario, translation quickly becomes a challenge. Whether multilingual content already exists or not, you may have the impression that adding translation is expensive, inefficient, or difficult to organize.

Your web site or software. English Only! Beebox

Your web site or software. English, French, Chinese, Russian, Spanish +++++!

Beebox provides a simple, effective way to make your web site, shop, mobile app or software multilingual. Built by developers for developers, Beebox delivers an exceptional toolbox to make fulfill your multilingual needs.

Table of contents 1

Welcome ........................................................................................................................................... 2

2

About Beebox ................................................................................................................................... 3

3

The Inner Workings........................................................................................................................... 5

4

Developers Guide.............................................................................................................................. 8

5

Machine and Human Translation ................................................................................................... 12

6

Licensing.......................................................................................................................................... 15

© 2015 Wordbee

2

2 About Beebox Beebox is a Microsoft Windows application, which can easily be installed to a development PC, a test server, or a production server. It not only takes care of translating your content, but also supports machine translation and human translation workflows. To use Beebox, you will first need to create a Beebox project for your data. The project exposes two file directories: IN and OUT. The “IN” directory is used to copy texts or files for translation and the “OUT” directory is the storage location for translated files. This is depicted in the following diagram:

Your data/system

File copy, Api, Ftp, Dropbox...

Beebox project

(1) Send original files/texts English

Here your contents gets translated! Machine translation: Yes/No

in

Human translation: Yes/No Count words and cost: Yes/No

(2) Receive translated files/texts Chinese French Xml, Html, Json, Flat files, Php, .Net, Csv, Strings, Resource files, DB dumps, Word, Excel, Powerpoint, Indesign and many more

Remember and re-use translations!

out

Spanish ...

Any type of content or files may be sent to Beebox and translated files will have the exact same format as the original. For example, if you send an English PHP file to “IN”, a translated PHP file will be placed in the “OUT” directory. Beebox is capable of identifying what content needs to be translated. It further protects HTML, JavaScript, server, or other format-specific data and ensures that translators cannot alter this information. Beebox supports almost any file format out of the box including:

XML, Json, HTML, PHP, ASPX, Javascript Files, Razor, Code Files (c, java…) Flat Files, CSV, String Files, iOS Files, Java Properties, INI Files Word, Excel, PowerPoint, InDesign, Open Office, RTF, DITA, .Net resources, XLIFF… Each format may be customized to precisely fine-tune what needs to be translated by Beebox. For example, XML formats let you specify XPath expressions, HTML-based formats let you specify the HTML tags or attributes to translate, and still other formats use regular expressions to include or exclude certain strings. You may use the Beebox Web API (recommended) rather than performing steps to copy the files. Please see Chapter 4 for more details.

© 2015 Wordbee

3

2.1 Examples Translating texts contained in a database Let us imagine that you have a database storing product descriptions in HTML format. Your database is already designed to store texts in each of your required languages; however, the translations are missing! To solve this problem, you can write a tool or script that dumps the database to an XML file (CSV, Excel, or any other format). The XML might contain one node per product and an attribute containing the product id. First, copy the file to Beebox. As soon as the content has been translated, you can download the translated files, one per target language. The final step is to store the translations back into your database (i.e. The product id mentioned earlier lets you locate where to insert each product). Since Beebox remembers every completed machine or human translation, you are able to send a full dump as many times as needed without having to pay for translation again. Beebox locates the new or updated texts without requiring input from the user so that only the identified texts will go into a translation workflow. Of course, it is not necessary to dump the entire database. If you track which products are changed, it may be faster to extract only changed or added products to a single file, or one file per product. Translating software string or HTML Copy your HTML, PHP, ASPX, or other files to Beebox and it will again reuse previously completed translations to ensure that only the latest changes are translated. The resulting translated HTML files perfectly reconstitute markup, JavaScript code, or server code. You can customize Beebox to extract translatable JavaScript strings or server code strings next to the basic HTML contents.

2.2 Why Beebox? Beebox is the easiest means you will find when it comes to adding translation capabilities to your software or solutions. This is because Beebox does the following:         

Supports several file formats. There is no need to convert your content. Generates your translated files in the same original format. Remembers all translations, never pay twice for translating the same piece of text. Finds unique texts: A string like “Click here” may show up 500 times in your software. Beebox will send the string just once for translation. Supports a wide range of machine translation systems. Supports human translation workflows via XLIFF or direct web link to translation vendors. Includes smart alignment technology. Translated content can be aligned on the fly. Includes a web user interface to monitor operations, locate, filter, and change text content. Easy to integrate and offers multiple choices, like: File copy, Web API, Http callbacks, and PowerShell scripts.

© 2015 Wordbee

4

3 The Inner Workings The system may be configured to use any combination of machine translation1, pseudo translation2 and human translation. It adapts to your exact requirements. During development you will likely want to use a workflow that employs pseudo- or machine translation and skips any cost incurring human work. In production you will want to add human translation. Here is a visual explanation of what happens to files placed in the “IN” directory or sent via the API:

1. Text extraction

Text segments English

English

Original file

All (translatable) text is extracted from the file. The result is a list of segments, typically paragraphs or sentences. These segments are file format independent.

Product Alpha We are happy... Click here Unmatched ...

2. Pretranslation The translation memory remembers every single translation done in the past. We look up each segment to see if a translation already exists.

TM

French

German

Produit Alpha

Alpha Produkt

Nous sommes...

Wir sind...

Cliquer ici

Hier klicken

Chinese

Unschlagbar...

3. Machine translate French German Chinese

Untranslated segments can optionally be machine translated using Microsoft, Google or other systems. Preliminary translated files can be retrieved at this stage.

French

German

Chinese

Produit Alpha

Alpha Produkt

阿尔法产品

Nous sommes...

Wir sind...

我们很高兴...

Cliquer ici

Hier klicken

点击这里

Inégalée ...

Unschlagbar...

无与伦比的...

Preliminary translations

Quality check

4. Human translate French Missing or machine-only translations can optionally be sent to translators or LSPs. The result are human quality translations.

5. Delivery French German Chinese

The translated files are created from the translations. Optionally notify by email, callback urls or custom scripts.

German

Chinese

Produit Alpha

Alpha Produkt

阿尔法产品

Nous sommes...

Wir sind...

我们很高兴...

Cliquer ici

Hier klicken

点击这里

Inégalée ...

Unschlagbar...

无与伦比的...

Inhouse staff can use the web based Beebox UI to do quality checks, searches, fixes... This step is optional.

Final translated files

1 2

The Beebox interfaces with major online machine translation systems such as Google and Microsoft. To simulate translation workflows. « Translates » by converting text to lower or uppercase. Shift letters, etc.

© 2015 Wordbee

5

3.1 What is the purpose of the translation memory? As you can see, all translations ever completed in a Beebox project are remembered. The system also keeps track of quality levels. It knows which translations have been approved by a human translator or proofreader and which ones have been completed by machine without approval. We have further seen that Beebox splits your text content into smaller segments, typically the size of a sentence or paragraph. If a specific sentence was translated into Chinese earlier and the exact same sentence shows up in another file, Beebox can pre-translate this second sentence. Think of a text such as “Click here”. It may show up hundreds of times in a web site, but Beebox will send it to a MT or a human translator just once!

English

French

Alpha Produkt

阿尔法产品

We are happy...

Nous sommes...

Wir sind...

我们很高兴...

Click here

Cliquer ici

Hier klicken

点击这里

Unschlagbar...

无与伦比的...

Hello world

Bonjour tout le..

Hallo Welt

你好世界

Welcome!

Bienvenue!

Willkommen!

欢迎！

Original segment (sentence, phrase, link...) Human translation or approved Machine translation, unapproved Not translated

Chinese

Produit Alpha

Unmatched ...

TM

German

Product Alpha

Language: German Text: Bienvenue! File: products\welcome.htm Origin: Machine translation Human approval: None Context: {Text before + text right after} Flags: Lock translation, Bookmarks, Comments

When a new or updated file is added to Beebox, it will look up each « segment » in the translation memory. Even if the file is completely new, there is very high likelihood that some translations from other files can be reused. This helps to reduce overall translation costs. A Beebox project can be configured for different leveraging modes:  Leverage translations of identical texts from wherever they come from, like in the “Click here” example above. This is the default mode and fine with most content.  Leverage translations only if the source text is identical as well as the context, i.e. the segment above and below the text. Sometimes translations are done differently depending on the context. This mode produces potentially higher quality, but also more segments to translate.

As a developer, these details are not as important as it is typically the language service provider who makes this choice.

© 2015 Wordbee

6

3.2 What happens if I send a new version of an already translated file? Beebox will start by extracting the text content and split it into segments. It then attempts to pretranslate all segments from the translation memory. If the new file version did not change, then Beebox will find all translations in the memory. It is then consequently able to fully translate this information and send back the translated file to you. Otherwise, it will pre-translate all unchanged segments and move the added or changed segments to MT or human translation. For example, if a single word is replaced inside a huge iOS strings-file, only the one string containing the changed word will be sent for MT or Human translation. All the other strings are reused from the translation memory. As you can see Beebox is a safeguard for spending money on translation where this is not necessary. When Beebox pre-translates a piece of text (segment), there may be multiple choices. To maximize pre-translation quality, the system uses heuristics to choose the best translation: -

It first attempts to pre-translate a file from the previous file version (if one exists). It gives more weight to approved translations. It gives more weight to pre-translations when the context matches (text before and after). It gives more weight to pre-translations done or post-edited by human translators. It gives less weight to translations with QA (quality assurance) problems

3.3 Does Beebox run automatically and unattended? Yes, Beebox is capable of running automatically with little to no human intervention – like a true black box. However, you can also configure Beebox to wait for confirmation at certain steps in the workflow. Examples are: 1. Confirming work before it is sent to translators (e.g. to check cost). 2. Validating work received from human translators. In the second example, you would connect to the web interface of your Beebox to filter translations, run checks, manually correct problems, or send the translation back to the translator. In general, it is recommended to automate all steps.

3.4 Alignment - Can I send already translated files? Yes, Beebox incorporates alignment technology as translated versions may be sent with your source files. Beebox splits source and translations into segments and aligns them. In other words, you can send existing translations to Beebox without having to create translation memories in advance (using 3rd party alignment software). It is as simple as that! The ability to include translated files also serves another purpose. Imagine that a file had been translated in the past. Now, someone edits the translated content (outside Beebox) and changes one or two sentences. The next time you send the source and translated files to Beebox, it will align the files and identify the changes in the translation. The changes go to the human translation team for approval. This use case is specifically interesting with CMS connections where CMS proofreaders may

© 2015 Wordbee

7

want to edit the translated content and then send the content to Beebox so that its memories are updated.

4 Developers Guide Beebox is translation software that was built by developers for developers. We know how hard it can be to integrate third party products or libraries. Therefore, our design goal was to make your life as easy as possible. In this chapter we outline the multiple means to interface with Beebox. We encourage you to choose the methods and options that you feel most comfortable with. View online documentation.

4.1 File copy Integration truly is simple with the File Copy option. With this feature, you can copy files requiring translation directly to the “IN” directory, wait, and then fetch translations from the “OUT” directory. If Beebox is located on a server, you may do this transfer using a Windows file-share, FTP, Dropbox, or other file sharing system. For evaluating Beebox, we suggest you install it on your development PC.

4.2 Web API If copying files sounds too “primitive” in your ears, then please check out the web-based API. In most scenarios, you usually only need 6 to 10 methods to complete translation objectives. By default, the API listens on port 8089. The first method you will need to use in the web-based API establishes a connection and obtains a connection token. Here are a few examples: (GET) http://localhost:8089/api/connect?project=...&login=myname&password=whatever

To send a file to the “IN” directory, use: (PUT) /api/files/file?token={token}&locale=en-US&folder=&filename=products\product1.xml

To poll which files have been fully translated and ready in “OUT”, use: (GET) /api/workprogress/translatedfiles?token={token}&filter=&skip=&count=

To download a translated file, use: (GET) /api/files/file?token={token}&locale=fr&folder=&filename=products\product1.xml

There are also calls to obtain cost quotations in case the human translation workflow is linked to a Wordbee Translator platform, which is typically owned by a freelance translator or a language service provider.

© 2015 Wordbee

8

A word on security: You assign dedicated API credentials to each Beebox project. This is useful if your project requires credentials to be handed out to your customers. Imagine if you gave them software, which needs to call your Beebox API. When you connect to the API with a set of credentials, the caller can access the associated project data only.

4.3 Instructions files With each source file you save to the “IN” directory, you can include a so-called “instructions file”. This optional Json formatted file, lets you:  Specify the target languages for translation. By default, a file is translated into all project target languages. This is helpful in instances where certain content must be translated into certain specified languages.  Specify a deadline for human translation steps.  Request alignment of source file and translated file(s)  Exclude the file from any machine translation or pseudo-translation operations (kind of dummy automatic translation).  Specify that file content shall never be sent for human translation. This is useful if machine translation is sufficient for some content, but not sufficient for others. Suppose the source file is “products\product1.xml”, then the instructions file might be: Products\product1.xml.beebox { "locales": [ "fr" ], "disableMT": true } Suppose translated files are saved to Beebox for alignment purposes. The instructions include the “align” node with the target languages in your saved translations: Products\product1.xml.beebox { "align": { “locales”: [ "fr" ] } }

4.4 Callbacks You can configure Beebox to call a web URL with any important events. For example, Beebox can notify your systems when a file was translated and saved to the “OUT” directory. 1. New translated files!

Beebox project 2. Beebox calls your url

Http://myserver.com/notify?event=translation&date=...

out 3. Get translation from « OUT »

© 2015 Wordbee

9

The callback mechanism makes it unnecessary to poll Beebox at regular intervals in order to know whether translations have arrived.

4.5 PowerShell scripts PowerShell scripts are another mechanism that will prove to be beneficial in certain integration scenarios. You can develop a script that is automatically executed by Beebox whenever new files to translate are received and whenever files have been translated. 1. New translated files!

Beebox project 2. Beebox calls your Powershell script

out Myscript.ps1

The script can be used to transfer translated files to another server or location, to initiate an FTP, or to do whatever is needed to integrate translations into your systems. Learn more.

4.6 Microsoft.Net extensions Extend Beebox functionality using c# and Visual Studio. You can code translation algorithms, custom filters and more, and upload your code to a Beebox. From a developer point of view, things are quite straightforward: Start with creating a class library project in Visual Studio. Add one class per extension. Finally, you compile your project and upload the dll to the Beebox server. Learn more.

4.7 Email notifications Finally, Beebox can send out email notifications, such as “File translated”, “Texts sent to translator”, “Texts received from translator”, etc. for easy identification of a translation’s current status.

4.8 A word on projects and languages One Beebox installation can manage any number of Beebox projects in any languages. Each project is for one specific source language, which is the language in which content to translate is written. A project can have one or more target languages, which are the languages into which content shall be translated. If your original content is for multiple source languages, simply create as many projects as needed to manage the current number of source languages.

© 2015 Wordbee

10

Languages are managed with their ISO two letter (sometimes three letter) codes. You can use the Beebox API to obtain a complete list of language codes together with their English names. Languages are expressed in their neutral form (“en” for English) or with a region indicator (“en-GB” UK English, “en-US” US English). Both are commonly used codes as you can see by viewing this page http://en.wikipedia.org/wiki/List_of_ISO_639-2_codes

4.9 Testing and production It is fairly simple to work with Beebox during development, testing, and production. Your first tests To begin trying out Beebox, you will first need to download and install Beebox to your development PC. Create a project and configure an automated workflow for “pseudo-translation”. This simulates a zero-cost translation workflow: All source texts are simply converted to uppercase, lowercase or letter shifted. Now, whatever you put into the “IN” directory will be “translated” within seconds. The “translated” files immediately show up in the “OUT” directory. Machine translation is not very expensive and you might consider enabling one of the web based machine translation system such as Google Translate or Microsoft Translator. Development in a team If you have multiple developers who all need to use Beebox, we recommend installing it to a development server. Please note, that a Beebox license includes one production install and one test install. So you are immediately fine! Production environments Install Beebox to your production environment. This step is of course not required if your customers or a language service provider would host Beebox. In production, create the Beebox projects you need and configure the “true” workflows. For example, you may want to enable a human translation workflow or a combination of human and machine translation workflows. Staging environments For staging environments, you can reuse the same Beebox you installed for development or for production. Then simply create dedicated Beebox projects for staging use.

© 2015 Wordbee

11

5 Machine and Human Translation Each Beebox project can be configured with its own translation workflows and level of automation. Please also refer to the user documentation for additional details and helpful screenshots. Basically, a Beebox project can provide these workflow steps:

5.1 Pseudo translation If enabled, new or changed text segments are first pseudo translated. Pseudo-translation refers to conversion of text into uppercase, lowercase, letter shifts or length changes. This is useful for zerocost and rapid test workflows.

5.2 Machine translation If enabled, new or changed text segments are first machine translated. Machine translation refers to online systems such as Google Translate or Microsoft Translator. You can then decide whether machine translations shall be “considered” approved or still require a subsequent human approval or correction. In the former case, the system is told that the machinetranslated files can be created immediately and placed into the “OUT” directory. You would enable this option if your workflow does not require any subsequent human validation.

5.3 Human translation Unapproved content is sent to a human translator, team or language service provider. Please note that the system does not send your original files! Instead it sends all the “unapproved” or “untranslated” text segments earlier extracted from your files in the “IN” directory. Beebox creates translation jobs from those segments. It then provides two workflow options to choose from: a) XLIFF - Use XLIFF to exchange translation work with human translators. In this case, Beebox will automatically create a translation job (or jobs) with all new or changed content. Within the Beebox user interface, you are then able to download the XLIFF files and send them to your translators. Once the translation is finished, the translated XLIFF files are uploaded back to Beebox and translated files are created in the “OUT” directory.

The exchange of the Xliff jobs with a translation management or other system can be fully

© 2015 Wordbee

12

automated by means of hot-folders. In this scenario Beebox would extract new Xliff jobs to one directory and scan another directory for translated Xliffs, all unattended.

b) Wordbee Translator - Use a direct link to a translation management system (TMS). Currently Beebox can link to the Wordbee Translator management tool, which is a web based SaaS solution (www.wordbee.com). With this option, all exchanges between your Beebox and a translator or language service provider can run automatically (if desired). This occurs via the web and with absolutely no intervention from your side.

The Wordbee Translator translation environment. See www.wordbee.com

5.4 Validation In a fully automated workflow, you would not manually verify machine or human translations and the system would immediately create the final translated files. However, if you want to put in place a manual verification step, you can do so. Beebox has a user interface that is capable of finding and filtering approved, unapproved, or QA check translations. If problems are found you can either fix them yourself or flag them for resending to a translator (which would create a new translation job).

© 2015 Wordbee

13

The user interface also lets you preview translated files:

5.5 Delivery Once all translations are approved (automatically or semi-automatically), Beebox creates the required translated files from the translated content. The files are saved to the “OUT” directory and can be retrieved directly from this location or by using the web based API.

5.6 Incorporating legacy translations What if you already have some translated content? You do not want to translate those once again with Beebox, right? The following options are available in these instances: Existing translation memories First of all, Beebox lets you upload legacy translations as an Excel file with one column per language. If a freelance translator or language service provider completed your translations, please ask them to send you the translations as translation memories. This approach typically yields non-perfect results and not all content may get translated from the memories. This is due to potential differences in how text is segmented and marked up in Beebox vs. the translation tools used for the legacy translation. Use Beebox alignment features to build memories on the fly This is both the recommended and the easiest approach. Basically, you would save the existing source files and any existing translated files to Beebox. Beebox will then align each source-target couple and build memories on the fly. With this approach, no manual intervention is needed.

© 2015 Wordbee

14

The process is also designed to leverage legacy memories. Simply upload any memories you have from your translators and Beebox will use them to optimize the alignment process. If you do not have any memories you can also improve the alignment by uploading dictionaries.

6 Licensing One Beebox license grants you the right to install Beebox to one Windows instance for production use. In addition, you have the right to install a second Beebox to another Windows instance for staging or testing purposes. If you require a volume license plan, please contact Wordbee for more details. After downloading the Beebox software, a 30-day trial is activated. This trial may be extended in specific instances and you will need to contact Wordbee to acquire an extension of your trial when needed. Each Beebox installation can manage an unlimited number of Beebox projects, limited only by the hardware on which Beebox is installed. You can further assign dedicated API credentials to each project. This means that when your one Beebox serves 10 customers you can create 10 different sets of credentials and disclose this information to each individual customer. These are the most typical licensing scenarios: You want to add translation capabilities to your own web app or software: Beebox would need to be installed on your servers. You do not need to distribute Beebox with your software to your customers. A single Beebox license is sufficient. Add translation capabilities to your software and distribute the software: In this case you have two options: (a) Distribute Beebox with your software, or, (b) Have your software communicate with your own Beebox installation through the Beebox web API. In the first scenario, you need to obtain one Beebox license per product installation. Develop an off-the-shelf connector to localize a specific CMS: For example, you want to distribute a plugin to a CMS (like Drupal, Sitefinity, Adobe CQ5 …) In that case, the plugin communicates with Beebox via the web API. You would likely not install Beebox at each customer premises (although you could). Instead, you will either need to complete the install of Beebox on one of your servers in order to translate content for all of your customers or have the customers’ own language service provider install a Beebox.

© 2015 Wordbee

15