Rationale

Magnolia 5.0 went slightly backwards - we lost some translations. The mechanisms are still in place, but we "forgot" to implement them; many dialogs are configured with "hardcoded" labels. Some code, too.

More importantly, the way we apply i18n through the UI is currently inconsistent or inexistent. Most new constructs (apps, action bars, ...) have no notion of i18n. Blindly applying the same slightly dated concept we've been using up to 4.5 would blow the configuration out of proportion. Below are a few suggestions to explore and make this simpler, smaller, or more consistent.

Analysis of current situation

getLabel() vs getI18nBasename() vs usages 

Each translatable item currently has a getLabel (or getTitle or getWhatever) method, some have several (getDescription). They also have a getI18nBasename() method. This combination makes it unclear whose responsibility it is to actually translate (i.e call MessagesManager) the piece of text. Should it the the FieldDefinition itself ? The FieldFactory ? The FormBuilder ? Something we hide in Vaadin (we could wrap com.vaadin.ui.Component#setCaption)

Would it help if we actually removed that stuff from our APIs and use an annotation to indicate which configured Strings need to be translated ? (e.g add an annotation on info.magnolia.ui.form.field.definition.FieldDefinition#getLabelthe title, t

Different translation "scopes"

We need to differentiate between in-code translatable text, in-template translatable text and in-config translatable text.

In-code translations

Some UI elements are not configured. Their texts can be considered "hardcoded". IntelliJ (Eclipse too, probably) provides analysis tools to find out where i18n strings have been hardcoded. Attached is a sample report executed on a couple of modules: i18-analysis.zip (main, ui, activation, cache, imaging, workflow). Note that the html-exported report is not very useable, but the in-app report is much more practical: Screen Shot 2013-07-31 at 14.21.14.png

My initial thought was that for these, we could benefit from a tool like Localizer or GWT's i18n generator tool. Unfortunately, these tools generate code (interfaces and impl) based on the keys found in a message bundle file. They are well thought out, in that they provide the correct methods depending on parameters found in the keys, for example. (generate a String getFileCount(int count) based on a file.count=There are {0} files message, for example). This is great for code completion and type-safety. However, their generated code isn't ioc-friendly (tends to rely on static/threadlocal for Locale retrieval), and tends to be laced with static dependencies. These tools also don't handle changes to the generated code well. Which means that if you need to add a message - or change its name, or change its "signature" - you need to edit the properties file, rather than the code.

If we end up having a lot of these, or if we have some extra time, we could think about rolling out our own tool - or write code that behaves similarly to that sort of generated code. Typically an interface with methods like String getSomeKey() and String getFileCount(int fileCount) and the tool would generate the implementation and the message bundle file.

In-code translations also include things like the login form and some other AdminCentral templated components, where the i18n call might currently be done in the FreeMarker template. magnolia-ui-admincentral/src/main/resources/mgnl-resources/defaultLoginForm/login.html: it's actually hardcoded.

The login form also has this special construct for translating LoginException messages - to be taken into account.

In-template translations

These are translations that are meant to be "consumed" by a site visitor; i.e used in template scripts. Some template components of STK which need a translated text item are not meant to be translated by authors; "Skip" or "Read more" type of labels.

Those currently suffer from the fact they are in the same message bundle (i18nBasename) as their respective template definition; as such, if one wants to change those text items for a given project, they have to either copy the complete STK message bundles, or customize the components. Neither is ideal. Currently, at least for FreeMarker templates, we pass definition.getI18nBasename() to info.magnolia.freemarker.FreemarkerHelper#addDefaultData, which adds an i18n object to the FM context with the definition's message bundle.

It should be possible for templates to use a different message bundle than the one used to translate the definitions' labels and descriptions.

Ultimately, we probably want authors to be able to translate these items; this is where a JCR-based translation tool (and/or MessageManager implementation) would make the most sense.

In-config translations

Translations in-configuration will require very little change to existing code, but will require some work on update tasks and bootstrap files. It is however where the bulk of this concept is focused - we're changing the way translation are applied quite drastically.

Status in 5.0

A complete code review is required to identify all places with a direct String output (such as button labels, column headers, etc.). Such places have to be replaced with a proper i18n mechanism (MessagesManager.getWithDefault(key, defaultMsg), where the original String will be used as defaultMsg).

A complete manual code review is not necessary to discover where to apply i18n. A search for usages of the following (to be completed) should cover 95% of our bases.

• info.magnolia.ui.form.field.definition.FieldDefinition#getLabel
• info.magnolia.ui.form.field.definition.FieldDefinition#getDescription
• info.magnolia.ui.form.definition.TabDefinition#getLabel
• com.vaadin.ui.Component#setCaption (39 usages in the codebase I have)
• info.magnolia.ui.dialog.FormDialogPresenterImpl#buildView
• ...


Convention over configuration

We want to introduce a naming convention, both for basenames (i.e the location of the message files) and the key themselves. That convention already exists in a informal way; most items in STK for example have a fairly consistent key naming scheme.

To avoid a lot of redundant and verbose configuration, we could take this one step further and use the conventional name - if none is configured - to lookup a particular text. This will also help for cases where the dialog configuration is (very!) redundant (99% of dialog actions are "save" and "cancel" - these are currently configured for each and every dialog, making maintenance a nightmare)

With some clever fallback mechanisms, this could make translations easier and simpler.

Suggestions for a fallback chain are given below. Given a bundle, the keys are "tried" in order. The first value to be found is used.

try {
   ... something that fails ...
} catch (SomeException e) {
   uiComponent.alert(e.getMessage());
} 

try {
   ... something that fails ...
} catch (SomeException e) {
   // if there is added value: log.error("sumfin happened: {}", e.getMessage(), e);
   // don't feel forced to bloat the logs, though.
 
   // if this is something the *user* should be informed about, and *can* do something about:
   uiComponent.alert(i18n.getMessage(e));
 
   // but really, is it ? Can a user do anything about a JCR Exception ?
   throw new RuntimeException(e); // did you need to catch it in the first place ?
} 

Implementation

Introducing a couple of concepts. The basic API will be in its own module in magnolia_main; it doesn't need to be in core, and will maybe not even depend on it. It could even be outside of magnolia_main if there is no dependency to core, but we currently lack a good location for such modules 

Module and package

magnolia-i18n in magnolia_main contains the API and the underlying. Key generators live near their counterparts. (i.e in _ui mostly)

The main package is info.magnolia.i18nsystem as a package name.

API

• @I18nable annotation. "Internationalizable": marks any object as a candidate for translations. Used on interfaces/classes such as FieldDefinition. Is inherited.
• @I18nText annotation. Marks a String as to be translated.
• I18nKeyGenerator<T> interface. Implementations generate translation keys for <T>.
• TranslationService
• I18nizer. Transforms/decorate a given object and internationalizes it.
• I18nParentable. This is an interface that allows setting a "parent" on an object. This interface should not be used by client code.

Implementation "details"

• Tried to implement a Guice module that would do this transparently; also tried to implement this at Node2Bean level. Neither worked - didn't find a way to "swap" instances in Guice (i.e replace bean created by n2b by a proxy)
• I18nizer thus needs to be invoked explicitely. We have one implementation based on ProxyToys which creates a proxy. It intercepts methods that return other @I18nable objects (by returning one instance, a collection, or a map where value are of an annotated type) and decorate those. Those proxies in turn intercept method calls annotated with @I18nText and returns translated values by delegating to i18nKeyGenerator and TranslatorService.
• I18nizer also "injects" the I18nParentable interface and sets the parent object while intercepting calls.

Update tools and tasks

To migrate our own modules, we can write a tool which:

• Starts up a repo and a specific (or several) component managers
• Load up a translation file we want to migrate (or all its language counter parts)
• Imports a bootstrap file we want to migrate
• This should instantiate a whole bunch of forms etc
• Go through these one by one
  
  • Go through each @I18nText property of the object
    
    • Does it correspond to a key currently existing in the translation file ?
      
      • yes: replace key in translation file by deduced key, remove property from jcr
      • no: add key in translation file
    • If it's a key but it doesn't have an existing translation
      
      • add deduced key to translation file (instead of configured key), remove property from jcr
    • If it's not a key, i.e current configuration has an "hardcoded" text
      
      • warn, blow up, panic, ...?
      • add deduced key to translation file (instead of configured key), remove property from jcr
  • Track unused keys in translation files
  • Track possible duplicates
• Re-export file to replace bootstrap file
• Re-export translation file(s)

The same tool could maybe be used to generate update tasks, or generate some sort of config/mapping file passed to a specific update task: it will only need a list of properties to be removed from nodes, with their original value so that we don't remove a property that's been modified by a user.

Message bundles for non-english texts

Bundle languages in separate jars

• english/master language still bundled with each module, other languages bundled in 1 jar (1 jar per language, containing translations for many modules)
• maintenance is somewhat easier
• but at the same time we might get "dependency" issues when modules add/remove keys
• if we have tools for migration/validation of existing translations, the same tools could be used, perhaps as a maven plugin or sthg.
  
  • such a tool could potentially help enforcing compatibility between versions (i.e keep a key that was removed in version X+1 of module M)
• Need some sort of version handling - keep keys for older versions, add keys for newer ones ...
  • Chain (overlay) current translation file with older ones ?

Translation processes

• Enable inline translations within a dialog
  
  • I'm not sure the current proposal would work to enable inline translations. It'd be nice for translators to have at least a hint of what the key used for a specific item is. And if we somehow have elements explicitly use the KeyGenerator and other API methods for this, we might as well get rid of the proxy magic and use consistently... 
• Review process for in-house maintained translations as well as for contributed ones (currently relying on Google spreadsheet)
  • Have a Magnolia-hosted tool to replace the google spreadsheet
    
    • some rules like "a translation needs to be validated by 2 other persons to be applied", "once applied it can't be changed directly - only via a 'request'", ...
    • could have a "MessagesManager" impl that fetches translations from this service

Roadmap

• 5.1 : system in place, translations migrated for AdminCentral and most modules (DAM, STK, ...)
  • Module updates can be postponed
• ? : extract languages other than english into language-based bundles - needs a separate concept.
• ? : review processes for translating Magnolia, both internally and externally. Get contributions. With language files extracted from their modules, it might be easier. - needs a separate concept.

Status

Main issues:

• MAGNOLIA-5268 - Getting issue details... STATUS
• MGNLUI-1826 - Getting issue details... STATUS

 Validate Concepts

 Validate Roadmap

 API

 Finalize module name, location, and package name.

 Implementation details (proxy, Guice module, ...)

 Parental relationships

 How does this work with Multi/composite fields

 Check things like info.magnolia.ui.form.field.definition.DateFieldDefinition#getDateFormat - might actually work with @I18nText or another annotation

 Is there a conflict with merged objects ? (i.e template definitions merged with site def prototype - but also look at other merge cases!!)

 An i18Basename could also be defined in site definitions (for in-template translations) - does this work ?

 Clarify what to be with i18nBasename properties - the current tendency is to get rid of them. Adapt update tools as needed.

 Replace MessagesManager with a cleaner impl

 Migration and updates

 While this implementation has focused on definition objects, (some) "live" objects already know their parent (see for example info.magnolia.ui.form.FormItem) - validate and justify the choice to decorate definition objects.

Known bugs or problems:

• MAGNOLIA-5315 - Getting issue details... STATUS
• MAGNOLIA-5317 - Getting issue details... STATUS
• we have no proper way, yet to i18n-ize GWT- / Vaadin-Client-Classes; neither info.magnolia.cms.i18n.MessagesUtil nore the i18n-toolz can be applied on those client-classes

Topics to validate or research

• A form can be used in different contexts. It should be translatable according to context.
  
  • TBD: details, examples

Existing uses and inconsistencies to fix for 5.1

•  info.magnolia.ui.form.AbstractFormItem#getMessages - should not be public
  
  • Is broken: doesn't use the user locale afaict.
•  info.magnolia.ui.form.AbstractFormItem#getMessage - should not be public
•  AbstractFormItem defines a semi-arbitrary message bundles chain (see AbstractFormItem#UI_BASENAMES)
•  Definition objects( TabDefinition, etc) have an i18nBasename property, which is very redundant with that of the "runtimes" objects ( FormTab, ...). Usage seems consistent ( return definition.getI18nBasename();) but I don't know why this isn't implemented in info.magnolia.ui.form.AbstractFormItem.
•  Definition objects don't have a common interface. If they did, we could move i18nBasename and label in there. OTOH, some of these objects have more than 1 item to translate (label and description, for example).
•  view.addFormSection(tab.getMessage(tabDefinition.getLabel()), tab.getContainer()); ...translates the message from the tab and passes it translated before it's actually "displayed". (while the method argument is called tabName not tabLabel - but that passed object becomes an argument called caption later down the stack) - the below would make this sort of code much more explicit. You pass an object meant to be a label. You translate it explicitly - most likely at the last possible moment. Or we even extend Vaadin component so that they know about I18nItem.
•  getI18nBasename is defined in too many places. It's inconsistent and unintuitive. Why the redundancy between info.magnolia.ui.dialog.Dialog#getI18nBasename and info.magnolia.ui.dialog.definition.DialogDefinition#getI18nBasename for example ?
•  info.magnolia.ui.form.FormBuilder#buildForm still does "StringUtils.isNotBlank(description)", but since this is now decorated, it will never be blank or null. See  MAGNOLIA-5315 - Getting issue details... STATUS  though.

Suggested key patterns

The below is a rough outline. The 3 goals below are somewhat hard to reach all at once.

• avoid redundancy or noise (no dialog. prefix, ...)
• avoid conflicts (but allow them - on purpose - for labels that are actually meant to be the same in most situations)
• be consistent (this part is hard - sometimes we prefix with module name, sometimes with app, sometimes with nothing)

Note: as the dialog names usually follow the pattern moduleName:dialogNameWithinTheModule, the <dialog-name> part of the keys mentioned bellow is in fact <module-name>.<dialog-name-within-the-module> (as the ':' character cannot be part of a key).

Field labels: – optional fallback to a key without a .label suffix to make things less verbose

<dialog-name>.<tab-name>.<field-name>.label

<dialog-name>.<tab-name>.<field-name>

<dialog-name>.<field-name>.label

<dialog-name>.<field-name>

Field descriptions - here we can't fallback to a key without the .desc suffix

<dialog-name>.<tab-name>.<field-name>.desc

<dialog-name>.<field-name>.desc

Tab labels:

<dialog-name>.<tab-name>.label (or .tablabel for explicitness?)

Dialog (Form) labels:

<dialog-name>.label

<dialog-name>

Action labels:

 99% of our dialogs have the same save/cancel actions. Those should be defaults. Labels should still be overridable on a dialog-per-dialog basis.

<dialog-name>.actions.<action-name>.label

<dialog-name>.actions.<action-name>

<app-id>.label

<app-id>.icon # because the icon might have to be localized

<app-id>.description # because a mouse-over title of the app might be interesting ?

app-launcher.<app-group-name>.label # we use the app-launcher prefix, as if app-launcher was an app (which we should consider considering, I suppose)

<module-name>.<template-name>.title # I think "title" is what we've been using in 4.x - we could use label for consistency, or simply name

<module-name>.<template-name> # same as above

<module-name>.<template-name>.desc # Useful in template selector

<module-name>.<component-name>.title # see remark above

<module-name>.<component-name>

<module-name>.<component-name>.desc

<app-id>.<sub-app-name>.views.<view-name>.<column-name>

Suggested i18nBasename patterns

• Defined in module descriptor
• /mgnl-i18n/<module-name>/messages

Dismissed proposals

Here are a couple of notes about things I tried and rejected

Change info.magnolia.ui.form.field.definition.FieldDefinition#getLabel. Node2bean should be made aware of I18nItem class (by registering a transformer)

String getLabel();

into this

 I18nItem getLabel();

• N2B would instanciate these
• it should be made aware of its "parent" (FieldDefinition etc)
• via injection, the impl would know what locale to use and delegate to MessagesManager (or its replacement)
• we'd have a default null-pattern type of implementation (which returns the key or an empty string, or sthg else - this could even be swapped depending on dev mode etc).
• Would this impact performance/memory usage ? 
• Quick prototype of this attached - doesn't do anything, and doesn't know about Locales or message bundles - just showing the "structural" change on classes like FieldDefinition (patch file - disregard class names and packages !)

Relying on N2B will not work

• can't really get the parent into the object (children objects are instanciated first) - not without modifying usage code (ie in setLabel() { label.setParent(this) }, which would suck)
• can't really transform single properties other than with beanutils, which has even less notion of context, and whose usage is currently hardcoded in core 

Duplicate keys

Ideas of supporting things like some.key=${some.other.key}  - we have a lot of such redundant messages in english.

• mention of the CZ issue where in english a translation would be the same in 10 place but needs to be adapted in czech.
  
  • somewhat moderate issue if the mechanism is only supported with the same file - but why would it be
• sorta conflict with the deduction of keys (you would use the most "low" common key for all those same translations)
  
  • makes the CZ problem perhaps more difficult, since the translator then can't rely on keys being defined in the english file