Machine translation when copying from one language to another

[IsaiahPaget]

After doing some research on how other CMSs implement this feature, it is often baked into their translation plugin. Some have a separate page that allows you to do site-wide translations, some allow you to auto-translate field by field.

Personally, given the copy flow we have going right now, I think it would make sense to do it on a field-by-field basis. Also, I had heard in the videos and articles I read that site-wide auto-translate often made too many mistakes, and then you have to read your whole site and correct everything.

Requirements:

1. Use ISO-639 codes when choosing the language code instead of a free text field. This is because the machine translation api's require a target language code and often a source language code. This would cut down on mistakes and typos.
2. A config option that allows the user to specify a provider(s) using a common interface.
3. A trait that every MLFormwidget can use on its class to query the api based on the credentials specified in the driver.
4. UI flow that goes something like this.

Keep in mind that the styling would use prebuilt winter widgets and styles.

[LukeTowers]

@AIC-BV add your thoughts here and then once we're aligned on the scope of the proposed feature I'll send you an updated quote for implementing the defined scope for the ML / AI integration.

[AIC-BV]

For us the options are either field by field or entire model at once.
As discussed with @LukeTowers on mail, we decided on the one that is the easiest to implement.

The machine translation feature will allow you to provide a ChatGPT API key in the plugin configuration which will then be used to either bulk translate everything on the form at once OR do the machine translation on a field by field level; whichever one ends up being easier to implement.

So since we have the copy button already, field by field is probably the right way to go because it will reduce the error margin, especially on larger, complex datasets like repeaters/nestedforms.

Since we are going to do this through ChatGPT, I think it still needs a 'live' prompt field, default settings prompt field and a WinterCMS defined prompt as explained at the bottom of my research at #99 (comment)

I think google translate/deepl will not work for JSON objects like repeaters because it will translate the keys

What do you guys think?

[LukeTowers]

@AIC-BV I think it would actually be easier to deal with JSON fields using google translate as we can send each individual value as an API request to be translated rather than trying to send a fully formed JSON blob in a single request.

The UI suggested in #99 (comment) is interesting, but I feel like it's trying to do too much; we should probably limit the scope to achieving just a first initial pass, if editors want to be tweaking the prompt on a per field / value basis then they should probably take that out of the Winter CMS editor and go directly to ChatGPT etc to do that.

What are your thoughts on that?

[AIC-BV]

@LukeTowers I'm afraid the Google Translate (?) will be too basic for the complexity of the content or will give literally translated content instead of understanding the context

[IsaiahPaget]

@AIC-BV Hey there, have you tried translating with Google Translate recently? I think lately Google Translate has gotten more sophisticated. But I have also heard good things about DeepL, which is another api (also has a web app) like Google Translate. Would you mind giving those providers a try on some of the content on your site and letting us know how it does?

[AIC-BV]

@IsaiahPaget you are right, the translations are suprisingly nearly identical

I translated the same paragraph with 3 different tools
As native English speakers ; which one has your preference?

1
A stylish laundry cabinet transforms your laundry room into a practical and beautiful space. At Maatkasten Online, you can design your own cabinet for your washing machine and dryer, completely tailored to your space and surprisingly affordable.
From a floor-to-ceiling built-in cabinet for your washing machine and dryer to a low cabinet with a countertop, you can choose from dozens of clever configurations.
Whether you have one or four washing machines, easily create your ideal washing machine cabinet with the online design tool.

2
A stylish laundry cabinet turns your laundry room into a practical and attractive space. At Maatkasten Online, you can design your own cabinet for your washing machine and dryer, fully customized to your space and surprisingly affordable.
From a ceiling-high built-in cabinet for your washing machine and dryer to a low cabinet with a worktop, you can choose from dozens of smart configurations.
Whether you have one or four laundry appliances, you can easily put together your ideal washing machine cabinet with the online design tool.

3
A stylish laundry cabinet turns your utility room into a practical and beautiful space. At Maatkasten Online, you design your own custom cabinet for washing machine and dryer, fully tailored to your space and surprisingly affordable.
From a full-height built-in cabinet to a low cabinet with a worktop, you can choose from dozens of smart configurations.
Whether you have one or four appliances, the online design tool makes it easy to create your ideal washing machine cabinet.

[IsaiahPaget]

To me, these are all equally native sounding. There are no grammar or strange expression issues. I think I like 2 the most but its all really close.

[AIC-BV]

That would be Deepl
I have done the same test with 2 collegues so I'll get back on this

[AIC-BV]

@IsaiahPaget @LukeTowers Are you certain that Google Translate / Deepl can handle this kind of data?

size, type, color, cta_type, cta_style, _group, ... should not be translated

With ChatGPT it would be setting the right global prompt somewhere in the backend to ignore these fields by default

[AIC-BV]

@IsaiahPaget @LukeTowers Are you certain that Google Translate / Deepl can handle this kind of data?

If not a problem you can go ahead with the Deepl integration

[IsaiahPaget]

@AIC-BV Sorry for the late reply!

While developing the copying feature from before. I was only using the basic form widgets, which had only text inside a JSON object. But in your screenshot, it looks like there is a lot more info. Are you able to provide more detail on where that data in the screenshot is coming from? Is it the request that the MLBlocks.php gets when you use the current copy feature https://github.com/wintercms/wn-translate-plugin/blob/main/formwidgets/MLBlocks.php?

But if so, yeah I think we could add some 'ignore' in the translator provider configuration.

cc: @LukeTowers

[AIC-BV]

in your screenshot, it looks like there is a lot more info. Are you able to provide more detail on where that data in the screenshot is coming from?

It is a grouped repeater with nested forms

[AIC-BV]

@IsaiahPaget for example

blocks:
            tab: aic.bloginspiratie::lang.form.blocks
            prompt: aic.bloginspiratie::lang.form.blocks_add
            type: repeater
            cssClass: secondary-tab
            stretch: true
            style: collapsed
            groups: ~/plugins/aic/extendwinterpages/meta/blocks.yaml

block_text:
    name: Tekst
    icon: icon-bars
    fields:
        section:
            label: Tekst
            type: section
        type:
            label: Type
            type: balloon-selector
            default: default
            options:
                default: Standaard
                intro: Inleiding
            span: storm
            cssClass: col-xs-4
        color:
            label: Achtergrondkleur
            type: balloon-selector
            default: 0
            options:
                0: Geen
                1: Grijs
            span: storm
            cssClass: col-xs-4
        margin:
            label: Marge bovenaan
            type: balloon-selector
            default: 1
            options:
                0: Nee
                1: Ja
            span: storm
            cssClass: col-xs-4
            trigger:
                action: hide
                field: color
                condition: value[0]
        padding:
            label: Kleinere afstand bovenaan
            type: balloon-selector
            default: 0
            options:
                0: Nee
                1: Ja
            span: storm
            cssClass: col-xs-4
        center:
            label: Centreer tekst
            type: balloon-selector
            default: 0
            options:
                0: Nee
                1: Ja
            span: storm
            cssClass: col-xs-4
        data:
            type: nestedform
            usePanelStyles: false
            cssClass: custom-nested-form
            form:
                tabs:
                    icons:
                        Tekst: icon-keyboard
                        Link: icon-link
                    fields:
                        block_title:
                            tab: Tekst
                            label: Titel
                            type: text
                            span: auto
                        subtitle:
                            tab: Tekst
                            label: Subtitel
                            type: text
                            span: auto
                        text:
                            tab: Tekst
                            label: Tekst
                            type: richeditor
                            size: large
                        cta_is_active:
                            tab: Link
                            label: Link weergeven
                            type: balloon-selector
                            default: 0
                            options:
                                0: Nee
                                1: Ja
                        cta_repeater:
                            tab: Link
                            label: Links
                            type: repeater
                            style: collapsed
                            form:
                                fields:
                                    cta_style:
                                        tab: Link
                                        label: Link stijl
                                        type: balloon-selector
                                        default: 1
                                        options:
                                            0: Link
                                            1: Button
                                        span: auto
                                    cta_type:
                                        tab: Link
                                        label: Type
                                        type: balloon-selector
                                        default: intern
                                        options:
                                            intern: Intern
                                            extern: Extern
                                            mail: Mail
                                            download: Download
                                        span: auto
                                    cta_title:
                                        tab: Link
                                        label: Titel
                                        type: text
                                        span: auto
                                    cta_internal:
                                        tab: Link
                                        label: Link
                                        type: dropdown
                                        emptyOption: -- selecteer de pagina --
                                        options: \Aic\Globals\Helpers\FormHelper::getCtaInternalOptions
                                        trigger:
                                            action: show
                                            field: cta_type
                                            condition: value[intern]
                                        span: right
                                    cta_external:
                                        tab: Link
                                        label: Link
                                        type: text
                                        trigger:
                                            action: hide
                                            field: cta_type
                                            condition: value[intern]
                                        span: right

I think that in this case the following fields should be translated

• block_title
• subtitle
• text
• cta_repeater[cta_title]
• cta_repeater[cta_internal]
• cta_repeater[cta_external]

The other properties are CSS settings / booleans

[IsaiahPaget]

After discussing with Luke, we think it would make sense to have some sort of "translatable" whitelist for fields example: "json.*.block_title"

I think we would specify this in the config

[IsaiahPaget]

@AIC-BV

So after reviewing everything I think we have a good solution.

• The UI will look like the mockups shared here: Machine translation when copying from one language to another #102 (comment)
• The system will support translating the entire form widget and translation will be done per widget.
• A configuration file (e.g., translate-providers.php) will be created, or maybe use the existing config.php.
  • This file defines the API credentials for each translation provider.
  • It can also include a whitelist of translatable keys. This is especially important when a form widget contains nested widgets (such as repeaters or block components), so that only the intended fields are passed to the translation service.

From a user perspective, one will have to first get the API credentials to fill in the config and update the json white list for the keys in something like the blocks widget, and you should be able to use the key to auto translate the widgets that are translatable.

Hopefully that all makes sense, and please do let me know if I am missing any more requirements.