The best performance is obtained if the top 50% monolingual data is used for self-learning. Obviously, most of the self-learning methods outperform the NMT baseline. Lines 3–6 shows the BLEU scores when applying self-learning algorithm based on the source-side monolingual data. FOR SMALL DATAĪs shown above, Moses is the state-of-the-art SMT translation model, and RNNSearch is the NMT baseline model. Chinese sentences are segmented by using the Stanford Word Segmenter tool and preordered by using the syntax based reordering method (Berkeley parser). NIST 2003 (MT03) is used for dev set, and NIST 2004 (MT04), NIST 2005 (MT05), NIST 2006 (MT06) are used for test set. The author used a small dataset (including 0.63M sentence pairs and 6.5M monolingual data) and a large dataset (including 2.1M sentences and 12M monolingual data). The self-learning and multi-tasking learning methods are used for Chinese-to-English translation. Continue iteration until convergence (iteration number or no parameter change).The new learnt encoder network is used to initialize the encoder model for reordering task.Learn the encode-decoder parameters in the translation task for 4 epochs.The learnt encoder network parameters are employed to initialize the encoder model for translation task.Optimize the encoder-decoder parameters in the reordering task for 1 epoch.During the optimization, the iterative approach is computed as following: Which is a summation of log probabilities of machine translation and sentence reordering. Moreover, the source-side monolingual data should share the same source side language vocabulary as bilingual corpus, so that no new words can be generated. So in practice, the author froze the parameters of the decoder for synthetic data, and only the encoder parameters will be updated during synthetic data training. Another reason is that the syntax/semantic correctness for huge amounts of synthetic data cannot be guaranteed. One reason for this is that the addition of more unrelated monolingual data may lead to decreased translation quality. Note that the synthetic data may negatively influence the decoder performance. Combine the un-labelled data together to make new labelled data.Adopt the baseline to predict the un-labelled data.Hence, the overall goal is to generate more bilingual data using the source-side monolingual data. Given the sentence aligned source corpus and target corpus, there exists the source-side large scale monolingual data, which is related to the bitext.

In this paper, the author called it “ synthetic parallel data” by employing the self learning method. The main idea is to use a pre-trained network to predict the target sentences given the source-side monolingual data, and then combine them with the bilingual data to make a bigger bilingual corpus for training. The first method proposed in this paper is the self-learning algorithm. The figure above is an illustration for standard encoder-decoder based NMT with attention, where X=(x1,x2.x_ and parameters theta, the goal is to minimize the conditional log likelihood of the sentence aligned bilingual data.

Opus bitext and monolingual data