Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics
Under Review ACL 2023 ARR October Submission
Abstract
Song translation requires both translation of lyrics and alignment of music notes so that the resulting verse can be sung to the accompanying melody, which is a challenging problem that has attracted some interests in different aspects of the translation process. In this paper, we propose Lyrics-Melody Translation with Adaptive Grouping (LTAG), a holistic solution to automatic song translation by jointly modeling lyrics translation and lyrics-melody alignment. It is a novel encoder-decoder framework that can simultaneously translate the source lyrics and determine the number of aligned notes at each decoding step through an adaptive note grouping module. To address data scarcity, we commissioned a small amount of training data annotated specifically for this task and used large amounts of augmented data through back-translation. Experiments conducted on an English-Chinese song translation data set show the effectiveness of our model in both automatic and human evaluations.
Translated Scores and Sythesized Singing Audio Samples
En -> Zh
- “Is love I can be sure of” from “Will You Still Love Me Tomorrow”
| Source and Reference | GagaST | LTAG-cls | LTAG | |
|---|---|---|---|---|
| lyrics text | en:is love I can be sure of zh:是否能给我安全感 shì fǒu néng gěi wǒ ān quán gǎn |
this is love I can be sure of love 这 就是 爱， 我 可以 肯定的 爱吗 zhè jiù shì ài，wǒ kě yǐ kěn dìng de ài ma |
is love I can be sure of 是 爱 我 能 确定 shì ài wǒ néng kěn dìng |
is love I can be sure of 是 爱 我 可以 肯定 shì ài wǒ kě yǐ kěn dìng |
| scores |  |
 |
 |
 |
| wav |
- “I need you here I need you here to wipe away my tears” from “Run To You”
| Source and Reference | GagaST | LTAG-cls | LTAG | |
|---|---|---|---|---|
| lyrics text | en:I need you here I need you here to wipe away my tears zh:我需要你 我需要你帮我抹去泪水 wǒ xū yào nǐ wǒ xū yào nǐ bāng wǒ mǒ qù lèi shuǐ |
I need you I need you here to wipe tears 我 需要 你 我 需要 你 在这里 来 擦去 眼泪 wǒ xū yào nǐ wǒ xū yào nǐ zài zhè lǐ cā qù yǎn lèi |
I need you I need you wipe tears 我 需要 你 我 需要 你 擦干 眼泪 wǒ xū yào nǐ wǒ xū yào nǐ cā gān yǎn lèi |
I need you I need you wipe tears 我 需要 你 我 需要 你 擦干 眼泪 wǒ xū yào nǐ wǒ xū yào nǐ cā gān lèi shǔi |
| scores |  |
 |
 |
 |
| wav |
- “Aint nothing but a heartache Tell Me Why Aint nothing but a mistake” from “I Want It That Way”
| Source and Reference | GagaST | LTAG-cls | LTAG | |
|---|---|---|---|---|
| lyrics text | en:Aint nothing but a heartache Tell Me Why Aint nothing but a mistake zh:为什么让我心痛 告诉我 为什么你不行动 wèi shén mè ràng wǒ xīn tòng gào sù wǒ wèi shén mè nǐ bù xíng dòng |
aint nothing a heartache but tell me aint nothing a mistake 不过是 心痛 而已 告诉 我 只不过是 个 错误 bú guò shì xīn tòng ér yǐ gào sù wǒ zhǐ bú guò shì gè cuò wu |
aint nothing a heartache tell me aint nothing but a mistake 不过是 心痛 告诉 我 不过是 个 错 bú guò shì xīn tòng gào sù wǒ bú guò shì gè cuò |
aint nothing but a heartache tell me aint nothing but a mistake 没什么 只是 心痛 告诉 我 没什么 只是 个 错 méi shén mè zhǐ shì xīn tòng gào sù wǒ méi shén mè zhǐ shì gè cuò |
| scores |  |
 |
 |
 |
| wav |
- “And I’m thinking about how people fall in love in mysterious ways Maybe just the touch of a hand” from “Thinking Out Load”
| Source and Reference | GagaST | LTAG-cls | LTAG | |
|---|---|---|---|---|
| lyrics text | en:And I'm thinking about how people fall in love in mysterious ways Maybe just the touch of a hand zh:我一直在想人们如何神秘地坠入爱河 或许只是轻碰了手 wǒ yì zhí zài xiǎng rén men rú hé shén mì de zhuì rù ài hé huò xǔ zhǐ shì qīng pèng le shǒu |
And I'm thinking about people how in mysterious ways fall in love Maybe just a hand 而且 我 在想着 人们 是如何 以神秘的方式 坠入 爱河 也许 仅仅是 一只手 ér qǐe wǒ zài xiǎng zhe rén men shì rú hé yǐ shén mì de fāng shì zhuì rù ài hé yě xǔ jǐn jǐn shì yì zhī shǒu |
I'm thinking about people how in mysterious ways fall in Maybe just the touch of a hand 我 在想 人们 是如何 神秘地 坠落 也许 只是 一触即发 wǒ zài xiǎng rén men shì rú hé shén mì de zhuì luò yě xǔ zhǐ shì yì chù jí fā |
I'm thinking about people how in mysterious ways fall in love Maybe just the touch of a hand 我 在想 人们 如何 神秘地 坠入 爱河 也许 只是 一触即发 wǒ zài xiǎng rén men rú hé shén mì de zhuì rù ài hé yě xǔ zhǐ shì yì chù jí fā |
| scores |  |
 |
 |
 |
| wav |
- “Kiss me under the light of a thousand stars” from “Thinking Out Load”
| Source and Reference | GagaST | LTAG-cls | LTAG | |
|---|---|---|---|---|
| lyrics text | en:Kiss me under the light of a thousand stars zh:在 星空下 吻 我 吧 |
under a thousand stars the light of kiss me 在 成千上万颗 星星的 光下 吻 我 zài chéngß qiān shàng wàn kē xīng xing de guāng xià wěn wǒ |
a thousand stars under the light of kiss me 千星 之光下 吻 我 qiān xīng zhī guāng xià wěn wǒ |
under a thousand stars the light of kiss me 在 千星 之光下 吻 我 zài qiān xīng zhī guāng xià wěn wǒ |
| scores |  |
 |
 |
 |
| wav |
- “So full of the superficial Some people want it all But I don’t want nothing at all” from “If I Ain’t Got You”
| Source and Reference | GagaST | LTAG-cls | LTAG | |
|---|---|---|---|---|
| lyrics text | en:So full of the superficial Some people want it all But I don't want nothing at all zh:生活实在肤浅至极 有人想拥有一切 而我什么都不缺 shēng huó shí zài fū qiǎn zhì jí yǒu rén xiǎng yōng yǒu yí qiè ér wǒ shén me dōu bù quē |
So full of superficial Some people want it all But I nothing at all want 如此 充满了 肤浅的 有些 人 想要 一切 但 我 什么也不 想要 rú cǐ chōng mǎn le fū qiǎn de yǒu xiē rén xiǎng yào yí qiè dān wǒ shén me yě bù xiǎng yào |
So full of superficial people all want it But nothing at all 如此的 肤浅 有人 都 想要 却 一无所有 rú cǐ de fū qiǎn yǒu rén dōu xiǎng yào què yī wú suǒ yǒu |
So full of superficial Some people all want it But nothing at all want 如此的 肤浅 有些 人 都 想要 却 什么都不 要 rú cǐ de fū qiǎn yǒu xiē rén dōu xiǎng yào què shén me dōu bú yào |
| scores |  |
 |
 |
 |
| wav |
- “When the rain is blowing in your face And the whole world is on your case” from “Make you feel my love”
| Source and Reference | GagaST | LTAG-cls | LTAG | |
|---|---|---|---|---|
| lyrics text | en:When the rain is blowing in your face And the whole world is on your case zh:当雨吹打在你的脸上 全世界压在你行李箱 |
when the rain in your face is blowing the whole world is on your case 当 雨水 在 你的 脸上 吹来时 整个 世界 都 在 你的 案子上 dāng yǔ shuǐ zài nǐ de liǎn shàng chuī lái shí zhěng gè shì jiè dōu zài nǐ de àn zi shàng |
when the rain is blowing your face the whole world is on your case 当 雨 打在 你 脸上 全 世界 都在 你 左右 dāng yǔ dǎ zài nǐ liǎn shàng quán shì jiè dōu zài nǐ zuǒ yòu |
when the rain is blowing your face the whole world is on your case 当 雨 打在 你的 脸上 整个 世界 都在 你的 左右 dāng yǔ dǎ zài nǐ de liǎn shàng quán shì jiè dōu zài nǐ de zuǒ yòu |
| scores |  |
 |
 |
 |
| wav |
Zh -> En
- “她会有多幸运” from “小幸运”
| Source and Reference | GagaST | LTAG-cls | LTAG | |
|---|---|---|---|---|
| lyrics text | zh:她会有多幸运 tā huì yǒu duō xìng yùn en:How lucky she will be |
How lucky she would be to be | How lucky would she be | How lucky would she be |
| scores |  |
 |
 |
 |
- “也许当时忙着微笑和哭泣 忙着追逐天空中的流星” from “小幸运”
| Source and Reference | GagaST | LTAG-cls | LTAG | |
|---|---|---|---|---|
| lyrics text | zh:也许当时忙着微笑和哭泣 忙着追逐天空中的流星 yě xǔ dāng shí máng zhe wēi xiào hé kū qì máng zhe zhuī zhú tiān kōng zhōng de liú xīng en:Perhaps I've been busy smiling and shedding tears Or can't help chasing in the sky pretty meteors |
Perhaps I was busy smiling and crying at the time of the meeting I am busy chasing after the shooting stars that are in the sky | Maybe I was busy smiling and crying busy chasing the shooting in the sky | Maybe I was busy with a smile and a cry busy chasing the stars in the sky |
| scores |  |
 |
 |
 |
- “你张开了双翼 遇见你的注定” from “小幸运”
| Source and Reference | GagaST | LTAG-cls | LTAG | |
|---|---|---|---|---|
| lyrics text | zh:你张开了双翼 遇见你的注定 nǐ zhāng kāi le shuāng yì yù jiàn nǐ de zhù dìng en:You turn over old memory Come across your destiny |
You open up your wings for me It ’s the fate to meet you when I meet you | you spread your wings to meet you is meant to | You have spread your wings To meet you is destined |
| scores |  |
 |
 |
 |
- “好不好有亏欠我们都别再追究” from “说散就散”
| Source and Reference | GagaST | LTAG-cls | LTAG | |
|---|---|---|---|---|
| lyrics text | zh:好不好有亏欠我们都别再追究 hǎo bù hǎo yǒu kuī qiàn wǒ mén dōu bié zài zhuī jīu en:Good or bad no matter what we just let go |
Is it good to have a debt to pay ? We don't have to pursue it. | What good is there a debt we don't have to follow | We owe each other good or bad don't hold it up |
| scores |  |
 |
 |
 |