EATS-Speech: Emotion-Adaptive Transformation and Priority Synthesis for Zero-Shot Text-to-Speech

Abstract

Zero-shot text-to-speech (TTS) supports diverse speech synthesis without speaker-specific data but struggles to accurately transfer emotions from reference to target text. Traditional approaches treat emotion as part of a global style, leading to inconsistent emotional expressiveness. To address this, we propose EATS-Speech, an Emotion-Adaptive Transformation Synthesis framework. EATS-Speech employs Emotion Priority Synthesis through a parallel pipeline that decomposes speech into non-emotion style, emotion, and content. It prioritizes emotion generation to enhance expressiveness. Furthermore, it introduces Emotion-Adaptive Transformation Synthesis, where an LLM-based converter learns text-emotion mapping patterns from the reference speech and transfers them to the target text. Experiments on the LibriTTS dataset demonstrate the improvements in emotional expressiveness and accurate emotion adaptation.

Contents

• YourTTS: https://github.com/Edresson/YourTTS
• VALL-E: https://github.com/lifeiteng/VALL-E
• TransferTTS: https://github.com/hcy71o/TransferTTS
• E2-TTS: https://github.com/SWivid/F5-TTS
• CosyVoice: https://github.com/FunAudioLLM/CosyVoice

Sample for Seen Speaker - Development Set

Sample 1

Text: The play and slight agitation of the water, in its upward gush, wrought magically with these variegated pebbles, and made a continually shifting apparition of quaint figures, vanishing too suddenly to be definable.,

Sample 2

Text: "You'd better look in those big manufacturing houses along Franklin Street and just the other side of the river," he concluded. "Lots of girls work there.,

Sample 3

Text: We followed an Alpine path for some four miles, now hundreds of feet above a brawling stream which descended from the glaciers, and now nearly alongside it.,

Sample 4

Text: Tom Selwyn had been very sober during all this merry chatter; and now in his seat across the narrow aisle, he drummed his heels impatiently on the floor.,

Sample for Unseen Speaker - Test Set

Sample 1

Text: She never said much of herself in her letters, and Fanny's first exclamation when they met again, was an anxious "Why, Polly, dear! Have you been sick and never told me?",

Sample 2

Text: "We must hasten." Seizing the scissors that lay on the ground where Ethelried had dropped them, she opened and shut them several times, exclaiming:,

Sample 3

Text: She was sanguine, she was genial and companionable, and her spirits rose at the sight of a friendly face.,

Sample 4

Text: Next day he rose and prayed the dawn prayer and repaired to his namesake's house where, after the company was all assembled, the host began to relate,

Sample 5

Text: "Another case was that of a slave woman in a very delicate state, who was one day knocked down stairs by mrs Johnson herself, and in a few weeks after, the poor woman died from the effects of the injury thus received.,

Sample 6

Text: He'll look after this end of the Island hereafter, an' unless I'm much mistaken he'll do it a heap better than you did.",

Sample for Seen Speaker - Development Set

Sample 1

Text: He was discharged, whereupon the union notified us that unless the boy was taken back the whole body would go out.,

Sample 2

Text: "In the lonely barton by yonder coomb Our childhood used to know," I should go with him in the gloom, Hoping it might be so.,

Sample 3

Text: If I make the marriage she chooses, she thinks mr Binnie will leave me his money. I am to run after a man who does not care for me, and make myself attractive, in the hope that he will condescend to marry me because mr Binnie may leave me his money.,

Sample for Unseen Speaker - Test Set

Sample 1

Text: 'Yes, I will kiss you,' said the man's daughter, and she did it, but she thought it was the worst bit of work that she had ever had to do in her life.,

Sample 2

Text: "Well!" he exclaimed; "and I felt sorry for her as one might for one's sister at home, and hung back from getting her people into trouble.,