GitHub - 2noise/ChatTTS: A generative speech model for daily dialogue.

A generative speech model for daily dialogue. Contribute to 2noise/ChatTTS development by creating an account on GitHub.

Visit Website
GitHub - 2noise/ChatTTS: A generative speech model for daily dialogue.

Introduction

What is ChatTTS?

ChatTTS is a state-of-the-art text-to-speech model specifically designed for dialogue-based applications, making it ideal for interactive scenarios like conversational agents or virtual assistants. It supports multiple languages, including English and Chinese, with plans for further expansion. The model is optimized to deliver natural and expressive speech synthesis, ensuring a more engaging user experience.

Features of ChatTTS

  • Multi-Language Support: Currently supports English and Chinese, with additional languages planned for future releases.

  • Conversational Optimization: Tailored for dialogue-based tasks, enhancing the natural flow of interactions.

  • Fine-Grained Prosody Control: Users can control aspects like laughter, pauses, and interjections, enabling more expressive speech output.

  • Multiple Speakers: Allows for differentiation between various speakers, adding depth to conversations.

  • High-Quality Audio: The model surpasses many open-source TTS models in terms of prosody, delivering clearer and more natural speech.

How to Use ChatTTS

  1. Installation: Install the necessary packages using pip or conda, depending on your environment.

    • pip install --upgrade -r requirements.txt for direct installation.
    • Use conda for a more controlled setup: conda create -n chattts python=3.11 followed by conda activate chattts.

  2. Basic Usage: Import the library and start generating speech.

    import ChatTTS
import torch
import torchaudio

chat = ChatTTS.Chat()
chat.load(compile=False)

texts = ["Your text here"]
wavs = chat.infer(texts)

for i in range(len(wavs)):
    torchaudio.save(f"output{i}.wav", torch.from_numpy(wavs[i]), 24000)

  3. Advanced Usage: Customize your output with specific parameters and controls.

    # Sample a random speaker
rand_spk = chat.sample_random_speaker()

# Custom inference parameters
params_infer_code = ChatTTS.Chat.InferCodeParams(
    spk_emb=rand_spk,
    temperature=0.3,
    top_P=0.7,
    top_K=20,
)

# Text refinement parameters
params_refine_text = ChatTTS.Chat.RefineTextParams(
    prompt='[oral_2][laugh_0][break_6]',
)

wavs = chat.infer(
    texts,
    params_refine_text=params_refine_text,
    params_infer_code=params_infer_code,
)

Pricing

ChatTTS is open-source and free for academic and research purposes. However, it is licensed under AGPLv3+ for the code and CC BY-NC 4.0 for the model, restricting commercial use without permission.

Helpful Tips

  • Installation: Ensure all dependencies are correctly installed. Consider using a virtual environment to manage packages effectively.

  • Usage Limits: Be mindful of the model's intended use for academic purposes and adhere to licensing terms.

  • Performance: For better performance, set compile=True in the load method.

Frequently Asked Questions

  1. VRAM Requirements and Speed: A minimum of 4GB GPU memory is required. The RTX 4090 can generate about 7 semantic tokens per second with an RTF of 0.3.

  2. Model Stability: Challenges include multi-speaker support or audio quality. Experimenting with multiple samples may improve results.

  3. Emotion Control: Currently supports [laugh], [uv_break], and [lbreak]. Future updates may include more emotional controls.

  4. Ethical Use: The model includes safeguards like high-frequency noise to prevent misuse. Use responsibly and ethically.

By following these guidelines and exploring the features, you can effectively utilize ChatTTS for your text-to-speech needs, enhancing your applications with natural and expressive dialogue capabilities.