How to generate AI music with YuE2 in ComfyUI

TL;DR

YuE2 turns text prompts into editable sheet music and full songs, then can use reference audio to make cover versions. It supports multiple styles, languages, and iterative editing via an AI agent workflow, and can run with limited VRAM. The video also covers installation steps and benchmarks.

Transcript

This is currently the best open- source music generator you can use. You can run this with 4 GB of VRAM or less. Plus, you can even do cover songs and edit existing songs. And according to some benchmarks, this even beats Sunno V6, which is pretty crazy. So, this is called UA 2. And first of all, here are some demos. So, on the left is the prompt d... Read More

Key Insights

• YuE2 outputs an editable music score first, including vocal and instrumental notation, which then guides full song generation.
• The tool can use a reference audio to create covers, maintaining melody while changing style or arrangement.
• YuE2 supports multiple languages and genre styles such as jazz, boogie woogie, flamenco, folktronica, emo rock, and more.
• The workflow can be extended by linking YuE2 to AI agents like GBT Astra or GLM to automate editing or production tasks.
• Benchmarks indicate YuE2 outperforms other open source models and some closed models in song quality.
• The installation flow for YuE2 on ComfyUI requires updating ComfyUI, loading a pre-built UA2 workflow, and downloading specific model files.
• The system provides options for different VRAM budgets by offering smaller quantized model variants for low-end GPUs.
• The video emphasizes the value of using reference melodies by referencing classical pieces to demonstrate transformation into modern styles.

Questions & Answers

Q: How does YuE2 create a full song from a text prompt?

YuE2 starts by turning the text prompt into an editable score that includes both vocal and instrumental melodies, then uses this score as the backbone to generate the full audio track. This approach provides a clear musical framework, enabling precise edits and style changes without starting from scratch each time. The workflow supports verse and chorus tagging to control structure.

Q: Can YuE2 generate covers from existing songs?

Yes, YuE2 can generate cover songs by using a reference audio input. The system outputs an editable score first and then produces the final track, allowing the melody to be preserved while the arrangement, tempo, and instrumentation are altered. This makes it suitable for creating stylistically different interpretations of existing tunes.

Q: What genres or styles can YuE2 produce?

The video demonstrates several styles including smoky jazz, boogie woogie, flamenco, folktronica, and Korean emo rock, showing YuE2’s versatility across traditional and contemporary genres. The style prompt can specify tempo, instruments, and pacing to shape the final composition accordingly.

Q: How does YuE2 handle multilingual lyrics and singing?

YuE2 supports different languages as shown by examples in Spanish, Russian, and Korean. The system accepts lyrics in the text prompt, aligns them with the generated vocal melody, and can adapt phonetics and intonation to different language sounds, enabling multilingual music production.

Q: What is the role of the Comfy UI in running YuE2?

Comfy UI serves as the local environment to run YuE2 workflows. Users update Comfy UI, load the UA2 workflow, and integrate required models and encoders. This setup enables a graphical interface to manage text-to-music generation, reference audio, and iterative editing without extensive coding.

Q: What hardware considerations are mentioned for YuE2?

The video notes that YuE2 can run with 4 GB of VRAM or less for some configurations, and provides options to download different model variants to fit hardware constraints. Users can choose full 7.8 GB BF-16 or a smaller 3.96 GB quantized version based on available VRAM.

Q: How does the agent iterative editing feature work?

The agent iterative editing feature enables you to refine a song by instructing an AI agent to adjust aspects like reharmonization, removing or adding instruments, or tweaking the melody while keeping the core structure intact. This creates a loop of prompts and edits that progressively improves the arrangement.

Q: Where can viewers find the YuE2 resources and tutorials mentioned?

The video provides links to the YuE2 official repository, a Comfy workflow file, and YuE2 models hosted on Hugging Face, along with a ComfyUI tutorial. Viewers can follow these resources to install, configure, and run YuE2 locally and explore additional examples like text-to-music workflows.

Summary & Key Takeaways

• YuE2 converts prompts into a musical score and then into a song, enabling flexible editing and cover generation. It uses a two-part workflow, with text-to-music followed by optional reference audio processing. The system supports various genres and languages and can be integrated with agent-based editing tools.

• Benchmarks show YuE2 beating several open source models and even some closed models in song quality indices, while remaining accessible with modest hardware.

• The video explains installation in ComfyUI, including updating the UI, loading a pre-built workflow, and downloading necessary model files and encoders to run YuE2 smoothly.