What Is Spark TTS and How Does AI Voice Cloning Work?

TL;DR

Spark TTS clones any voice from a few seconds of audio, as short as 2 to 4 seconds, and makes it speak new text in English or Chinese, including cross-lingual output where a Chinese sample speaks fluent English. The models are free, open source, released on Hugging Face with a GitHub repo and a Gradio interface for local use.

Transcript

AI never sleeps and this week has been absolutely insane we have a tiny open source model that even beats deep seek we have a new voice cloner which is really good I'm very confident in my singing skills but there aren't many pieces of music that are worthy of my vocal prowess we have a new AI that can compose sheet music for an entire Orchestra an... Read More

Key Insights

• Spark TTS is a free, open source text to speech generator that needs only a few seconds of reference audio to clone a voice and make it say arbitrary new text, with models published on Hugging Face and installation instructions in a GitHub repo.
• Voice cloning quality in Spark TTS holds up on samples as short as 4 seconds, shown with a Rick and Morty clip, and 2 seconds, shown with a Chinese speaker, so sample length is not the limiting factor it once was.
• Cross-lingual cloning works in both directions: a cloned English voice can speak Chinese, and a 10 second Chinese clip of Jack Ma produces an English sentence that still sounds like the same speaker.
• Naturalness in Spark TTS comes from prosody details, not just timbre. The output takes breaths between sentences, pauses between words, and places emphasis in the right places, which is what separates it from earlier text to speech tools.
• Image to video is the more controllable mode for AI video because supplying a photograph or AI generated image as the first frame constrains the generation, which is why Hunyuan's image to video release matters more than its earlier text to video model.
• VRAM requirements published by a model's authors are not always binding. Hunyuan I2V lists 80 GB recommended and 60 GB minimum, yet the ComfyUI Hunyuan Video Wrapper fork runs it at around 12 GB using a downloadable JSON workflow.
• NotaGen composes sheet music rather than audio, producing parts for individual instruments in a string quartet, orchestra, or choir, and it also specifies dynamics and articulation such as legato and staccato passages.
• NotaGen's training pipeline is pre-training on 1.6 million pieces of music to learn general musical patterns and structure, followed by fine-tuning on almost 9,000 high quality classical sheets from 152 composers to acquire style; the fine-tuning stage could target pop music instead.

Questions & Answers

Q: What is Spark TTS and what can it do?

Spark TTS is a text to speech generator that clones a voice from a short audio sample and then makes that voice say any text you supply. Demonstrated samples run from 14 seconds down to 2 seconds, and the cloned output matches the input voice closely in both timbre and tone. It handles English and Chinese, including cross-lingual generation where a voice sampled in one language speaks the other. The models are on Hugging Face and the GitHub repo includes instructions for running it locally.

Q: How much audio does Spark TTS need to clone a voice?

Only a few seconds. The demos use a 14 second nature documentary style clip, a 9 second cooking narration, a 4 second Rick and Morty clip, roughly 10 seconds of Jack Ma, and a Chinese sample that is only 2 seconds long. All of these produce output that sounds like the original speaker, so cloning quality does not appear to require long reference recordings.

Q: Can Spark TTS make a voice speak a language the speaker did not use in the sample?

Yes. A cloned English speaking voice was made to read Chinese while still sounding like the same person, and the reverse also worked: a roughly 10 second clip of Jack Ma speaking Chinese produced an English sentence that sounds like him speaking English. Another Chinese voice sampled for two seconds was made to read a full English passage about speech synthesis technology. Cross-lingual transfer preserves the speaker identity in both directions.

Q: How do I run Spark TTS on my own computer?

The models have already been released on Hugging Face, and there is a GitHub repo whose bottom section contains the full instructions for downloading and running it locally. The repo also ships a Gradio interface, so you can use it through a browser UI without writing code. The project is completely free and open source and available to use immediately.

Q: What is Hunyuan image to video and why does it matter?

Hunyuan is a free, open source, uncensored AI video model. It previously shipped a text to video model that generates clips from a text prompt, and has now released image to video. With image to video you supply a photograph or an AI generated image as the first frame of the clip, which gives far more control over what the generation produces. Sample outputs, including an anime character animation, stay consistent without the warping or deformation typical of older video models.

Q: How much VRAM do you need to run Hunyuan image to video?

The official GitHub repo recommends 80 GB of VRAM, or 60 GB as a bare minimum, which is beyond what most consumer hardware offers. That requirement is not actually binding. A separate fork called ComfyUI Hunyuan Video Wrapper lets you run Hunyuan inside ComfyUI with as little as roughly 12 GB of VRAM. Its example workflows folder now includes an image to video workflow: download the JSON file and drag it into the ComfyUI interface to load the whole setup.

Q: What is NotaGen and what kind of music can it compose?

NotaGen is an AI that composes original classical music as sheet music. It can write for a single instrument such as piano, for a string quartet of two violins, viola, and cello, for a full orchestra spanning woodwinds like flute, oboe, and clarinet, percussion including triangle, and strings from first violin down to bass, and for a choir with four vocal sections plus piano accompaniment. Each row of the generated score corresponds to one instrument or section.

Q: How was NotaGen trained?

NotaGen was pre-trained on 1.6 million pieces of music, which lets the model learn general patterns and structures of music. It was then fine-tuned on high quality classical sheet music data consisting of almost 9,000 classical scores covering 152 composers, which teaches it the specific styles and characteristics of classical music. The fine-tuning stage is swappable: the same approach could be applied to pop music instead of classical.

Q: Does NotaGen only generate notes, or does it handle musical expression too?

It handles expression as well. In the string quartet example the model determined which parts should be played stronger and which softer, and included articulation such as legato passages and staccato, meaning shorter, punchier notes. The demo playback uses virtual instruments, so choir and piano timbres do not sound fully realistic, but the underlying sheet music can be printed and performed by real musicians.

Summary & Key Takeaways

• Spark TTS is a text to speech generator that clones a voice from short samples, demonstrated with clips of 14 seconds, 9 seconds, 4 seconds, and even 2 seconds. Cloned output reproduces tone and expressiveness, takes breaths between sentences, pauses between words, and places emphasis in the right spots.

• Hunyuan released an image to video model, letting a photo or AI generated image act as the first frame for more control over generation. The official GitHub repo recommends 80 GB of VRAM or a minimum of 60 GB, but a ComfyUI Hunyuan Video Wrapper fork runs it with roughly 12 GB.

• NotaGen composes original classical sheet music for solo piano, string quartet, full orchestra, and choir, deciding dynamics, legato, and staccato phrasing. It was pre-trained on 1.6 million pieces of music and fine-tuned on almost 9,000 classical sheets covering 152 composers.