Abstract
Singing Voice Conversion (SVC) transfers a source singer’s timbre to a target while keeping melody and lyrics. The key challenge in any-to-any SVC is adapting unseen speaker timbres to source audio without quality degradation. Existing methods either face timbre leakage or fail to achieve satisfactory timbre similarity and quality in the generated audio. To address these challenges, we propose DAFMSVC, where the self-supervised learning (SSL) features from the source audio are replaced with the most similar SSL features from the target audio to prevent timbre leakage. It also incorporates a dual-cross-attention mechanism for the adaptive fusion of speaker embeddings, melody, and linguistic content. Additionally, we introduce a flow matching module for high-quality audio generation from the fused features. Experimental results show that DAFMSVC significantly enhances timbre similarity and naturalness, outperforming state-of-the-art methods in both subjective and objective evaluations.
Snowflake represents the parameter that remains unchanged when training the framework.
Conversion Tasks
Our goal is to transfer the timbre of a source song to an unseen target singer while preserving the original content and melody. We will generate waveforms with high timbre similarity, naturalness and quality. Note that the reference audio in the SVC demo is approximately 30 seconds long. In the demo web, we have only included about 10 seconds of the target person’s audio to demonstrate their voice characteristics.
- Proposed - the DAFMSVC method.
- NeuCoSVC - This method is a novel neural concatenation-based approach for one-shot SVC, which adopts the FastSVC architecture to generate synthesized audio.
- So-VITS-SVC - This method is a popular open-source voice conversion tool based on VITS, which uses a Conditional Variational Autoencoder combined with adversarial learning.
- DDSP-SVC - This method is an end-to-end singing voice conversion system based on Differentiable Digital Signal Processing (DDSP) that uses a cascade diffusion model to reconstruct high-quality audio.
Demos
| NO | Source | Reference | Proposed | NeuCoSVC | So-VITS-SVC | DDSP-SVC |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 | ||||||
| 4 | ||||||
| 5 | ||||||
| 6 | ||||||
| 7 |