UniqAI/mini-omni
1
0
Fork 0
mirror of https://github.com/gpt-omni/mini-omni synced 2024-11-16 05:03:47 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

add datasets

Browse Source
This commit is contained in:
mini-omni 2024-09-14 00:00:28 +08:00
parent edca25a9bd
commit a95235a6d6
1 changed files with 18 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
README.md
View File

@ -8,7 +8,8 @@ Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
<p align="center"> <p align="center">
🤗 <a href="https://huggingface.co/gpt-omni/mini-omni">Hugging Face</a> | 📖 <a href="https://github.com/gpt-omni/mini-omni">Github</a> 🤗 <a href="https://huggingface.co/gpt-omni/mini-omni">Hugging Face</a> | 📖 <a href="https://github.com/gpt-omni/mini-omni">Github</a>
| 📑 <a href="https://arxiv.org/abs/2408.16725">Technical report</a> | 📑 <a href="https://arxiv.org/abs/2408.16725">Technical report</a> |
🤗 <a href="https://huggingface.co/datasets/gpt-omni/VoiceAssistant-400K">Datasets</a>
</p> </p>
Mini-Omni is an open-source multimodal large language model that can **hear, talk while thinking**. Featuring real-time end-to-end speech input and **streaming audio output** conversational capabilities. Mini-Omni is an open-source multimodal large language model that can **hear, talk while thinking**. Featuring real-time end-to-end speech input and **streaming audio output** conversational capabilities.
@ -18,6 +19,10 @@ Mini-Omni is an open-source multimodal large language model that can **hear, tal
</p> </p>
## Updates
- **2024.09:** **VoiceAssistant-400K** is uploaded to [Hugging Face](https://huggingface.co/datasets/gpt-omni/VoiceAssistant-400K).
## Features ## Features
✅ **Real-time speech-to-speech** conversational capabilities. No extra ASR or TTS models required. ✅ **Real-time speech-to-speech** conversational capabilities. No extra ASR or TTS models required.
@ -96,14 +101,22 @@ python inference.py
## Common issues ## Common issues
- Question: does the model support other languages?
**Answer**: No, the model is only trained on English. However, as we use whisper as the audio encoder, the model can understand other languages which is supported by whisper (like chinese), but the output is only in English.
- Question: what is `post_adapter` in the code? does the open-source version support tts-adapter?
**Answer**: `post_adapter` is the `tts-adapter` in the model.py, but the open-source version does not support `tts-adapter`.
- Error: `ModuleNotFoundError: No module named 'utils.xxxx'` - Error: `ModuleNotFoundError: No module named 'utils.xxxx'`
Answer: run `export PYTHONPATH=./` first. No need to run `pip install utils`, or just try: `pip uninstall utils` **Answer**: run `export PYTHONPATH=./` first. No need to run `pip install utils`, or just try: `pip uninstall utils`
- Error: can not run streamlit in local browser, with remote streamlit server, issue: https://github.com/gpt-omni/mini-omni/issues/37
- Error: can run streamlit in local browser, with remote streamlit server, issue: https://github.com/gpt-omni/mini-omni/issues/37 **Answer**: you need start streamlit **locally** with PyAudio installed.
Answer: you need start streamlit **locally** with PyAudio installed.
## Acknowledgements ## Acknowledgements