From 042b9500a88a0f0663ff7ab06bc1d892a3ee32b4 Mon Sep 17 00:00:00 2001 From: Xiaoming Liu Date: Thu, 12 Sep 2024 14:16:10 +0800 Subject: [PATCH 1/4] fix typo --- litgpt/generate/base.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/litgpt/generate/base.py b/litgpt/generate/base.py index ad59048..2c5d23b 100644 --- a/litgpt/generate/base.py +++ b/litgpt/generate/base.py @@ -19,7 +19,7 @@ def multinomial_num_samples_1(probs: torch.Tensor) -> torch.Tensor: return torch.multinomial(probs, num_samples=1) -def sample_top_p(logits_A: torch.Tensor, top_p: float) -> torch.Tensor: +def sample_top_p(logits: torch.Tensor, top_p: float) -> torch.Tensor: sorted_logits, sorted_indices = torch.sort(logits, descending=False) cumulative_probs = sorted_logits.softmax(dim=-1).cumsum(dim=-1) # Example: From edca25a9bd80edf6553e87c1abd4e1b524f6b3e3 Mon Sep 17 00:00:00 2001 From: mini-omni <> Date: Fri, 13 Sep 2024 13:12:41 +0800 Subject: [PATCH 2/4] update readme --- README.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index aa74c47..f805e7f 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ python3 server.py --ip '0.0.0.0' --port 60808 - run streamlit demo -NOTE: you need to run streamlit locally with PyAudio installed. For error: `ModuleNotFoundError: No module named 'utils.vad'`, please run `export PYTHONPATH=./` first. +NOTE: you need to run streamlit **locally** with PyAudio installed. For error: `ModuleNotFoundError: No module named 'utils.vad'`, please run `export PYTHONPATH=./` first. ```sh pip install PyAudio==0.2.14 @@ -98,7 +98,12 @@ python inference.py - Error: `ModuleNotFoundError: No module named 'utils.xxxx'` - Answer: run `export PYTHONPATH=./` first. + Answer: run `export PYTHONPATH=./` first. No need to run `pip install utils`, or just try: `pip uninstall utils` + + +- Error: can run streamlit in local browser, with remote streamlit server, issue: https://github.com/gpt-omni/mini-omni/issues/37 + + Answer: you need start streamlit **locally** with PyAudio installed. ## Acknowledgements From a95235a6d671acab4681f965db5b9bf31776510a Mon Sep 17 00:00:00 2001 From: mini-omni <> Date: Sat, 14 Sep 2024 00:00:28 +0800 Subject: [PATCH 3/4] add datasets --- README.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index f805e7f..7529ead 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,8 @@ Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming

🤗 Hugging Face | 📖 Github -| 📑 Technical report +| 📑 Technical report | +🤗 Datasets

Mini-Omni is an open-source multimodal large language model that can **hear, talk while thinking**. Featuring real-time end-to-end speech input and **streaming audio output** conversational capabilities. @@ -18,6 +19,10 @@ Mini-Omni is an open-source multimodal large language model that can **hear, tal

+## Updates + +- **2024.09:** **VoiceAssistant-400K** is uploaded to [Hugging Face](https://huggingface.co/datasets/gpt-omni/VoiceAssistant-400K). + ## Features ✅ **Real-time speech-to-speech** conversational capabilities. No extra ASR or TTS models required. @@ -96,14 +101,22 @@ python inference.py ## Common issues +- Question: does the model support other languages? + + **Answer**: No, the model is only trained on English. However, as we use whisper as the audio encoder, the model can understand other languages which is supported by whisper (like chinese), but the output is only in English. + +- Question: what is `post_adapter` in the code? does the open-source version support tts-adapter? + + **Answer**: `post_adapter` is the `tts-adapter` in the model.py, but the open-source version does not support `tts-adapter`. + - Error: `ModuleNotFoundError: No module named 'utils.xxxx'` - Answer: run `export PYTHONPATH=./` first. No need to run `pip install utils`, or just try: `pip uninstall utils` + **Answer**: run `export PYTHONPATH=./` first. No need to run `pip install utils`, or just try: `pip uninstall utils` - -- Error: can run streamlit in local browser, with remote streamlit server, issue: https://github.com/gpt-omni/mini-omni/issues/37 +- Error: can not run streamlit in local browser, with remote streamlit server, issue: https://github.com/gpt-omni/mini-omni/issues/37 - Answer: you need start streamlit **locally** with PyAudio installed. + **Answer**: you need start streamlit **locally** with PyAudio installed. + ## Acknowledgements From aafb97f765db805933c22668db9918c4a0bdef60 Mon Sep 17 00:00:00 2001 From: mini-omni <> Date: Sat, 14 Sep 2024 11:27:03 +0800 Subject: [PATCH 4/4] update readme --- README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 7529ead..d171e6a 100644 --- a/README.md +++ b/README.md @@ -99,23 +99,23 @@ cd mini-omni python inference.py ``` -## Common issues +## FAQ -- Question: does the model support other languages? +**1. Does the model support other languages?** - **Answer**: No, the model is only trained on English. However, as we use whisper as the audio encoder, the model can understand other languages which is supported by whisper (like chinese), but the output is only in English. +No, the model is only trained on English. However, as we use whisper as the audio encoder, the model can understand other languages which is supported by whisper (like chinese), but the output is only in English. -- Question: what is `post_adapter` in the code? does the open-source version support tts-adapter? +**2. What is `post_adapter` in the code? does the open-source version support tts-adapter?** - **Answer**: `post_adapter` is the `tts-adapter` in the model.py, but the open-source version does not support `tts-adapter`. +The `post_adapter` is `tts-adapter` in the model.py, but the open-source version does not support `tts-adapter`. -- Error: `ModuleNotFoundError: No module named 'utils.xxxx'` +**3. Error: `ModuleNotFoundError: No module named 'utils.xxxx'`** - **Answer**: run `export PYTHONPATH=./` first. No need to run `pip install utils`, or just try: `pip uninstall utils` +Run `export PYTHONPATH=./` first. No need to run `pip install utils`, or just try: `pip uninstall utils` -- Error: can not run streamlit in local browser, with remote streamlit server, issue: https://github.com/gpt-omni/mini-omni/issues/37 +**4. Error: can not run streamlit in local browser, with remote streamlit server**, issue: https://github.com/gpt-omni/mini-omni/issues/37 - **Answer**: you need start streamlit **locally** with PyAudio installed. +You need start streamlit **locally** with PyAudio installed. ## Acknowledgements