Tim Jaeryang Baek
91e758f3ec
Merge pull request #13165 from feddersen-group/perf/parallel_knowledge_search
...
perf: all knowledge searches in parallel in non-hybrid mode
2025-04-23 10:01:06 -07:00
Timothy Jaeryang Baek
09874ab83d
fix: FireCrawlLoader
2025-04-24 01:40:34 +09:00
Alexander Grimm
d182155fac
~ call knowledge searches in parallel in non-hybrid mode
2025-04-23 09:20:51 +00:00
Tim Jaeryang Baek
faa3cac0e4
Merge pull request #13107 from tth37/fix_tavily_max_results
...
fix: `max_results` in Tavily search handler
2025-04-22 23:47:36 -07:00
tth37
bc315bd530
fix: max_results in Tavily search api
2025-04-21 20:59:47 +08:00
Athanasios Oikonomou
1e291aff25
feat: Add abstract base class for vector database integration
...
- Created `VectorDBBase` as an abstract base class to standardize vector database operations.
- Added required methods for common vector database operations: `has_collection`, `delete_collection`, `insert`, `upsert`, `search`, `query`, `get`, `delete`, `reset`.
- The base class can now be extended by any vector database implementation (e.g., Qdrant, Pinecone) to ensure a consistent API across different database systems.
2025-04-21 08:27:27 +03:00
Athanasios Oikonomou
e000c56ef7
feat(vector-db): add support for Pinecone client
...
Adds Pinecone as a supported vector database option.
- Implements `PineconeClient` with support for common operations: `add`, `query`, `delete`, `reset`.
- Emulates namespace support using metadata filtering (`collection_name` prefix).
- Dynamically configures Pinecone via the following env vars:
- `PINECONE_API_KEY`
- `PINECONE_ENVIRONMENT`
- `PINECONE_INDEX_NAME`
- `PINECONE_DIMENSION`
- `PINECONE_METRIC`
- `PINECONE_CLOUD`
- Integrates cleanly with the vector DB abstraction layer.
- Includes markdown documentation under `docs/getting-started/env-configuration.md`.
BREAKING CHANGE: None
2025-04-20 11:08:51 +03:00
Tim Jaeryang Baek
87844a8042
Merge pull request #12822 from tth37/feat_external_search_loader
...
feat: Support for Self-Hosted/External Web Search/Loader Engines
2025-04-18 23:51:27 -07:00
Juan Calderon-Perez
6188c0c5b7
Add suport for Qdrant GRPC
2025-04-17 01:13:49 -04:00
Juan Calderon-Perez
b4d0d840d1
Fix formatting of qdrant.py
2025-04-15 08:56:51 -04:00
Athanasios Oikonomou
575c12f80c
feat: add QDRANT_ON_DISK configuration option for Qdrant integration
...
This commit will allow configuring the on_disk client parameter, to reduce the memory usage.
https://qdrant.tech/documentation/concepts/storage/?q=mmap#configuring-memmap-storage
Default is false, keeping vectors in memory.
2025-04-15 01:40:57 +03:00
tth37
008fec80c1
fix: Update external search/loader method to POST
2025-04-14 18:17:27 +08:00
tth37
22f0365cef
format
2025-04-14 02:05:58 +08:00
tth37
839ba22c90
feat: Backend for Self-Hosted/External Web Search/Loader Engines
2025-04-14 01:49:05 +08:00
Timothy Jaeryang Baek
91a455a284
chore: format
2025-04-12 16:35:11 -07:00
Timothy Jaeryang Baek
48a23ce3fe
refac: web/rag config
2025-04-12 16:33:36 -07:00
Tim Jaeryang Baek
62ef0bad6f
Merge pull request #12680 from lucyknada/patch-1
...
fix #12678
2025-04-10 08:46:41 -07:00
Timothy Jaeryang Baek
63e5200e2f
refac
2025-04-10 08:46:12 -07:00
Youggls
3e2a6df1fb
feat: Add sougou web search API for backend, add config panel in for frontend.
2025-04-10 14:51:44 +08:00
lucy
bc295546cd
fix #12678
2025-04-10 07:23:34 +02:00
Tim Jaeryang Baek
2575dac4ed
Merge pull request #12604 from maurerle/ddg_improve_stacktrace
...
Deploy to HuggingFace Spaces / check-secret (push) Waiting to run
Deploy to HuggingFace Spaces / deploy (push) Blocked by required conditions
Create and publish Docker images with specific build args / build-main-image (linux/amd64) (push) Waiting to run
Create and publish Docker images with specific build args / build-main-image (linux/arm64) (push) Waiting to run
Create and publish Docker images with specific build args / build-cuda-image (linux/amd64) (push) Waiting to run
Create and publish Docker images with specific build args / build-cuda-image (linux/arm64) (push) Waiting to run
Create and publish Docker images with specific build args / build-ollama-image (linux/amd64) (push) Waiting to run
Create and publish Docker images with specific build args / build-ollama-image (linux/arm64) (push) Waiting to run
Create and publish Docker images with specific build args / merge-main-images (push) Blocked by required conditions
Create and publish Docker images with specific build args / merge-cuda-images (push) Blocked by required conditions
Create and publish Docker images with specific build args / merge-ollama-images (push) Blocked by required conditions
Python CI / Format Backend (3.11.x) (push) Waiting to run
Python CI / Format Backend (3.12.x) (push) Waiting to run
Frontend Build / Format & Build Frontend (push) Waiting to run
Frontend Build / Frontend Unit Tests (push) Waiting to run
**fix** improve stack trace of duckduckgo exception
2025-04-08 13:03:57 -07:00
Robert Norberg
2337b36609
add debug logging to RAG utils
2025-04-08 12:08:32 -04:00
Florian Maurer
760ea3f4af
duckduckgo: backend api has been deprecated since december
...
also increase duckduckgo-search version
see 3ee8e08b1c
2025-04-08 14:02:06 +02:00
Florian Maurer
337c7caafa
improve stack trace of duckduckgo exception
...
* fix search_results out of scope
* ddgs.text does already always return a list
2025-04-08 13:52:23 +02:00
Timothy Jaeryang Baek
65ed76abe1
refac: embedding prefix
2025-04-06 17:17:24 -07:00
Timothy Jaeryang Baek
ef787e4a79
Merge pull request #12486 from FabioPolito24/text-file-handling-docling
...
fix: text file handling with docling
2025-04-05 09:55:51 -07:00
Fabio Polito
cd0a1b4852
fix: fix for text file handling with docling
2025-04-05 16:44:08 +00:00
Juan Calderon-Perez
324550423c
Fix formatting issues
2025-04-05 10:03:24 -04:00
Phlogi
8cf8121812
Update utils.py
...
Avoid running any tasks for collections that failed to fetch data (have assigned None)
2025-04-05 10:41:21 +02:00
Patrick Wachter
0ac00b9256
refactor: update import path for MistralLoader
2025-04-02 13:56:10 +02:00
Patrick Wachter
c5a8d2f857
refactor: update MistralLoader documentation and adjust parameters for signed URL retrieval
2025-04-01 20:14:34 +02:00
Patrick Wachter
93d7702e8c
refactor: move MistralLoader to a separate module and just use the requests package instead of mistralai
2025-04-01 20:14:34 +02:00
Patrick Wachter
1ac6879268
Add Mistral OCR integration and configuration support
2025-04-01 14:24:33 +02:00
Timothy Jaeryang Baek
391dd33da3
chore: format
2025-03-31 17:59:21 -07:00
Timothy Jaeryang Baek
3ba12e7a43
Merge pull request #12239 from Phlogi/dev-threads-on-hybrid
...
perf: parallelize hybrid search
2025-03-31 17:06:32 -07:00
Timothy Jaeryang Baek
cafc5413f5
refac
2025-03-31 14:13:27 -07:00
Phlogi
9c64310db5
Run hybrid_search in parallel
2025-03-31 16:43:37 +02:00
Timothy Jaeryang Baek
4b75966401
refac: embedding prefix var naming
2025-03-30 21:55:15 -07:00
Timothy Jaeryang Baek
433b5bddc1
Merge pull request #8594 from jayteaftw/main
...
feat: Support for instruct/prefixing embeddings
2025-03-30 21:54:44 -07:00
Timothy Jaeryang Baek
50b8dec3ac
fix/refac: hybrid search
2025-03-30 20:48:22 -07:00
Timothy Jaeryang Baek
ce0d82b55f
Merge pull request #12132 from Phlogi/dev-fetch-documents-once
...
Avoid multiple data fetching
2025-03-30 20:44:32 -07:00
Junaid Pinjari
e782e7d3a7
Fix: CSV loader encoding issue using autodetect_encoding=True
2025-03-29 13:14:53 +05:30
Phlogi
04bf9ddab2
Avoid multiple data fetching
2025-03-27 19:05:20 +01:00
Timothy Jaeryang Baek
4a79320253
chore: format
2025-03-27 01:40:28 -07:00
Timothy Jaeryang Baek
7490bc9100
Merge branch 'dev' into fix-db-order
2025-03-26 20:55:42 -07:00
Timothy Jaeryang Baek
9d834a8e90
Merge branch 'dev' into k_reranker
2025-03-26 20:50:31 -07:00
Marko Henning
7531b7dcaa
Satisfy github format check
2025-03-25 19:09:17 +01:00
Iván Baldo
115e46a6a2
Fix: Tika 3.1.0.0 sends a lot of blank lines which degrades the RAG results, strip them.
2025-03-25 14:53:14 -03:00
Marko Henning
94d9d3d590
Fix: Normalze all database distances to score in [0, 1]
2025-03-25 16:46:14 +01:00
Timothy Jaeryang Baek
38d524f6a0
chore: format
2025-03-24 11:35:32 -07:00
