Tim Jaeryang Baek
61580e9490
Merge pull request #13404 from NoMoreFood/dev
...
fix: Use SHA256 For Query Result Computation
2025-05-01 04:55:16 -07:00
Bryan Berns
32257089f9
Use SHA256 For Query Result Computation
2025-05-01 03:56:20 -04:00
Alexander Grimm
da9966aca1
~ truncate vectors for pgvector if too big
2025-04-30 05:35:17 +00:00
Tim Jaeryang Baek
4ee5dd58b7
Merge pull request #13177 from tth37/fix_firecrawl_loader_default_mode
...
Deploy to HuggingFace Spaces / check-secret (push) Waiting to run
Deploy to HuggingFace Spaces / deploy (push) Blocked by required conditions
Create and publish Docker images with specific build args / build-main-image (linux/amd64) (push) Waiting to run
Create and publish Docker images with specific build args / build-main-image (linux/arm64) (push) Waiting to run
Create and publish Docker images with specific build args / build-cuda-image (linux/amd64) (push) Waiting to run
Create and publish Docker images with specific build args / build-cuda-image (linux/arm64) (push) Waiting to run
Create and publish Docker images with specific build args / build-ollama-image (linux/amd64) (push) Waiting to run
Create and publish Docker images with specific build args / build-ollama-image (linux/arm64) (push) Waiting to run
Create and publish Docker images with specific build args / merge-main-images (push) Blocked by required conditions
Create and publish Docker images with specific build args / merge-cuda-images (push) Blocked by required conditions
Create and publish Docker images with specific build args / merge-ollama-images (push) Blocked by required conditions
Python CI / Format Backend (3.11.x) (push) Waiting to run
Python CI / Format Backend (3.12.x) (push) Waiting to run
Frontend Build / Format & Build Frontend (push) Waiting to run
Frontend Build / Frontend Unit Tests (push) Waiting to run
fix: FireCrawlLoader default mode to scrape
2025-04-29 08:39:06 -07:00
Tim Jaeryang Baek
e87f2669fa
Merge pull request #13191 from tth37/feat_firecrawl_search_engine
...
feat: Add Firecrawl search engine
2025-04-29 08:38:28 -07:00
Tim Jaeryang Baek
7b863465a9
Merge pull request #13311 from stephen304/yacy-support
...
feat: Yacy search support
2025-04-29 08:35:10 -07:00
Stephen Smith
ea16426a8d
Remove unused kwargs in yacy, update comments.
2025-04-27 00:41:46 -04:00
Stephen Smith
f9b9217e98
Set Yacy search to text
2025-04-26 23:13:31 -04:00
Stephen Smith
e6d43d70f3
Don't request nav and pass count to Yacy
2025-04-26 23:08:16 -04:00
Stephen Smith
240d91d38d
Add yacy config for user/pass, automatically add yacy json api path
2025-04-26 22:28:30 -04:00
Stephen Smith
0f73b96616
first pass at yacy support copied from searxng
2025-04-26 14:07:13 -04:00
tth37
92dbeb1939
feat: Add Firecrawl search engine
2025-04-24 14:57:28 +08:00
tth37
8f7195ceda
fix: FireCrawlLoader default mode to scrape
2025-04-24 01:17:35 +08:00
Tim Jaeryang Baek
91e758f3ec
Merge pull request #13165 from feddersen-group/perf/parallel_knowledge_search
...
perf: all knowledge searches in parallel in non-hybrid mode
2025-04-23 10:01:06 -07:00
Timothy Jaeryang Baek
09874ab83d
fix: FireCrawlLoader
2025-04-24 01:40:34 +09:00
Alexander Grimm
d182155fac
~ call knowledge searches in parallel in non-hybrid mode
2025-04-23 09:20:51 +00:00
Tim Jaeryang Baek
faa3cac0e4
Merge pull request #13107 from tth37/fix_tavily_max_results
...
fix: `max_results` in Tavily search handler
2025-04-22 23:47:36 -07:00
tth37
bc315bd530
fix: max_results in Tavily search api
2025-04-21 20:59:47 +08:00
Athanasios Oikonomou
1e291aff25
feat: Add abstract base class for vector database integration
...
- Created `VectorDBBase` as an abstract base class to standardize vector database operations.
- Added required methods for common vector database operations: `has_collection`, `delete_collection`, `insert`, `upsert`, `search`, `query`, `get`, `delete`, `reset`.
- The base class can now be extended by any vector database implementation (e.g., Qdrant, Pinecone) to ensure a consistent API across different database systems.
2025-04-21 08:27:27 +03:00
Athanasios Oikonomou
e000c56ef7
feat(vector-db): add support for Pinecone client
...
Adds Pinecone as a supported vector database option.
- Implements `PineconeClient` with support for common operations: `add`, `query`, `delete`, `reset`.
- Emulates namespace support using metadata filtering (`collection_name` prefix).
- Dynamically configures Pinecone via the following env vars:
- `PINECONE_API_KEY`
- `PINECONE_ENVIRONMENT`
- `PINECONE_INDEX_NAME`
- `PINECONE_DIMENSION`
- `PINECONE_METRIC`
- `PINECONE_CLOUD`
- Integrates cleanly with the vector DB abstraction layer.
- Includes markdown documentation under `docs/getting-started/env-configuration.md`.
BREAKING CHANGE: None
2025-04-20 11:08:51 +03:00
Tim Jaeryang Baek
87844a8042
Merge pull request #12822 from tth37/feat_external_search_loader
...
feat: Support for Self-Hosted/External Web Search/Loader Engines
2025-04-18 23:51:27 -07:00
Juan Calderon-Perez
6188c0c5b7
Add suport for Qdrant GRPC
2025-04-17 01:13:49 -04:00
Juan Calderon-Perez
b4d0d840d1
Fix formatting of qdrant.py
2025-04-15 08:56:51 -04:00
Athanasios Oikonomou
575c12f80c
feat: add QDRANT_ON_DISK configuration option for Qdrant integration
...
This commit will allow configuring the on_disk client parameter, to reduce the memory usage.
https://qdrant.tech/documentation/concepts/storage/?q=mmap#configuring-memmap-storage
Default is false, keeping vectors in memory.
2025-04-15 01:40:57 +03:00
tth37
008fec80c1
fix: Update external search/loader method to POST
2025-04-14 18:17:27 +08:00
tth37
22f0365cef
format
2025-04-14 02:05:58 +08:00
tth37
839ba22c90
feat: Backend for Self-Hosted/External Web Search/Loader Engines
2025-04-14 01:49:05 +08:00
Timothy Jaeryang Baek
91a455a284
chore: format
2025-04-12 16:35:11 -07:00
Timothy Jaeryang Baek
48a23ce3fe
refac: web/rag config
2025-04-12 16:33:36 -07:00
Tim Jaeryang Baek
62ef0bad6f
Merge pull request #12680 from lucyknada/patch-1
...
fix #12678
2025-04-10 08:46:41 -07:00
Timothy Jaeryang Baek
63e5200e2f
refac
2025-04-10 08:46:12 -07:00
Youggls
3e2a6df1fb
feat: Add sougou web search API for backend, add config panel in for frontend.
2025-04-10 14:51:44 +08:00
lucy
bc295546cd
fix #12678
2025-04-10 07:23:34 +02:00
Tim Jaeryang Baek
2575dac4ed
Merge pull request #12604 from maurerle/ddg_improve_stacktrace
...
Deploy to HuggingFace Spaces / check-secret (push) Waiting to run
Deploy to HuggingFace Spaces / deploy (push) Blocked by required conditions
Create and publish Docker images with specific build args / build-main-image (linux/amd64) (push) Waiting to run
Create and publish Docker images with specific build args / build-main-image (linux/arm64) (push) Waiting to run
Create and publish Docker images with specific build args / build-cuda-image (linux/amd64) (push) Waiting to run
Create and publish Docker images with specific build args / build-cuda-image (linux/arm64) (push) Waiting to run
Create and publish Docker images with specific build args / build-ollama-image (linux/amd64) (push) Waiting to run
Create and publish Docker images with specific build args / build-ollama-image (linux/arm64) (push) Waiting to run
Create and publish Docker images with specific build args / merge-main-images (push) Blocked by required conditions
Create and publish Docker images with specific build args / merge-cuda-images (push) Blocked by required conditions
Create and publish Docker images with specific build args / merge-ollama-images (push) Blocked by required conditions
Python CI / Format Backend (3.11.x) (push) Waiting to run
Python CI / Format Backend (3.12.x) (push) Waiting to run
Frontend Build / Format & Build Frontend (push) Waiting to run
Frontend Build / Frontend Unit Tests (push) Waiting to run
**fix** improve stack trace of duckduckgo exception
2025-04-08 13:03:57 -07:00
Robert Norberg
2337b36609
add debug logging to RAG utils
2025-04-08 12:08:32 -04:00
Florian Maurer
760ea3f4af
duckduckgo: backend api has been deprecated since december
...
also increase duckduckgo-search version
see 3ee8e08b1c
2025-04-08 14:02:06 +02:00
Florian Maurer
337c7caafa
improve stack trace of duckduckgo exception
...
* fix search_results out of scope
* ddgs.text does already always return a list
2025-04-08 13:52:23 +02:00
Timothy Jaeryang Baek
65ed76abe1
refac: embedding prefix
2025-04-06 17:17:24 -07:00
Timothy Jaeryang Baek
ef787e4a79
Merge pull request #12486 from FabioPolito24/text-file-handling-docling
...
fix: text file handling with docling
2025-04-05 09:55:51 -07:00
Fabio Polito
cd0a1b4852
fix: fix for text file handling with docling
2025-04-05 16:44:08 +00:00
Juan Calderon-Perez
324550423c
Fix formatting issues
2025-04-05 10:03:24 -04:00
Phlogi
8cf8121812
Update utils.py
...
Avoid running any tasks for collections that failed to fetch data (have assigned None)
2025-04-05 10:41:21 +02:00
Patrick Wachter
0ac00b9256
refactor: update import path for MistralLoader
2025-04-02 13:56:10 +02:00
Patrick Wachter
c5a8d2f857
refactor: update MistralLoader documentation and adjust parameters for signed URL retrieval
2025-04-01 20:14:34 +02:00
Patrick Wachter
93d7702e8c
refactor: move MistralLoader to a separate module and just use the requests package instead of mistralai
2025-04-01 20:14:34 +02:00
Patrick Wachter
1ac6879268
Add Mistral OCR integration and configuration support
2025-04-01 14:24:33 +02:00
Timothy Jaeryang Baek
391dd33da3
chore: format
2025-03-31 17:59:21 -07:00
Timothy Jaeryang Baek
3ba12e7a43
Merge pull request #12239 from Phlogi/dev-threads-on-hybrid
...
perf: parallelize hybrid search
2025-03-31 17:06:32 -07:00
Timothy Jaeryang Baek
cafc5413f5
refac
2025-03-31 14:13:27 -07:00
Phlogi
9c64310db5
Run hybrid_search in parallel
2025-03-31 16:43:37 +02:00
