216 Commits

Author	SHA1	Message	Date
teampen	efb4a710c8	adding Serply as an alternative web search	2024-06-09 20:44:34 -04:00
Timothy J. Baek	f2b9a5f5bf	refac: rag	2024-06-09 03:01:25 -07:00
mindspawn	6f9148ac4c	Update main.py	2024-06-07 21:41:30 -07:00
mindspawn	4ecc1c06d3	Update main.py	2024-06-07 21:18:04 -07:00
Timothy J. Baek	0495f01acb	feat: reset upload dir	2024-06-03 21:45:36 -07:00
Timothy J. Baek	61867c1545	Update searxng.py	2024-06-03 17:02:50 -07:00
Timothy J. Baek	4068a421bf	fix	2024-06-03 17:00:35 -07:00
Timothy Jaeryang Baek	768941bded	Merge pull request #2785 from cheahjs/feat/openai-embeddings-batch ... feat: add RAG_EMBEDDING_OPENAI_BATCH_SIZE to batch multiple embeddings	2024-06-03 13:50:14 -07:00
Jun Siang Cheah	7fefbb316d	fix: add backwards compat with older searxng urls	2024-06-03 21:13:10 +01:00
Timothy Jaeryang Baek	92d9b38110	Merge branch 'dev' into feat/openai-embeddings-batch	2024-06-03 12:39:09 -07:00
Lyuboslav Petrov	08443b3c55	Revert log level to debug	2024-06-03 12:48:40 +03:00
Lyuboslav Petrov	7e761a69a7	FIX searxng URL construction using params for arg passing ... Accept additional parameters such as language, time_range, and categories to tailor the search results. Raise an exception if a request error occurs during the search process. Use params argument to construct the query string Sort by relevance Expand docstring	2024-06-03 12:44:46 +03:00
Jun Siang Cheah	0cb8163321	feat: add RAG_EMBEDDING_OPENAI_BATCH_SIZE to batch multiple embeddings	2024-06-02 15:34:31 +01:00
Timothy J. Baek	a53796270f	refac: web search config	2024-06-01 20:08:08 -07:00
Timothy J. Baek	fbdfb7e4fa	refac: web search	2024-06-01 19:57:00 -07:00
Timothy J. Baek	999d2bc21b	refac: web search	2024-06-01 19:52:12 -07:00
Timothy J. Baek	912a704fdc	refac: web search settings	2024-06-01 19:40:48 -07:00
Timothy J. Baek	ea6b8984ab	refac: web search	2024-06-01 19:03:56 -07:00
Timothy J. Baek	74a8deb19f	refac	2024-05-27 14:25:36 -07:00
Timothy J. Baek	4685f523b6	refac	2024-05-27 12:48:08 -07:00
Jun Siang Cheah	276b7b90b8	Merge remote-tracking branch 'upstream/dev' into feat/backend-web-search	2024-05-26 11:31:23 +01:00
Timothy J. Baek	84bfebd05e	fix	2024-05-26 01:17:57 -07:00
Jun Siang Cheah	224a578e6b	Merge remote-tracking branch 'upstream/dev' into feat/backend-web-search	2024-05-20 19:53:23 +01:00
Jun Siang Cheah	eb509c460a	Merge remote-tracking branch 'origin/dev' into feat/backend-web-search	2024-05-20 18:01:29 +01:00
Timothy J. Baek	322db31dc9	fix: rag	2024-05-20 07:22:43 -07:00
Timothy J. Baek	5376525777	refac	2024-05-19 06:51:32 -07:00
Timothy J. Baek	400bfa5a02	fix: rag config.json	2024-05-17 19:53:38 -07:00
Jun Siang Cheah	5e1c408937	Merge branch 'dev' into feat/backend-web-search	2024-05-14 14:03:23 +08:00
Jun Siang Cheah	9ed1a31575	fix: continue with failures when bulk loading urls with WebBaseLoader	2024-05-12 15:19:07 +08:00
Jun Siang Cheah	77928ae141	Merge branch 'dev' of https://github.com/open-webui/open-webui into feat/web-search-toggle	2024-05-11 23:51:37 +08:00
Jun Siang Cheah	2660a6e5b8	feat: prototype frontend web search integration	2024-05-11 23:44:34 +08:00
Jun Siang Cheah	fb8069123e	feat: add WEB_SEARCH_RESULT_COUNT to control max number of results	2024-05-11 23:18:59 +08:00
Jun Siang Cheah	298e6848b3	feat: switch to config proxy, remove config_get/set	2024-05-10 15:03:24 +08:00
Jun Siang Cheah	058eb76568	feat: save UI config changes to config.json	2024-05-10 13:51:50 +08:00
Timothy J. Baek	06cbe337de	feat: youtube loader language env var	2024-05-08 10:51:29 -07:00
Timothy J. Baek	d3822f782c	feat: non-english youtube support	2024-05-08 10:47:05 -07:00
Aarni Koskela	61bb1f1dc8	fix: do not use hardware ID in document ID generation	2024-05-07 11:42:05 +03:00
Timothy Jaeryang Baek	635951b55c	Merge branch 'dev' into feat/backend-web-search	2024-05-06 16:26:44 -07:00
Timothy J. Baek	64ed0d1089	refac: include source name to citation	2024-05-06 16:16:26 -07:00
Timothy J. Baek	4c490132ba	refac: styling	2024-05-06 16:16:26 -07:00
Jun Siang Cheah	0872bea790	feat: show RAG query results as citations	2024-05-06 16:14:10 -07:00
Timothy J. Baek	cecb87b8c2	feat: web_loader_ssl_verification setting	2024-05-06 14:50:55 -07:00
Timothy J. Baek	95f579cabe	feat: rag ssl verification env var ... Co-Authored-By: Tobias Steidle <tobias.steidle@softwaredev.de>	2024-05-06 13:12:08 -07:00
Jun Siang Cheah	8b3e370a6e	fix: run formatter	2024-05-06 17:11:04 +08:00
Jun Siang Cheah	83f086ccdd	fix: do not return raw search exception due to API keys in URLs	2024-05-06 17:09:04 +08:00
Jun Siang Cheah	99e4edd364	feat: add websearch endpoint to RAG API ... fix: google PSE endpoint uses GET fix: google PSE returns link, not url fix: serper wrong field	2024-05-06 17:09:04 +08:00
Jun Siang Cheah	501ff7a98b	feat: backend implementation of various search APIs	2024-05-06 12:28:41 +08:00
tabacoWang	fffd283b0c	fix: ... fix: Change the type from int to float	2024-05-02 13:45:19 +08:00
Timothy J. Baek	0595c04909	feat: youtube rag	2024-05-01 17:17:00 -07:00
Yanyutin753	c0bb32d768	📌 fixed a bug where RAG would not reply after not reading the file correctly	2024-04-30 13:51:30 +08:00
Timothy Jaeryang Baek	1afc49c1e4	Merge pull request #1862 from cheahjs/feat/filter-local-rag-fetch ... feat: add ENABLE_LOCAL_WEB_FETCH to protect against SSRF attacks	2024-04-29 15:51:17 -07:00
Jun Siang Cheah	1c4e63f71e	feat: add ENABLE_LOCAL_WEB_FETCH to protect against SSRF attacks	2024-04-29 20:55:17 +01:00
Steven Kreitzer	5b8fd14470	fix: various api rag results	2024-04-29 12:17:36 -05:00
Yanyutin753	b0245a7eff	✨feat added environment variables and sync.yml	2024-04-28 06:54:26 +08:00
Timothy J. Baek	ce9a5d12e0	refac: rag pipeline	2024-04-27 15:38:50 -04:00
Timothy J. Baek	8f1563a7a5	fix: typo	2024-04-27 15:03:49 -04:00
Timothy J. Baek	9be56d68e0	refac: naming convention	2024-04-27 15:02:57 -04:00
Timothy J. Baek	cebf733b9d	refac: naming convention	2024-04-26 14:41:39 -04:00
Steven Kreitzer	69822e4c25	fix: sort ranking hybrid	2024-04-26 07:56:41 -05:00
Steven Kreitzer	9755cd5baa	feat: toggle hybrid search	2024-04-25 17:51:38 -05:00
Timothy J. Baek	984dbf13ab	revert: original rag pipeline	2024-04-25 17:03:00 -04:00
Steven Kreitzer	1c1d2c254d	fix: query collection api call	2024-04-25 13:38:18 -05:00
Steven Kreitzer	72090fab88	chore: update log line	2024-04-25 13:28:31 -05:00
Steven Kreitzer	c9c9660459	fix: address comment in pr #1687	2024-04-25 07:50:42 -05:00
Steven Kreitzer	c0259aad67	feat: hybrid search and reranking support	2024-04-24 07:55:10 -05:00
Steven Kreitzer	4e0b32b505	feat: hybrid search	2024-04-22 18:33:43 -05:00
Steven Kreitzer	f3e5700d49	feat: move to native sentence_transformer	2024-04-22 14:20:41 -05:00
Timothy J. Baek	713934edb6	refac	2024-04-20 15:21:52 -05:00
Timothy J. Baek	710850e442	refac: audio	2024-04-20 15:15:59 -05:00
Timothy J. Baek	741ed5dc4c	fix	2024-04-14 19:56:33 -04:00
Timothy J. Baek	b1b72441bb	feat: openai embeddings integration	2024-04-14 19:48:15 -04:00
Timothy J. Baek	b48e73fa43	feat: openai embeddings support	2024-04-14 19:15:39 -04:00
Timothy J. Baek	36ce157907	fix: integration	2024-04-14 18:47:45 -04:00
Timothy J. Baek	9cdb5bf9fe	feat: frontend integration	2024-04-14 18:31:40 -04:00
Timothy J. Baek	2952e61167	feat: external embeddings support	2024-04-14 17:55:00 -04:00
Timothy Jaeryang Baek	b9cadff16b	Merge pull request #1419 from lainedfles/embedding-model-fix-and-manual-update ... feat: improve embedding model update & resolve network dependency	2024-04-10 01:10:07 -07:00
Timothy J. Baek	582d11f191	refac: RAG_EMBEDDING_MODEL_PATH removed	2024-04-10 00:59:05 -07:00
Timothy J. Baek	cb2158a794	fix	2024-04-10 00:51:16 -07:00
Timothy J. Baek	abfcceecef	refac	2024-04-10 00:46:09 -07:00
Timothy J. Baek	f4b87ecb23	refac	2024-04-10 00:33:45 -07:00
Steven Kreitzer	0bae789d39	fix: support batching chromadb	2024-04-09 10:13:29 -05:00
lainedfles	506a061387	Merge branch 'dev' into embedding-model-fix-and-manual-update	2024-04-08 14:57:54 -06:00
Jannik S	3b3d0cce1e	Merge branch 'dev' into dockerfile-optimisation	2024-04-08 09:15:00 +02:00
Timothy J. Baek	e61e1b079f	fix: file upload issue	2024-04-04 17:38:59 -07:00
Self Denial	9f82f5abba	Formatting...	2024-04-04 12:09:48 -06:00
Self Denial	075fbedb02	More format fixes	2024-04-04 12:07:42 -06:00
Self Denial	bcf79c8366	Format fixes	2024-04-04 12:02:48 -06:00
Self Denial	3b66aa55c0	Improve embedding model update & resolve network dependency ... * Add config variable RAG_EMBEDDING_MODEL_AUTO_UPDATE to control update behavior * Add RAG utils embedding_model_get_path() function to output the filesystem path in addition to update of the model using huggingface_hub * Update and utilize existing RAG functions in main: get_embedding_model() & update_embedding_model() * Add GUI setting to execute manual update process	2024-04-04 11:01:23 -06:00
Mmx233	947c392f72	fix: manually check the docs' filename	2024-04-03 23:37:13 +08:00
Jannik Streidl	9bcb37ea10	fixes and updates	2024-04-02 14:47:52 +02:00
Jannik S	099b1d066b	Revert "Merge Updates & Dockerfile improvements" (#3) ... This reverts commit 9763d885be.	2024-04-02 11:28:04 +02:00
lainedfles	9763d885be	Merge Updates & Dockerfile improvements	2024-04-02 11:25:20 +02:00
Timothy J. Baek	5558514ff1	fix	2024-04-01 15:23:12 -07:00
KoreLogic Disclosures	6c96361402	Suggested mitigation for KL-CAN-2024-002.	2024-04-01 15:55:14 -05:00
Timothy J. Baek	a6c154d839	feat: rag context logging	2024-03-31 14:02:31 -07:00
Self Denial	144c9059a3	Improve logging. Move print() statements to appropiate log(). ... Add COMFYUI and WEBHOOK logging and associated environment variable control. Add WEBHOOK payload & request debug logs.	2024-03-31 13:17:29 -06:00
Timothy J. Baek	3688955c77	fix: encoding issue	2024-03-25 23:50:52 -07:00
Timothy J. Baek	6307adfba1	feat: better error handling	2024-03-25 23:47:08 -07:00
Doug Danat	c91a5d8b1f	switch to using BeautifulSoup HTML loader so title is also captured	2024-03-25 11:26:18 +01:00
Doug Danat	784a6ec85e	include html langchain loader for RAG	2024-03-25 09:50:53 +01:00