OpenWebUI/open-webui
1
0
Fork 0
mirror of https://github.com/open-webui/open-webui synced 2025-06-26 02:07:15 +00:00
Code Issues Actions 4 Packages Projects Releases Wiki Activity

Update package.json

Browse Source
This commit is contained in:
karl 2025-06-22 12:04:21 +08:00
parent 8a15949b84
commit b19dda393b
13 changed files with 897 additions and 216 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
.serena/cache/python/document_symbols_cache_v20-05-25.pkl vendored Normal file
View File

Binary file not shown.

68
.serena/memories/code_structure.md Normal file
View File

@ -0,0 +1,68 @@
# Code Structure and Organization
## Backend Structure (`backend/open_webui/`)
- **`main.py`**: Main FastAPI application entry point
- **`config.py`**: Configuration management and environment variables
- **`env.py`**: Environment setup and constants
- **`constants.py`**: Application constants and message templates
- **`functions.py`**: Function execution and management
- **`tasks.py`**: Background task management
## Router Organization (`backend/open_webui/routers/`)
Each router handles a specific domain:
- **`auths.py`**: Authentication and authorization
- **`users.py`**: User management
- **`chats.py`**: Chat conversations
- **`models.py`**: AI model management
- **`prompts.py`**: Prompt templates
- **`tools.py`**: Tool management
- **`functions.py`**: Function management
- **`files.py`**: File upload/management
- **`images.py`**: Image generation
- **`audio.py`**: Speech-to-text and text-to-speech
- **`retrieval.py`**: RAG and document processing
- **`memories.py`**: Memory management
- **`knowledge.py`**: Knowledge base management
- **`ollama.py`**: Ollama integration
- **`openai.py`**: OpenAI API integration
- **`pipelines.py`**: Pipeline management
- **`configs.py`**: Configuration management
## Database Models (`backend/open_webui/models/`)
- **`users.py`**: User model and settings
- **`chats.py`**: Chat conversations
- **`models.py`**: AI model definitions
- **`files.py`**: File metadata
- **`auths.py`**: Authentication data
- **`prompts.py`**: Prompt templates
- **`tools.py`**: Tool definitions
- **`functions.py`**: Function definitions
- **`memories.py`**: Memory storage
- **`knowledge.py`**: Knowledge base
- **`channels.py`**: Communication channels
- **`folders.py`**: Organization folders
- **`feedbacks.py`**: User feedback
## Frontend Structure (`src/`)
- **`app.html`**: Main HTML template
- **`app.css`**: Global styles
- **`lib/`**: Reusable components and utilities
- **`routes/`**: SvelteKit page routes
## Utilities (`backend/open_webui/utils/`)
- **`auth.py`**: Authentication utilities
- **`misc.py`**: General utilities
- **`models.py`**: Model utilities
- **`chat.py`**: Chat processing
- **`middleware.py`**: Request/response processing
- **`tools.py`**: Tool execution
- **`embeddings.py`**: Embedding generation
- **`code_interpreter.py`**: Code execution
- **`filter.py`**: Content filtering
- **`plugin.py`**: Plugin management

66
.serena/memories/code_style_conventions.md Normal file
View File

@ -0,0 +1,66 @@
# Code Style and Conventions
## Python Backend Style
- **Formatter**: Black with default settings
- **Linter**: Pylint
- **Type Hints**: Strongly encouraged, especially for function signatures
- **Docstrings**: Use for public APIs and complex functions
- **Import Organization**: Follow PEP 8 standards
- **Variable Naming**: snake_case for variables and functions, PascalCase for classes
- **Constants**: UPPER_CASE for module-level constants
## Code Quality Standards
- **Line Length**: Black default (88 characters)
- **String Quotes**: Black will standardize (double quotes preferred)
- **Trailing Commas**: Black handles automatically
- **Function Organization**: Keep functions focused and single-purpose
- **Error Handling**: Use proper exception handling with specific exception types
## API Design Patterns
- **FastAPI Routers**: Organize endpoints by domain (users, chats, models, etc.)
- **Pydantic Models**: Use for request/response validation
- **Response Models**: Consistent JSON structure with proper HTTP status codes
- **Authentication**: JWT-based with dependency injection
- **Database Models**: SQLAlchemy ORM with proper relationships
## Frontend Style
- **Framework**: SvelteKit with TypeScript
- **Styling**: Tailwind CSS utility classes
- **Component Organization**: Modular components in `src/lib/`
- **State Management**: Svelte stores for global state
- **Type Safety**: TypeScript throughout the frontend
## Configuration Management
- **Environment Variables**: Extensive use of env vars for configuration
- **Default Values**: Sensible defaults in `config.py`
- **Validation**: Pydantic for configuration validation
- **Documentation**: Document all configuration options
## Database Design
- **Migrations**: Alembic for database schema changes
- **Relationships**: Proper foreign keys and relationships
- **Indexes**: Strategic indexing for performance
- **Naming**: Descriptive table and column names
## Security Practices
- **Authentication**: JWT tokens with proper expiration
- **Authorization**: Role-based access control
- **Input Validation**: Pydantic models for all inputs
- **SQL Injection**: SQLAlchemy ORM prevents direct SQL
- **CORS**: Proper CORS configuration
- **Environment Secrets**: Never commit secrets to version control
## Testing Conventions
- **Backend Tests**: Pytest with fixtures
- **Frontend Tests**: Vitest for unit tests
- **E2E Tests**: Cypress for integration testing
- **Test Organization**: Mirror source code structure
- **Mocking**: Mock external dependencies in tests

124
.serena/memories/macos_development_guide.md Normal file
View File

@ -0,0 +1,124 @@
# macOS Development Environment Setup
## System Requirements
- **Operating System**: macOS
- **Python**: 3.11+ (required for backend)
- **Node.js**: 18.13.0+ (required for frontend)
- **Package Managers**: npm 6.0.0+, optionally uv for Python
## macOS Specific Commands
### System Information
```bash
# Check macOS version
sw_vers
# Check available memory
vm_stat | head -5
# Check disk space
df -h
# Check CPU information
sysctl -n machdep.cpu.brand_string
# Check running processes
ps aux | grep open-webui
```
### Package Management
```bash
# Install Homebrew (if not installed)
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# Install Python with Homebrew
brew install python@3.11
# Install Node.js with Homebrew
brew install node
# Install Docker Desktop for Mac
brew install --cask docker
# Install uv (modern Python package manager)
brew install uv
```
### File System Operations
```bash
# Navigate to project directory
cd /path/to/open-webui-next
# Find files
find . -name "*.py" -type f # Find Python files
find . -name "*.svelte" -type f # Find Svelte files
# Search in files
grep -r "search_term" backend/ # Search in backend
grep -r "search_term" src/ # Search in frontend
# File permissions
chmod +x backend/start.sh # Make script executable
```
### Network and Ports
```bash
# Check if port is in use
lsof -i :8080 # Check port 8080
lsof -i :3000 # Check port 3000 (frontend dev)
# Kill process on port
kill -9 $(lsof -ti:8080) # Kill process on port 8080
```
### Environment Management
```bash
# Python virtual environment
python3 -m venv venv
source venv/bin/activate
deactivate
# Environment variables
export OPENAI_API_KEY="your-key"
echo $OPENAI_API_KEY
printenv | grep WEBUI
```
### Troubleshooting Commands
```bash
# Check Python installation
which python3
python3 --version
# Check Node.js installation
which node
node --version
npm --version
# Check Docker
docker --version
docker ps
docker images
# Clear npm cache
npm cache clean --force
# Clear Python cache
find . -type d -name "__pycache__" -delete
find . -name "*.pyc" -delete
```
## Development Workflow on macOS
1. Use Terminal or iTerm2 for command line operations
2. Consider using VS Code or PyCharm for development
3. Use Docker Desktop for containerized development
4. Monitor system resources with Activity Monitor
5. Use Homebrew for package management

46
.serena/memories/project_overview.md Normal file
View File

@ -0,0 +1,46 @@
# Open WebUI Project Overview
## Purpose
Open WebUI is an extensible, feature-rich, and user-friendly self-hosted AI platform designed to operate entirely offline. It supports various LLM runners like Ollama and OpenAI-compatible APIs, with built-in inference engine for RAG, making it a powerful AI deployment solution.
## Key Features
- Effortless setup with Docker or Kubernetes
- Ollama/OpenAI API integration
- Granular permissions and user groups
- Responsive design with PWA support
- Full Markdown and LaTeX support
- Voice/video call functionality
- Model builder for custom models
- Native Python function calling
- Local RAG integration
- Web search capabilities
- Image generation integration
- Multi-model conversations
- Role-based access control (RBAC)
- Multilingual support
- Plugin framework with Pipelines
## Tech Stack
- **Backend**: Python 3.11+ with FastAPI
- **Frontend**: SvelteKit with TypeScript
- **Database**: SQLAlchemy with support for PostgreSQL, MySQL, SQLite
- **Vector Database**: Chroma, Milvus, Qdrant, OpenSearch, Elasticsearch, PGVector, Pinecone
- **Deployment**: Docker, Kubernetes
- **Build Tools**: Vite, Node.js
- **Styling**: Tailwind CSS
- **Testing**: Pytest (backend), Vitest (frontend), Cypress (e2e)
## Architecture
The project follows a modern full-stack architecture:
- **Backend**: Python FastAPI application serving REST APIs and WebSocket connections
- **Frontend**: SvelteKit SPA that communicates with the backend APIs
- **Database Layer**: SQLAlchemy ORM with Alembic migrations
- **Vector Storage**: Pluggable vector database support for RAG functionality
- **Authentication**: JWT-based authentication with OAuth support
- **Real-time**: WebSocket support for live features
- **File Storage**: Configurable storage providers (Local, S3, GCS, Azure)

111
.serena/memories/suggested_commands.md Normal file
View File

@ -0,0 +1,111 @@
# Development Commands and Scripts
## Essential Commands for Development
### Backend Development
```bash
# Install dependencies
pip install -r backend/requirements.txt
# Run backend in development mode
cd backend && python -m uvicorn open_webui.main:app --host 0.0.0.0 --port 8080 --reload
# Run with uv (modern Python package manager)
cd backend && uv run uvicorn open_webui.main:app --host 0.0.0.0 --port 8080 --reload
# Database migrations
alembic upgrade head
alembic revision --autogenerate -m "description"
# Run tests
pytest backend/
# Code formatting
black . --exclude ".venv/|/venv/"
# Linting
pylint backend/
```
### Frontend Development
```bash
# Install dependencies
npm install
# Development server
npm run dev
npm run dev:5050 # Run on port 5050
# Build for production
npm run build
# Build with watch mode
npm run build:watch
# Preview production build
npm run preview
# Type checking
npm run check
npm run check:watch
# Linting
npm run lint:frontend
# Formatting
npm run format
# Prepare Pyodide
npm run pyodide:fetch
# Internationalization
npm run i18n:parse
```
### Full Stack Commands
```bash
# Format both frontend and backend
npm run format && npm run format:backend
# Lint everything
npm run lint # Runs lint:frontend, lint:types, lint:backend
# Testing
npm run test:frontend
pytest backend/ # Backend tests
# End-to-end testing
npm run cy:open
```
### Docker Development
```bash
# Using Makefile
make install # docker-compose up -d
make start # docker-compose start
make stop # docker-compose stop
make startAndBuild # docker-compose up -d --build
make update # Update and rebuild
# Direct docker-compose
docker-compose up -d
docker-compose up -d --build
docker-compose logs -f
```
### Database Commands
```bash
# Reset database
rm backend/data/webui.db # For SQLite
# Run migrations
cd backend && alembic upgrade head
# Create new migration
cd backend && alembic revision --autogenerate -m "migration description"
```

106
.serena/memories/task_completion_workflow.md Normal file
View File

@ -0,0 +1,106 @@
# Task Completion Workflow
## When a Development Task is Completed
### 1. Code Quality Checks
```bash
# Format code
npm run format # Frontend formatting
npm run format:backend # Backend formatting (Black)
# Lint code
npm run lint # Full linting (frontend + backend)
pylint backend/ # Backend specific linting
# Type checking
npm run check # TypeScript type checking
```
### 2. Testing
```bash
# Run unit tests
npm run test:frontend # Frontend tests with Vitest
pytest backend/ # Backend tests with Pytest
# Run integration tests (if applicable)
npm run cy:open # Cypress e2e tests
```
### 3. Build Verification
```bash
# Test production build
npm run build # Build frontend
npm run preview # Preview production build
# Test backend startup
cd backend && python -m uvicorn open_webui.main:app --host 0.0.0.0 --port 8080
```
### 4. Database Migrations (if schema changed)
```bash
# Generate migration if database models were modified
cd backend && alembic revision --autogenerate -m "description of changes"
# Apply migrations
cd backend && alembic upgrade head
```
### 5. Documentation Updates
- Update README.md if new features added
- Update API documentation if endpoints changed
- Update configuration documentation if new env vars added
- Update CHANGELOG.md following semantic versioning
### 6. Git Workflow
```bash
# Stage changes
git add .
# Commit with descriptive message
git commit -m "feat: add new feature description"
# or
git commit -m "fix: resolve bug description"
# or
git commit -m "docs: update documentation"
# Push changes
git push origin feature-branch
```
### 7. System Verification Commands (macOS)
```bash
# Check system resources
ps aux | grep open-webui # Check if processes are running
lsof -i :8080 # Check if port is in use
df -h # Check disk space
free -m # Check memory usage (if available)
# Docker verification (if using Docker)
docker ps # Check running containers
docker logs open-webui # Check container logs
```
### 8. Performance Verification
- Check application startup time
- Verify API response times
- Test memory usage under load
- Verify frontend bundle sizes are reasonable
### 9. Pre-deployment Checklist
- [ ] All tests passing
- [ ] Code properly formatted and linted
- [ ] Documentation updated
- [ ] Environment variables documented
- [ ] Database migrations tested
- [ ] No secrets in code
- [ ] Performance is acceptable
- [ ] Security considerations addressed

66
.serena/project.yml Normal file
View File

@ -0,0 +1,66 @@
# language of the project (csharp, python, rust, java, typescript, javascript, go, cpp, or ruby)
# Special requirements:
# * csharp: Requires the presence of a .sln file in the project folder.
language: python
# whether to use the project's gitignore file to ignore files
# Added on 2025-04-07
ignore_all_files_in_gitignore: true
# list of additional paths to ignore
# same syntax as gitignore, so you can use * and **
# Was previously called `ignored_dirs`, please update your config if you are using that.
# Added (renamed)on 2025-04-07
ignored_paths: []
# whether the project is in read-only mode
# If set to true, all editing tools will be disabled and attempts to use them will result in an error
# Added on 2025-04-18
read_only: false
# list of tool names to exclude. We recommend not excluding any tools, see the readme for more details.
# Below is the complete list of tools for convenience.
# To make sure you have the latest list of tools, and to view their descriptions,
# execute `uv run scripts/print_tool_overview.py`.
#
# * `activate_project`: Activates a project by name.
# * `check_onboarding_performed`: Checks whether project onboarding was already performed.
# * `create_text_file`: Creates/overwrites a file in the project directory.
# * `delete_lines`: Deletes a range of lines within a file.
# * `delete_memory`: Deletes a memory from Serena's project-specific memory store.
# * `execute_shell_command`: Executes a shell command.
# * `find_referencing_code_snippets`: Finds code snippets in which the symbol at the given location is referenced.
# * `find_referencing_symbols`: Finds symbols that reference the symbol at the given location (optionally filtered by type).
# * `find_symbol`: Performs a global (or local) search for symbols with/containing a given name/substring (optionally filtered by type).
# * `get_current_config`: Prints the current configuration of the agent, including the active and available projects, tools, contexts, and modes.
# * `get_symbols_overview`: Gets an overview of the top-level symbols defined in a given file or directory.
# * `initial_instructions`: Gets the initial instructions for the current project.
# Should only be used in settings where the system prompt cannot be set,
# e.g. in clients you have no control over, like Claude Desktop.
# * `insert_after_symbol`: Inserts content after the end of the definition of a given symbol.
# * `insert_at_line`: Inserts content at a given line in a file.
# * `insert_before_symbol`: Inserts content before the beginning of the definition of a given symbol.
# * `list_dir`: Lists files and directories in the given directory (optionally with recursion).
# * `list_memories`: Lists memories in Serena's project-specific memory store.
# * `onboarding`: Performs onboarding (identifying the project structure and essential tasks, e.g. for testing or building).
# * `prepare_for_new_conversation`: Provides instructions for preparing for a new conversation (in order to continue with the necessary context).
# * `read_file`: Reads a file within the project directory.
# * `read_memory`: Reads the memory with the given name from Serena's project-specific memory store.
# * `remove_project`: Removes a project from the Serena configuration.
# * `replace_lines`: Replaces a range of lines within a file with new content.
# * `replace_symbol_body`: Replaces the full definition of a symbol.
# * `restart_language_server`: Restarts the language server, may be necessary when edits not through Serena happen.
# * `search_for_pattern`: Performs a search for a pattern in the project.
# * `summarize_changes`: Provides instructions for summarizing the changes made to the codebase.
# * `switch_modes`: Activates modes by providing a list of their names
# * `think_about_collected_information`: Thinking tool for pondering the completeness of collected information.
# * `think_about_task_adherence`: Thinking tool for determining whether the agent is still on track with the current task.
# * `think_about_whether_you_are_done`: Thinking tool for determining whether the task is truly completed.
# * `write_memory`: Writes a named memory (for future reference) to Serena's project-specific memory store.
excluded_tools: []
# initial prompt for the project. It will always be given to the LLM upon activating the project
# (contrary to the memories, which are loaded on demand).
initial_prompt: ""
project_name: "open-webui-next"

2
backend/open_webui/routers/utils.py
View File

@ -63,7 +63,7 @@ async def execute_code(
else None
),
request.app.state.config.CODE_EXECUTION_JUPYTER_TIMEOUT,
form_data.chat_id # Pass chat_id to the enhanced function
form_data.chat_id, # Pass chat_id to the enhanced function
)
return output

259
backend/open_webui/utils/code_interpreter.py
View File

@ -10,6 +10,7 @@ import websockets
from pydantic import BaseModel
from open_webui.env import SRC_LOG_LEVELS
# Import necessary models for chat and file operations
from open_webui.models.chats import Chats
from open_webui.models.files import Files
@ -50,13 +51,15 @@ def get_attached_files_from_chat(chat_id: str) -> List[Dict[str, Any]]:
"type": file_info.get("type", "file"),
"size": file_info.get("size"),
"url": file_info.get("url"),
"message_id": message_id
"message_id": message_id,
}
# Only include files with valid IDs
if file_data["id"]:
attached_files.append(file_data)
logger.debug(f"Found attached file: {file_data['name']} (ID: {file_data['id']})")
logger.debug(
f"Found attached file: {file_data['name']} (ID: {file_data['id']})"
)
logger.info(f"Found {len(attached_files)} attached files in chat {chat_id}")
return attached_files
@ -66,7 +69,9 @@ def get_attached_files_from_chat(chat_id: str) -> List[Dict[str, Any]]:
return []
async def auto_prepare_chat_files(chat_id: str, data_dir: str = "data") -> Dict[str, Any]:
async def auto_prepare_chat_files(
chat_id: str, data_dir: str = "data"
) -> Dict[str, Any]:
"""
Automatically prepare files attached to chat messages for use in the Jupyter environment.
Creates symbolic links in the Jupyter data directory pointing to the uploaded files.
@ -88,7 +93,7 @@ async def auto_prepare_chat_files(chat_id: str, data_dir: str = "data") -> Dict[
"skipped_files": [],
"errors": [],
"total_files": 0,
"method": None # Will be "symlink" or "copy"
"method": None, # Will be "symlink" or "copy"
}
try:
@ -112,7 +117,9 @@ async def auto_prepare_chat_files(chat_id: str, data_dir: str = "data") -> Dict[
# use_symlinks = await _test_symlink_accessibility(chat_data_dir, data_dir)
method = "symlink" if use_symlinks else "copy"
result["method"] = method
logger.info(f"Using {method} method for file preparation (hardcoded for Docker compatibility)")
logger.info(
f"Using {method} method for file preparation (hardcoded for Docker compatibility)"
)
# Track successfully processed files to avoid duplicates
processed_file_ids = set()
@ -125,28 +132,31 @@ async def auto_prepare_chat_files(chat_id: str, data_dir: str = "data") -> Dict[
# Skip if already processed (deduplication)
if file_id in processed_file_ids:
logger.debug(f"Skipping duplicate file {file_name} (ID: {file_id})")
result["skipped_files"].append({
"name": file_name,
"id": file_id,
"reason": "duplicate"
})
result["skipped_files"].append(
{"name": file_name, "id": file_id, "reason": "duplicate"}
)
continue
# Get file from database
file_record = Files.get_file_by_id(file_id)
if not file_record:
logger.warning(f"File record not found for ID: {file_id}")
result["errors"].append(f"File record not found: {file_name} (ID: {file_id})")
result["errors"].append(
f"File record not found: {file_name} (ID: {file_id})"
)
continue
# Use the actual file path from the database
if not file_record.path:
logger.warning(f"File path not found in record for ID: {file_id}")
result["errors"].append(f"File path not found: {file_name} (ID: {file_id})")
result["errors"].append(
f"File path not found: {file_name} (ID: {file_id})"
)
continue
# Get the actual file path (handles different storage providers)
from open_webui.storage.provider import Storage
source_file_path = Storage.get_file(file_record.path)
# Check if source file exists
@ -172,23 +182,28 @@ async def auto_prepare_chat_files(chat_id: str, data_dir: str = "data") -> Dict[
# Create symbolic link using absolute path to ensure it resolves correctly
source_file_path_abs = os.path.abspath(source_file_path)
os.symlink(source_file_path_abs, target_path)
logger.info(f"Created symlink: {target_path} -> {source_file_path_abs}")
logger.info(
f"Created symlink: {target_path} -> {source_file_path_abs}"
)
else:
# Copy file
import shutil
shutil.copy2(source_file_path, target_path)
logger.info(f"Copied file: {source_file_path} -> {target_path}")
# Record successful preparation
result["prepared_files"].append({
result["prepared_files"].append(
{
"name": file_name,
"id": file_id,
"target_path": target_path,
"source_path": source_file_path,
"size": file_info.get("size"),
"type": file_info.get("type"),
"method": method
})
"method": method,
}
)
processed_file_ids.add(file_id)
@ -198,12 +213,16 @@ async def auto_prepare_chat_files(chat_id: str, data_dir: str = "data") -> Dict[
result["errors"].append(error_msg)
# Set success if we prepared at least some files or if there were no errors
result["success"] = len(result["prepared_files"]) > 0 or len(result["errors"]) == 0
result["success"] = (
len(result["prepared_files"]) > 0 or len(result["errors"]) == 0
)
logger.info(f"Auto-prepare completed for chat {chat_id}: "
logger.info(
f"Auto-prepare completed for chat {chat_id}: "
f"{len(result['prepared_files'])} prepared using {method}, "
f"{len(result['skipped_files'])} skipped, "
f"{len(result['errors'])} errors")
f"{len(result['errors'])} errors"
)
return result
@ -262,7 +281,9 @@ async def _test_symlink_accessibility(chat_data_dir: str, data_dir: str) -> bool
logger.warning("Symlink accessibility test failed - content mismatch")
return False
except Exception as e:
logger.warning(f"Symlink accessibility test failed - cannot read through symlink: {e}")
logger.warning(
f"Symlink accessibility test failed - cannot read through symlink: {e}"
)
return False
# Test 3: Can we stat the symlink target?
@ -280,9 +301,13 @@ async def _test_symlink_accessibility(chat_data_dir: str, data_dir: str) -> bool
except OSError as e:
if "Operation not supported" in str(e) or "Function not implemented" in str(e):
logger.info("Symlinks not supported on this filesystem - using file copying")
logger.info(
"Symlinks not supported on this filesystem - using file copying"
)
else:
logger.warning(f"Symlink test failed with OS error: {e} - using file copying")
logger.warning(
f"Symlink test failed with OS error: {e} - using file copying"
)
return False
except Exception as e:
logger.warning(f"Symlink test failed: {e} - using file copying")
@ -290,7 +315,9 @@ async def _test_symlink_accessibility(chat_data_dir: str, data_dir: str) -> bool
finally:
# Clean up test files
try:
if test_symlink and (os.path.exists(test_symlink) or os.path.islink(test_symlink)):
if test_symlink and (
os.path.exists(test_symlink) or os.path.islink(test_symlink)
):
os.unlink(test_symlink)
if test_source and os.path.exists(test_source):
os.remove(test_source)
@ -300,7 +327,9 @@ async def _test_symlink_accessibility(chat_data_dir: str, data_dir: str) -> bool
logger.debug(f"Test cleanup failed (non-critical): {e}")
async def prepare_multiple_chats_files(chat_ids: List[str], data_dir: str = "data") -> Dict[str, Any]:
async def prepare_multiple_chats_files(
chat_ids: List[str], data_dir: str = "data"
) -> Dict[str, Any]:
"""
Prepare files for multiple chats at once (bulk operation).
@ -322,8 +351,8 @@ async def prepare_multiple_chats_files(chat_ids: List[str], data_dir: str = "dat
"summary": {
"total_prepared_files": 0,
"total_skipped_files": 0,
"total_errors": 0
}
"total_errors": 0,
},
}
for chat_id in chat_ids:
@ -338,8 +367,12 @@ async def prepare_multiple_chats_files(chat_ids: List[str], data_dir: str = "dat
overall_result["success"] = False
# Update summary
overall_result["summary"]["total_prepared_files"] += len(chat_result["prepared_files"])
overall_result["summary"]["total_skipped_files"] += len(chat_result["skipped_files"])
overall_result["summary"]["total_prepared_files"] += len(
chat_result["prepared_files"]
)
overall_result["summary"]["total_skipped_files"] += len(
chat_result["skipped_files"]
)
overall_result["summary"]["total_errors"] += len(chat_result["errors"])
except Exception as e:
@ -347,12 +380,14 @@ async def prepare_multiple_chats_files(chat_ids: List[str], data_dir: str = "dat
logger.error(error_msg)
overall_result["chat_results"][chat_id] = {
"success": False,
"errors": [error_msg]
"errors": [error_msg],
}
overall_result["failed_chats"] += 1
overall_result["success"] = False
logger.info(f"Bulk prepare completed: {overall_result['successful_chats']}/{overall_result['total_chats']} successful")
logger.info(
f"Bulk prepare completed: {overall_result['successful_chats']}/{overall_result['total_chats']} successful"
)
return overall_result
@ -369,12 +404,7 @@ def test_filesystem_support(data_dir: str = "data") -> Dict[str, Any]:
"""
logger.info(f"Testing filesystem support in {data_dir}")
test_result = {
"success": True,
"tests": {},
"errors": [],
"recommendations": []
}
test_result = {"success": True, "tests": {}, "errors": [], "recommendations": []}
test_dir = os.path.join(data_dir, "test_auto_prepare")
@ -412,7 +442,9 @@ def test_filesystem_support(data_dir: str = "data") -> Dict[str, Any]:
logger.debug("✓ Symlink creation test passed")
else:
test_result["tests"]["symlink_creation"] = False
test_result["errors"].append("Cannot test symlink: source file doesn't exist")
test_result["errors"].append(
"Cannot test symlink: source file doesn't exist"
)
except Exception as e:
test_result["tests"]["symlink_creation"] = False
test_result["errors"].append(f"Symlink creation failed: {str(e)}")
@ -434,7 +466,9 @@ def test_filesystem_support(data_dir: str = "data") -> Dict[str, Any]:
test_result["errors"].append("Symlink path resolution incorrect")
else:
test_result["tests"]["path_resolution"] = False
test_result["errors"].append("Cannot test path resolution: symlink doesn't exist")
test_result["errors"].append(
"Cannot test path resolution: symlink doesn't exist"
)
except Exception as e:
test_result["tests"]["path_resolution"] = False
test_result["errors"].append(f"Path resolution test failed: {str(e)}")
@ -450,7 +484,9 @@ def test_filesystem_support(data_dir: str = "data") -> Dict[str, Any]:
logger.debug("✓ Symlink accessibility test passed")
else:
test_result["tests"]["symlink_accessibility"] = False
test_result["errors"].append("Symlink content mismatch - possible Docker volume issue")
test_result["errors"].append(
"Symlink content mismatch - possible Docker volume issue"
)
test_result["recommendations"].append(
"Symlinks may not work in Docker environment. Auto-prepare will use file copying."
)
@ -485,7 +521,9 @@ def test_filesystem_support(data_dir: str = "data") -> Dict[str, Any]:
if test_result["success"]:
logger.info("✓ All filesystem tests passed")
else:
logger.warning(f"⚠ Some filesystem tests failed: {len(test_result['errors'])} errors")
logger.warning(
f"⚠ Some filesystem tests failed: {len(test_result['errors'])} errors"
)
return test_result
@ -720,14 +758,18 @@ class EnterpriseGatewayCodeExecutor:
# Auto-prepare files for this chat before code execution
self.prepare_result = None
if self.chat_id:
logger.info(f"Auto-preparing files for chat {self.chat_id} before code execution")
logger.info(
f"Auto-preparing files for chat {self.chat_id} before code execution"
)
try:
# Note: This is synchronous but auto_prepare_chat_files is async
# We'll need to handle this in the run() method instead
self._auto_prepare_needed = True
logger.debug(f"Marked auto-prepare as needed for chat {self.chat_id}")
except Exception as e:
logger.error(f"Failed to mark auto-prepare for chat {self.chat_id}: {str(e)}")
logger.error(
f"Failed to mark auto-prepare for chat {self.chat_id}: {str(e)}"
)
self._auto_prepare_needed = False
else:
self._auto_prepare_needed = False
@ -735,7 +777,9 @@ class EnterpriseGatewayCodeExecutor:
if self.base_url[-1] != "/":
self.base_url += "/"
logger.info(f"Initializing Enterprise Gateway connection to {self.base_url} with kernel {self.kernel_name}")
logger.info(
f"Initializing Enterprise Gateway connection to {self.base_url} with kernel {self.kernel_name}"
)
if self.chat_id:
logger.info(f"Using chat ID {self.chat_id} for path replacement")
self.session = aiohttp.ClientSession(trust_env=True, base_url=self.base_url)
@ -748,17 +792,25 @@ class EnterpriseGatewayCodeExecutor:
return
try:
self.prepare_result = await auto_prepare_chat_files(self.chat_id, self.data_dir)
self.prepare_result = await auto_prepare_chat_files(
self.chat_id, self.data_dir
)
if self.prepare_result["success"]:
prepared_count = len(self.prepare_result["prepared_files"])
if prepared_count > 0:
logger.info(f"Successfully prepared {prepared_count} files for chat {self.chat_id}")
logger.info(
f"Successfully prepared {prepared_count} files for chat {self.chat_id}"
)
else:
logger.debug(f"No files to prepare for chat {self.chat_id}")
else:
logger.warning(f"File preparation had issues for chat {self.chat_id}: {self.prepare_result['errors']}")
logger.warning(
f"File preparation had issues for chat {self.chat_id}: {self.prepare_result['errors']}"
)
except Exception as e:
logger.error(f"Failed to auto-prepare files for chat {self.chat_id}: {str(e)}")
logger.error(
f"Failed to auto-prepare files for chat {self.chat_id}: {str(e)}"
)
# Continue with execution even if file preparation fails
def _prepare_code_with_path_replacement(self, code: str) -> str:
@ -812,7 +864,9 @@ class EnterpriseGatewayCodeExecutor:
async def __aexit__(self, exc_type, exc_val, exc_tb):
if self.kernel_id:
try:
async with self.session.delete(f"api/kernels/{self.kernel_id}", headers=self.headers) as response:
async with self.session.delete(
f"api/kernels/{self.kernel_id}", headers=self.headers
) as response:
response.raise_for_status()
logger.info(f"Closed kernel {self.kernel_id}")
except Exception as err:
@ -843,7 +897,7 @@ class EnterpriseGatewayCodeExecutor:
"env": {
"KERNEL_USERNAME": self.username,
"KERNEL_ID": str(uuid.uuid4()),
}
},
}
logger.info(f"Starting {self.kernel_name} kernel for user {self.username}")
@ -870,7 +924,9 @@ class EnterpriseGatewayCodeExecutor:
async def execute_code(self) -> None:
websocket_url, headers = self.init_ws()
try:
async with websockets.connect(websocket_url, additional_headers=headers) as ws:
async with websockets.connect(
websocket_url, additional_headers=headers
) as ws:
await self.execute_in_gateway(ws)
except websockets.exceptions.WebSocketException as e:
logger.error(f"WebSocket error: {e}")
@ -893,7 +949,7 @@ class EnterpriseGatewayCodeExecutor:
"msg_type": "execute_request",
"username": self.username,
"session": str(uuid.uuid4()),
"version": "5.4"
"version": "5.4",
},
"parent_header": {},
"metadata": {},
@ -903,10 +959,10 @@ class EnterpriseGatewayCodeExecutor:
"store_history": True,
"user_expressions": {},
"allow_stdin": False,
"stop_on_error": True
"stop_on_error": True,
},
"buffers": [],
"channel": "shell"
"channel": "shell",
}
logger.debug(f"Sending execute request with msg_id {msg_id}")
@ -947,7 +1003,9 @@ class EnterpriseGatewayCodeExecutor:
results.append(result_text)
logger.debug(f"Result text: {result_text}")
if "image/png" in response["content"]["data"]:
results.append(f"data:image/png;base64,{response['content']['data']['image/png']}")
results.append(
f"data:image/png;base64,{response['content']['data']['image/png']}"
)
logger.debug("Added image result")
elif msg_type == "display_data":
@ -958,29 +1016,35 @@ class EnterpriseGatewayCodeExecutor:
results.append(result_text)
logger.debug(f"Display text: {result_text}")
if "image/png" in response["content"]["data"]:
results.append(f"data:image/png;base64,{response['content']['data']['image/png']}")
results.append(
f"data:image/png;base64,{response['content']['data']['image/png']}"
)
logger.debug("Added image display")
elif msg_type == "error":
error = {
"ename": response["content"]["ename"],
"evalue": response["content"]["evalue"],
"traceback": response["content"]["traceback"]
"traceback": response["content"]["traceback"],
}
stderr_content += "\n".join(error["traceback"])
logger.debug(f"Execution error: {error}")
elif msg_type == "execute_reply":
logger.debug(f"Execute reply status: {response['content']['status']}")
logger.debug(
f"Execute reply status: {response['content']['status']}"
)
if response["content"]["status"] == "ok":
logger.debug("Received execute_reply with status=ok")
break
elif response["content"]["status"] == "error":
if not error: # Only add if we haven't already processed an error message
if (
not error
): # Only add if we haven't already processed an error message
error = {
"ename": response["content"]["ename"],
"evalue": response["content"]["evalue"],
"traceback": response["content"]["traceback"]
"traceback": response["content"]["traceback"],
}
stderr_content += "\n".join(error["traceback"])
logger.debug("Received execute_reply with status=error")
@ -996,9 +1060,15 @@ class EnterpriseGatewayCodeExecutor:
logger.warning(f"Execution timed out after {self.timeout}s")
break
self.result.stdout = self._prepare_results_with_path_replacement(stdout_content.strip())
self.result.stderr = self._prepare_results_with_path_replacement(stderr_content.strip())
self.result.result = self._prepare_results_with_path_replacement("\n".join(results).strip() if results else "")
self.result.stdout = self._prepare_results_with_path_replacement(
stdout_content.strip()
)
self.result.stderr = self._prepare_results_with_path_replacement(
stderr_content.strip()
)
self.result.result = self._prepare_results_with_path_replacement(
"\n".join(results).strip() if results else ""
)
logger.debug(f"Final result - stdout: {self.result.stdout}")
logger.debug(f"Final result - stderr: {self.result.stderr}")
@ -1015,6 +1085,7 @@ async def deprecated_execute_code_jupyter(
result = await executor.run()
return result.model_dump()
async def execute_code_jupyter(
base_url: str,
code: str,
@ -1022,7 +1093,7 @@ async def execute_code_jupyter(
password: str = "",
timeout: int = 60,
chat_id: str = "",
data_dir: str = "data"
data_dir: str = "data",
) -> dict:
async with EnterpriseGatewayCodeExecutor(
base_url, code, token, password, timeout, chat_id=chat_id, data_dir=data_dir
@ -1034,7 +1105,7 @@ async def execute_code_jupyter(
def generate_dynamic_code_interpreter_prompt(
base_prompt: str,
chat_id: str = "",
attached_files: Optional[List[Dict[str, Any]]] = None
attached_files: Optional[List[Dict[str, Any]]] = None,
) -> str:
"""
Generate a dynamic code interpreter prompt that includes information about attached files.
@ -1059,7 +1130,9 @@ def generate_dynamic_code_interpreter_prompt(
# Create file information section
file_info_lines = []
file_info_lines.append("\n#### Available Files")
file_info_lines.append("The following files have been attached to this conversation and are available in `/mnt/data/`:")
file_info_lines.append(
"The following files have been attached to this conversation and are available in `/mnt/data/`:"
)
file_info_lines.append("")
for file_info in attached_files:
@ -1077,30 +1150,48 @@ def generate_dynamic_code_interpreter_prompt(
else:
size_str = f" ({file_size / (1024 * 1024):.1f} MB)"
file_info_lines.append(f"- **{file_name}**{size_str} - Available at `/mnt/data/{file_name}`")
file_info_lines.append(
f"- **{file_name}**{size_str} - Available at `/mnt/data/{file_name}`"
)
# Add file type specific suggestions
if file_name.lower().endswith(('.csv', '.tsv')):
file_info_lines.append(f" - Data file - Use `pd.read_csv('/mnt/data/{file_name}')` to load")
elif file_name.lower().endswith(('.xlsx', '.xls')):
file_info_lines.append(f" - Excel file - Use `pd.read_excel('/mnt/data/{file_name}')` to load")
elif file_name.lower().endswith(('.json', '.jsonl')):
file_info_lines.append(f" - JSON file - Use `pd.read_json('/mnt/data/{file_name}')` or `json.load()` to load")
elif file_name.lower().endswith(('.txt', '.md', '.py', '.js', '.html', '.css')):
file_info_lines.append(f" - Text file - Use `open('/mnt/data/{file_name}', 'r').read()` to load")
elif file_name.lower().endswith(('.png', '.jpg', '.jpeg', '.gif', '.bmp', '.tiff')):
file_info_lines.append(f" - Image file - Use `PIL.Image.open('/mnt/data/{file_name}')` or `cv2.imread()` to load")
elif file_name.lower().endswith(('.pdf')):
file_info_lines.append(f" - PDF file - Use `PyPDF2` or `pdfplumber` to extract text/data")
if file_name.lower().endswith((".csv", ".tsv")):
file_info_lines.append(
f" - Data file - Use `pd.read_csv('/mnt/data/{file_name}')` to load"
)
elif file_name.lower().endswith((".xlsx", ".xls")):
file_info_lines.append(
f" - Excel file - Use `pd.read_excel('/mnt/data/{file_name}')` to load"
)
elif file_name.lower().endswith((".json", ".jsonl")):
file_info_lines.append(
f" - JSON file - Use `pd.read_json('/mnt/data/{file_name}')` or `json.load()` to load"
)
elif file_name.lower().endswith((".txt", ".md", ".py", ".js", ".html", ".css")):
file_info_lines.append(
f" - Text file - Use `open('/mnt/data/{file_name}', 'r').read()` to load"
)
elif file_name.lower().endswith(
(".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff")
):
file_info_lines.append(
f" - Image file - Use `PIL.Image.open('/mnt/data/{file_name}')` or `cv2.imread()` to load"
)
elif file_name.lower().endswith((".pdf")):
file_info_lines.append(
f" - PDF file - Use `PyPDF2` or `pdfplumber` to extract text/data"
)
file_info_lines.append("")
file_info_lines.append("**Important**: These files are immediately ready to use - no upload needed. Reference them directly by their paths above.")
file_info_lines.append(
"**Important**: These files are immediately ready to use - no upload needed. Reference them directly by their paths above."
)
# Insert file information after the main code interpreter description but before the final note
file_info_section = "\n".join(file_info_lines)
# Find a good insertion point in the base prompt
prompt_lines = base_prompt.split('\n')
prompt_lines = base_prompt.split("\n")
# Look for the line about /mnt/data and insert file info after it
insertion_point = -1
@ -1112,11 +1203,11 @@ def generate_dynamic_code_interpreter_prompt(
if insertion_point > 0:
# Insert file information after the /mnt/data line
enhanced_lines = (
prompt_lines[:insertion_point] +
file_info_section.split('\n') +
prompt_lines[insertion_point:]
prompt_lines[:insertion_point]
+ file_info_section.split("\n")
+ prompt_lines[insertion_point:]
)
return '\n'.join(enhanced_lines)
return "\n".join(enhanced_lines)
else:
# Fallback: append file information at the end
return base_prompt + "\n" + file_info_section

9
backend/open_webui/utils/middleware.py
View File

@ -81,7 +81,10 @@ from open_webui.utils.filter import (
get_sorted_filter_ids,
process_filter_functions,
)
from open_webui.utils.code_interpreter import execute_code_jupyter, generate_dynamic_code_interpreter_prompt
from open_webui.utils.code_interpreter import (
execute_code_jupyter,
generate_dynamic_code_interpreter_prompt,
)
from open_webui.tasks import create_task
@ -854,7 +857,7 @@ async def process_chat_payload(request, form_data, user, metadata, model):
enhanced_prompt = generate_dynamic_code_interpreter_prompt(
base_prompt=base_prompt,
attached_files=attached_files,
chat_id=metadata.get("chat_id", "")
chat_id=metadata.get("chat_id", ""),
)
form_data["messages"] = add_or_update_user_message(
@ -2278,7 +2281,7 @@ async def process_chat_response(
),
request.app.state.config.CODE_INTERPRETER_JUPYTER_TIMEOUT,
chat_id=metadata.get("chat_id", ""),
data_dir="data"
data_dir="data",
)
else:
output = {

4
package-lock.json generated
View File

@ -1,12 +1,12 @@
{
"name": "open-webui",
"version": "0.6.15",
"version": "0.6.15c",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "open-webui",
"version": "0.6.15",
"version": "0.6.15c",
"dependencies": {
"@azure/msal-browser": "^4.5.0",
"@codemirror/lang-javascript": "^6.2.2",

2
package.json
View File

@ -1,6 +1,6 @@
{
"name": "open-webui",
"version": "0.6.15",
"version": "0.6.15c",
"private": true,
"scripts": {
"dev": "npm run pyodide:fetch && vite dev --host",