GITBOOK-28: No subject

docs/developer-guide/hexabot-api.md

@@ -0,0 +1,137 @@
+# Hexabot API
+
+## Hexabot API
+
+[Hexabot](https://hexabot.ai/)'s API is a RESTful API built with NestJS, designed to handle requests from both the UI admin panel and various communication channels. The API powers core functionalities such as chatbot management, message flow, NLP (Natural Language Processing), and plugin integrations.
+
+### Key Features
+
+* **RESTful Architecture:** Simple, standardized API architecture following REST principles.
+* **Multi-Channel Support:** Handles requests from different communication channels (e.g., web, mobile).
+* **Modular Design:** Organized into multiple modules for better scalability and maintainability.
+* **Real-Time Communication:** Integrates WebSocket support for real-time features.
+
+### API Modules
+
+The API is divided into several key modules, each responsible for specific functionalities:
+
+#### Core Modules
+
+* **Analytics:** Tracks and serves analytics data such as the number of messages exchanged and end-user retention statistics.
+* **Attachment:** Manages file uploads and downloads, enabling attachment handling across the chatbot.
+* **Channel:** Manages different communication channels through which the chatbot operates (e.g., web, mobile apps, etc.).
+* **Chat:** The core module for handling incoming channel requests and managing the chat flow as defined by the visual editor in the UI.
+* **CMS:** Content management module for defining content types, managing content, and configuring menus for chatbot interactions.
+* **NLP:** Manages NLP entities such as intents, languages, and values used to detect and process user inputs intelligently.
+* **Plugins:** Manages extensions and plugins that integrate additional features and functionalities into the chatbot.
+* **User:** Manages user authentication, roles, and permissions, ensuring secure access to different parts of the system.
+* **Extensions:** A container for all types of extensions (channels, plugins, helpers) that can be added to expand the chatbot's functionality.
+* **Settings:** A module for management all types of settings that can be adjusted to customize the chatbot.
+
+#### Utility Modules
+
+* **WebSocket:** Adds support for Websicket with Socket.IO, enabling real-time communication for events like live chat and user interactions.
+* **Logger:** Provides logging functionality to track and debug API requests and events.
+
+### Installation
+
+```bash
+$ npm install
+```
+
+### Running the app in standalone
+
+```bash
+# development
+$ npm run start
+
+# watch mode
+$ npm run start:dev
+
+# production mode
+$ npm run start:prod
+```
+
+### Test
+
+```bash
+# unit tests
+$ npm run test
+
+# e2e tests
+$ npm run test:e2e
+
+# test coverage
+$ npm run test:cov
+```
+
+### Migrations
+
+The API includes a migrations feature to help manage database schema and data changes over time. Migrations allow you to apply or revert changes to the database in a consistent and controlled manner.
+
+#### Creating a Migration
+
+You need to navigate to the `api` folder to run the following commands. To create a new migration, use the following command:
+
+```bash
+$ npm run create-migration <migration-name>
+```
+
+Example:
+
+```bash
+$ npm run create-migration all-users-language-fr
+```
+
+This command generates a new migration file in the `./migrations` folder. The file will look like this:
+
+```typescript
+import getModels from '@/models/index';
+
+export async function up(): Promise<void> {
+  // Write migration here
+}
+
+export async function down(): Promise<void> {
+  // Write migration here
+}
+```
+
+Within the migration file, you can define the changes to be made in the up() function. For example, if you want to update the language field of all users to 'fr', your migration might look like this:
+
+```typescript
+import getModels from '@/models/index';
+
+export async function up(): Promise<void> {
+  const { UserModel } = await getModels();
+  await UserModel.updateMany({}, { language: 'fr' });
+}
+
+export async function down(): Promise<void> {}
+```
+
+#### Running Migrations Up
+
+All migrations are run automatically when the app starts.
+
+Alternatively, you can run the following command in the `root` directory to run all pending migrations:
+
+```bash
+$ make migrate-up
+```
+
+#### Running Migrations Manually
+
+If you want to run specific actions manually, you need to gain access to the `api` container and use the following command to run what you specifically want:
+
+```bash
+$ npm run migrate -h
+```
+
+### Documentation
+
+Access the Swagger API documentation by visiting the API url `/docs` once run it in development mode.
+
+It's also possible to access the API reference documentation by running `npm run doc`.
+
+For detailed information about the API routes and usage, refer to the API documentation or visit [https://docs.hexabot.ai](https://docs.hexabot.ai).

docs/developer-guide/hexabot-live-chat-widget.md

@@ -0,0 +1,82 @@
+# Hexabot Live Chat Widget
+
+The [Hexabot](https://hexabot.ai/) Live Chat Widget is a React-based embeddable widget that allows users to integrate real-time chat functionality into their websites. It connects to the Hexabot API and facilitates seamless interaction between end-users and chatbots across multiple channels.
+
+### Key Features
+
+* **Real-Time Chat:** Engage in real-time conversations with users directly through your website.
+* **Customizable:** Easily customize the widget's appearance and behavior to fit your brand and website.
+* **Multi-Channel Support:** Integrates with multiple messaging platforms through the Hexabot API.
+* **Embeddable:** Simple to embed and integrate into any web page with just a few lines of code.
+
+### Directory Structure
+
+The Hexabot Live Chat Widget is organized into the following directory structure, under `src` we have:
+
+* **src/assets:** Static assets like icons, fonts, and images used in the widget.
+* **src/components:** Reusable React components that make up the chat widget interface, such as message bubbles, input fields, and buttons.
+* **src/constants:** Hard coded values that are used like colors.
+* **src/hooks:** Custom React hooks for managing widget state and handling side effects like API calls or real-time events.
+* **src/services:** Handles external services, such as communication with the Hexabot API or other third-party integrations.
+* **src/styles:** Contains the styling for the widget, including CSS or SCSS files used to define the look and feel of the chat interface.
+* **src/providers:** Context providers for managing global state, such as user session, chat messages, and widget configurations.
+* **src/translations:** Contains transalation of a couple of strings.
+* **src/types:** Defines the typescript interfaces, types, and enums used.
+* **src/utils:** Utility functions and helpers used throughout the widget, such as formatting, validations, or data transformations.
+* **/public:** Contains static files that are publicly accessible. This includes the main HTML template where the widget is embedded for local development.
+
+### Run the Live Chat Widget
+
+#### Dev Mode
+
+To run the widget in development mode, execute the following command at the project root level:
+
+```bash
+npm run dev:widget
+```
+
+The live chat widget will be accessible at http://localhost:5173.
+
+#### Build for Production
+
+To build the widget for production, execute the following command at the widget folder level:
+
+```bash
+npm run build
+```
+
+This will generate a production-ready build in the dist folder.
+
+### Embed Chat Widget
+
+Once the widget is built, you can easily embed it into any webpage. Here's an example of how to add it to your website:
+
+```js
+<script crossorigin src="https://unpkg.com/react@18/umd/react.production.min.js"></script>
+<script crossorigin src="https://unpkg.com/react-dom@18/umd/react-dom.production.min.js"></script>
+<link rel="stylesheet" href="./style.css">
+<script src="<<WIDGET URL>>/hexabot-widget.umd.js"></script>
+
+<div id="hb-chat-widget"></div>
+<script>
+  const el = React.createElement;
+  const domContainer = document.getElementById('hb-chat-widget');
+  ReactDOM.render(
+    el(HexabotWidget, {
+      apiUrl: 'https://api.yourdomain.com',
+      channel: 'offline',
+      token: 'token123',
+    }),
+    domContainer,
+  );
+</script>
+```
+
+Replace the values in apiUrl and token with your configuration details.
+
+### Customization
+
+You can customize the look and feel of the chat widget by modifying the widget’s scss styles or behavior. The widget allows you to:
+
+* Change colors and fonts to match your website's branding.
+* Configure user settings like language and chatbot response preferences.

docs/developer-guide/hexabot-nlu-engine.md

@@ -0,0 +1,163 @@
+# Hexabot NLU Engine
+
+The [Hexabot](https://hexabot.ai/) NLU (Natural Language Understanding) engine is a Python-based project that provides tools for building, training, and evaluating machine learning models for natural language tasks such as intent detection and language recognition. It also includes a REST API for inference, built using FastAPI.
+
+### Directory Structure
+
+* **/run.py:** The CLI tool that provides commands for training, evaluating, and managing models.
+* **/models:** Contains the different model definitions and logic for training, testing, and evaluation.
+* **/data:** Placeholder for datasets used during training and evaluation.
+* **/experiments:** Placeholder for stored models generated during training.
+* **/data\_loaders:** Classes that define the way to load datasets to be used by the different models.
+* **/main.py:** The FastAPI-based REST API used for inference, exposing endpoints for real-time predictions.
+
+### Setup
+
+**No dependencies needed besides Python 3.11.6, virtualenv, and TensorFlow.** Start developing your new model on top of this workflow by cloning this repository:
+
+```bash
+# Set up a virtualenv
+pip install virtualenv
+
+python3.11 -m venv venv
+
+source env.sh
+
+pip install -r requirements.txt
+```
+
+### Directory structure
+
+* `data`: gitignore'd, place datasets here.
+* `experiments`: gitignore'd, trained models written here.
+* `data_loaders`: write your data loaders here.
+* `models`: write your models here.
+
+### Usage
+
+**Check `models/mlp.py` and `data_loaders/mnist.py` for fully working examples.**
+
+You should run `source env.sh` on each new shell session. This activates the virtualenv and creates a nice alias for `run.py`:
+
+```bash
+$ cat env.sh
+source env/bin/activate
+alias run='python run.py'
+```
+
+Most routines involve running a command like this:
+
+```bash
+# Usage: run [method] [save_dir] [model] [data_loader] [hparams...]
+run fit myexperiment1 mlp mnist --batch_size=32 --learning_rate=0.1
+```
+
+Examples :
+
+```bash
+# Intent classification
+run fit intent-classifier-en-30072024 jisf  --intent_num_labels=88 --slot_num_labels=17 --language=en
+run predict intent-classifier-fr-30072024  --intent_num_labels=7 --slot_num_labels=2 --language=fr
+
+# Language classification
+run fit language-classifier-26082023 tflc
+run predict language-classifier-26082023
+run evaluate language-classifier-26082023
+```
+
+where the `model` and `data_loader` args are the module names (i.e., the file names without the `.py`). The command above would run the Keras model's `fit` method, but it could be any custom as long as it accepts a data loader instance as argument.
+
+**If `save_dir` already has a model**:
+
+* Only the first two arguments are required and the data loader may be changed, but respecifying the model is not allowed-- the existing model will always be used.
+* Specified hyperparameter values in the command line WILL override previously used ones (for this run only, not on disk).
+
+#### `tfbp.Model`
+
+Models pretty much follow the same rules as Keras models with very slight differences: the constructor's arguments should not be overriden (since the boilerplate code handles instantiation), and the `save` and `restore` methods don't need any arguments.
+
+```python
+import tensorflow as tf
+import boilerplate as tfbp
+
+@tfbp.default_export
+class MyModel(tfbp.Model):
+    default_hparams = {
+        "batch_size": 32,
+        "hidden_size": 512,
+        "learning_rate": 0.01,
+    }
+
+    # Don't mess with the args and keyword args, `run.py` handles that.
+    def __init__(self, *a, **kw):
+        super().__init__(*a, **kw)
+
+        self.dense1 = tf.keras.layers.Dense(self.hparams.hidden_size)
+        ...
+
+    def call(self, x):
+        z = self.dense1(x)
+        ...
+```
+
+You can also write your own training loops à la pytorch by overriding the `fit` method or writing a custom method that you can invoke via `run.py` simply by adding the `@tfbp.runnable` decorator. Examples of both are available in `models/mlp.py`.
+
+#### `tfbp.DataLoader`
+
+Since model methods invoked by `run.py` receive a data loader instance, you may name your data loader methods whatever you wish and call them in your model code. A good practice is to make the data loader handle anything that is specific to a particular dataset, which allows the model to be as general as possible.
+
+```python
+import tensorflow as tf
+import boilerplate as tfbp
+
+@tfbp.default_export
+class MyDataLoader(tfbp.DataLoader):
+    default_hparams = {
+        "batch_size": 32,
+    }
+
+    def __call__(self):
+        if self.method == "fit":
+            train_data = tf.data.TextLineDataset("data/train.txt").shuffle(10000)
+            valid_data = tf.data.TextLineDataset("data/valid.txt").shuffle(10000)
+            return self.prep_dataset(train_data), self.prep_dataset(valid_data)
+
+        elif self.method == "eval":
+            test_data = tf.data.TextLineDataset("data/test.txt")
+            return self.prep_dataset(test_data)
+
+    def prep_dataset(self, ds):
+        return ds.batch(self.hparams.batch_size).prefetch(1)
+```
+
+#### API
+
+API is built using FastAPI : https://fastapi.tiangolo.com/
+
+Run the dev server in standalone with:
+
+```sh
+ENVIRONMENT=dev uvicorn main:app --host 0.0.0.0 --port 5000 --reload
+```
+
+Run the project with Docker :
+
+```sh
+docker compose -f "docker-compose.yml" up -d --build
+```
+
+### Pushing models to HuggingFace
+
+Please refer to official HF documentation on how to host models : https://huggingface.co/docs/hub/en/repositories-getting-started
+
+What is important to note is that big files should be tracked with git-lfs, which you can initialize with:
+
+```
+git lfs install
+```
+
+and if your files are larger than 5GB you’ll also need to run:
+
+```
+huggingface-cli lfs-enable-largefiles .
+```

docs/developer-guide/hexabot-ui-admin-panel.md

@@ -0,0 +1,48 @@
+# Hexabot UI Admin Panel
+
+The [Hexabot](https://hexabot.ai/) UI Admin Panel is a React/Next.js application that serves as the admin interface for managing chatbot configurations, workflows, and interactions. The admin panel allows users to easily create and manage chat flows, monitor analytics, manage content, handle NLP datasets, and configure system settings.
+
+### Key Features
+
+* **Visual Editor:** An intuitive drag-and-drop interface for managing chat flows, including text messages, quick replies, carousels, and more.
+* **Multi-Channel Management:** Configure and manage multiple communication channels (e.g., web, mobile, social media) from a single interface.
+* **Analytics Dashboard:** Track user interactions, messages sent, and retention rates through detailed analytics.
+* **NLP Management:** Manage datasets for training machine learning models for intent detection and language recognition.
+* **Content Management (CMS):** Seamlessly integrate and manage content such as product catalogs, lists of stores, or any dynamic content needed by the chatbot.
+* **User, Roles, and Permissions:** Administer user access controls, roles, and permissions to ensure secure and appropriate access to different parts of the system.
+
+### Directory Structure
+
+* **app-components/:** Reusable components that are used across the admin panel.
+* **components/:** Components organized by the page where they are being used.
+* **hooks/:** Hooks defined.
+* **pages/:** The core of the Next.js application, defining different routes for the admin panel.
+* **services/:** API service calls to interact with the Hexabot API.
+* **types/:** Defines the typescript interfaces, types, and enums used.
+* **styles/:** Global and component-specific styles for the application.
+* **utils/:** Utility functions and helpers used throughout the frontend.
+
+### Run
+
+The Hexabot project is structured as a monorepo using npm workspaces to manage multiple packages. The packages in the monorepo include:
+
+* **frontend:** The admin interface for managing the chatbot system.
+* **widget:** An embeddable live chat widget that can be integrated into any website. Note: The API is not part of the monorepo and is managed separately.
+
+To run the different UI components in development mode, execute the following commands at the project root level:
+
+* **Run the Admin Interface:**
+
+```
+npm run dev:frontend
+```
+
+The admin interface will be accessible at [http://localhost:8081](http://localhost:8081).
+
+* **Run the Live Chat Widget:**
+
+```
+npm run dev:widget
+```
+
+The live chat widget will be accessible at [http://localhost:5173](http://localhost:5173).

docs/developer-guide/install-hexabot.md

@@ -0,0 +1,65 @@
+# Install Hexabot
+
+## Prerequisites
+
+To ensure Hexabot runs smoothly, you'll need the following:
+
+* **Docker:** We recommend using Docker to start the app since multiple services are required (MongoDB, Redis, Prometheus, etc.). All the necessary Docker Compose files are located in the docker folder.
+* **Node.js:** For development purposes, ensure you have Node.js >= v18.17.0 installed. We recommend using nvm (Node Version Manager) to easily manage and update your Node.js versions.
+
+### # Installation
+
+1. **Clone the Repository:**
+
+```
+$ git clone https://github.com/hexastack/hexabot.git
+```
+
+2. **Environment Setup:** \
+   To configure the environment variables, use the Makefile at the root folder for initialization:
+
+```
+$ make init
+```
+
+This will copy the `.env.example` file to `.env` in the `./docker` directory if the file does not already exist.
+
+3. **Running the Application:** Once your environment is set up, you can start the app. Use either of the following commands:
+
+```
+$ make start
+```
+
+or for development mode:
+
+```
+$ make dev
+```
+
+**Note:** The first time you run the app, Docker will take some time to download all the required images.
+
+### Usage
+
+UI Admin Panel is accessible via [http://localhost:8080](http://localhost:8080), the default credentials are :
+
+* **Username:** [admin@admin.admin](mailto:admin@admin.admin)
+* **Password:** adminadmin
+
+Live Chat Widget is accessible via [http://localhost:5173](http://localhost:5173)
+
+
+
+### Useful Commands
+
+* `make init` : Copies the .env.example file to .env in the ./docker directory if .env does not exist. This is usually used for initial setup.
+* `make dev` : Starts all configured Docker services in development mode. It first checks the .env file for completeness against .env.example.
+* `make start` : Similar to dev, but explicitly builds the Docker images before starting the services. This target also checks the .env file for required variables.
+* `make stop` : Stops all running Docker services defined in the compose files.
+* `make destroy` : Stops all services and removes all volumes associated with the Docker compose setup, ensuring a clean state.
+* `make check-env` : Checks if the ./docker/.env file exists and contains all the necessary environment variables as defined in ./docker/.env.example. If the file does not exist, it is created from the example. It also lists missing variables if any.
+
+Example on how to start the stack by adding the Nginx service :
+
+```
+make start NGINX=1
+```

docs/developer-guide/integrations/README.md

@@ -0,0 +1,9 @@
+# Plugins
+
+Hexabot plugins allow you to connect your Hexabot instance to some of your favorite platforms and services. You can install integrations into your Hexabot from the _Integrations Marketplace_ menu in the top left.
+
+<figure><img src="https://gitbookio.github.io/onboarding-template-images/integrations-hero.png" alt=""><figcaption></figcaption></figure>
+
+### Types of integrations
+
+<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="files"></th><th data-hidden></th></tr></thead><tbody><tr><td><strong>Analytics</strong></td><td>Track analytics from your docs</td><td><a href="https://www.gitbook.com/integrations#analytics">https://www.gitbook.com/integrations#analytics</a></td><td><a href="../../.gitbook/assets/2.png">2.png</a></td><td></td></tr><tr><td><strong>Support</strong></td><td>Add support widgets to your docs</td><td><a href="https://www.gitbook.com/integrations#support">https://www.gitbook.com/integrations#support</a></td><td><a href="../../.gitbook/assets/3.png">3.png</a></td><td></td></tr><tr><td><strong>Interactive</strong></td><td>Add extra functionality to your docs</td><td><a href="https://www.gitbook.com/integrations#interactive">https://www.gitbook.com/integrations#interactive</a></td><td><a href="../../.gitbook/assets/4.png">4.png</a></td><td></td></tr><tr><td><strong>Visitor Authentication</strong></td><td>Protect your docs and require sign-in</td><td><a href="https://www.gitbook.com/integrations#visitor-authentication">https://www.gitbook.com/integrations#visitor-authentication</a></td><td><a href="../../.gitbook/assets/1.png">1.png</a></td><td></td></tr></tbody></table>

docs/developer-guide/integrations/build-your-first-plugin.md

@@ -0,0 +1,6 @@
+# Build Your First plugin
+
+{% hint style="info" %}
+This section is currently under development (WIP)
+{% endhint %}
+

docs/developer-guide/integrations/openapi.md

@@ -0,0 +1,6 @@
+# RAG Plugin
+
+{% hint style="info" %}
+This section is currently under development (WIP)
+{% endhint %}
+