> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Ollama

> Run local AI models with Ollama

## Overview

Ollama allows you to run open-source AI models locally on your own infrastructure. Perfect for privacy-sensitive applications, development, and cost optimization.

## Available Models

> [card] Llama 3.3 (70B) Local
>
> **0 credits** • Local 70B Llama model. Powerful performance for self-hosted deployments.
>
> * 128K context window
> * Excellent reasoning
> * Speed: Medium • Cost: Very\_low
> * Model id: `llama3.3:70b`

> [card] Llama 3.1 (8B) Local
>
> **0 credits** • Efficient local 8B model. Good for basic tasks with local processing.
>
> * 128K context window
> * Good reasoning
> * Speed: Fast • Cost: Very\_low
> * Model id: `llama3.1:8b`

> [card] Llama 3.1 (70B) Local
>
> **0 credits** • Local 70B model. Strong performance for self-hosted applications.
>
> * 128K context window
> * Excellent reasoning
> * Speed: Medium • Cost: Very\_low
> * Model id: `llama3.1:70b`

> [card] Llama 3.1 (405B) Local
>
> **0 credits** • Massive local model. Exceptional capabilities for self-hosted solutions.
>
> * 128K context window
> * Exceptional reasoning
> * Speed: Slow • Cost: Very\_low
> * Model id: `llama3.1:405b`

## Setup

### Prerequisites

**1. Install Ollama**

Download and install Ollama from [ollama.com](https://ollama.com)

```bash theme={null}
# macOS/Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download installer from ollama.com
```

**2. Pull a Model**

```bash theme={null}
# Pull Llama 3.1 8B (recommended for testing)
ollama pull llama3.1:8b

# Or pull Llama 3.3 70B for production
ollama pull llama3.3:70b
```

**3. Start Ollama Server**

```bash theme={null}
# Ollama runs on localhost:11434 by default
ollama serve
```

### Configure BoostGPT

> [tab] Dashboard Setup
>
> **1. Navigate to Integrations**
>
> Go to [app.boostgpt.co](https://app.boostgpt.co/integrations) and select **Integrations**
>
> **2. Select Ollama**
>
> Find and click on the **Ollama** provider
>
> **3. Configure Host**
>
> Enter your Ollama host URL (default: `http://localhost:11434`)
>
> Select which AI teammates will use Ollama
>
> **4. Save Configuration**
>
> Click save to apply your Ollama configuration

> [tab] Core SDK
>
> ```javascript bot.js theme={null}
> import { BoostGPT } from 'boostgpt';
>
> const client = new BoostGPT({
>   project_id: process.env.BOOSTGPT_PROJECT_ID,
>   key: process.env.BOOSTGPT_API_KEY
> });
>
> // Create bot with Ollama model
> const botResponse = await client.createBot({
>   name: 'My Local Ollama Bot',
>   model: 'llama3.1:8b', // Must match pulled model
>   instruction: 'You are a helpful local assistant.',
>   max_reply_tokens: 1000,
>   status: 'active'
> });
>
> if (botResponse.err) {
>   console.error('Error:', botResponse.err);
> } else {
>   console.log('Bot created:', botResponse.response);
> }
>
> // Chat with Ollama bot using provider_host
> const chatResponse = await client.chat({
>   bot_id: botResponse.response.id,
>   provider_host: 'http://localhost:11434', // Specify Ollama host
>   message: 'Hello, how are you?'
> });
>
> if (chatResponse.err) {
>   console.error('Error:', chatResponse.err);
> } else {
>   console.log('Response:', chatResponse.response.chat.reply);
> }
> ```

> [tab] Router SDK
>
> ```javascript router.js theme={null}
> import { Router, WhatsAppAdapter } from '@boostgpt/router';
>
> const router = new Router({
>   apiKey: process.env.BOOSTGPT_API_KEY,
>   projectId: process.env.BOOSTGPT_PROJECT_ID,
>   defaultBotId: process.env.BOOSTGPT_BOT_ID, // Ollama bot
>   adapters: [
>     new WhatsAppAdapter({
>       useLocalAuth: true
>     })
>   ]
> });
>
> // Router uses your local Ollama bot
> router.onMessage(async (message, context) => {
>   // Handle commands
>   if (message.content === '/status') {
>     return 'Running locally with Ollama!';
>   }
>
>   return null; // Ollama handles it
> });
>
> await router.start();
> ```

## Hardware Requirements

| Model | Min VRAM | Recommended RAM | CPU Cores | Best Hardware |
| - | - | - | - | - |
| **Llama 3.1 (8B)** | 8GB | 16GB | 4+ | Gaming PC, M1 Mac |
| **Llama 3.1 (70B)** | 40GB | 64GB | 8+ | Workstation, A100 |
| **Llama 3.3 (70B)** | 40GB | 64GB | 8+ | Workstation, A100 |
| **Llama 3.1 (405B)** | 200GB+ | 256GB+ | 16+ | Multi-GPU server |

> [!TIP]
>
> Start with Llama 3.1 (8B) for development and testing. It runs well on consumer hardware and M-series Macs.

## Best Practices

### Using provider\_host for Ollama

When using the Core SDK chat method, specify the Ollama host with `provider_host`:

```javascript theme={null}
// Chat with local Ollama instance
const chatResponse = await client.chat({
  bot_id: 'your-bot-id',
  provider_host: 'http://localhost:11434', // Required for Ollama
  message: 'Analyze this code for bugs'
});

// Use custom Ollama host (e.g., remote server)
const remoteResponse = await client.chat({
  bot_id: 'your-bot-id',
  provider_host: 'http://192.168.1.100:11434', // Custom host
  message: 'Hello!'
});
```

> [!NOTE]
>
> The `provider_host` parameter is required when using Ollama with the Core SDK chat method, as it tells BoostGPT where your Ollama instance is running.

### Model Selection for Hardware

```bash theme={null}
# For MacBook Pro M1/M2 (16GB RAM)
ollama pull llama3.1:8b

# For Workstation with RTX 4090 (24GB VRAM)
ollama pull llama3.3:70b

# For Server with A100 (80GB VRAM)
ollama pull llama3.1:405b
```

### Production Deployment

**1. Use Docker**

```bash theme={null}
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
```

**2. Configure Firewall**

Ensure Ollama port (11434) is accessible to your BoostGPT application

**3. Monitor Resources**

Watch GPU/CPU usage and scale hardware as needed

**4. Set Up Load Balancing**

For high volume, run multiple Ollama instances behind a load balancer

## Troubleshooting

> [accordion] Connection refused
>
> **Cause:** Ollama server not running or firewall blocking
>
> **Solutions:**
>
> * Run `ollama serve` to start server
> * Check firewall allows port 11434
> * Verify host URL in configuration

> [accordion] Model not found
>
> **Cause:** Model not pulled locally
>
> **Solutions:**
>
> * Run `ollama pull <model-name>`
> * Verify model name matches exactly
> * Check `ollama list` for available models

> [accordion] Slow responses
>
> **Cause:** Insufficient hardware resources
>
> **Solutions:**
>
> * Use smaller model (8B instead of 70B)
> * Add more RAM/VRAM
> * Reduce max\_reply\_tokens
> * Close other GPU-intensive applications

> [accordion] Out of memory errors
>
> **Cause:** Model too large for available VRAM
>
> **Solutions:**
>
> * Switch to smaller model
> * Reduce context window
> * Use CPU fallback (slower but works)
> * Upgrade hardware

## Next Steps

> [card] [Model Comparison](/docs/providers/model-comparison.md)
>
> Compare Ollama with cloud providers

> [card] [Deployment Guide](/docs/developers/deployment.md)
>
> Learn about production deployment

> [card] [SDK Reference](/docs/sdk/core/api-reference.md)
>
> Full API documentation

> [card] [Provider Overview](/docs/providers/overview.md)
>
> See all providers


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.