Launch multiple instances of Ollama Engines

Configure and start first instance on default port

export OLLAMA_HOST="0.0.0.0:11434"

export OLLAMA_KEEP_ALIVE="18h"

export OLLAMA_NUM_PARALLEL="4"

export OLLAMA_CONTEXT_LENGTH="8192" 

ollama serve

Configure and start second instance on another port

export OLLAMA_HOST="0.0.0.0:11435"

export OLLAMA_MODELS="$HOME/.ollama/models-instance-2" 

export OLLAMA_KEEP_ALIVE="10m"

export OLLAMA_NUM_PARALLEL="2"

export OLLAMA_CONTEXT_LENGTH="8192" 

ollama serve

Configure Ollama client pointing to first instance

export OLLAMA_HOST="0.0.0.0:11434"

ollama list 

Configure Ollama client pointing to second instance

export OLLAMA_HOST="0.0.0.0:11435"

ollama list 

 


Comments

Post a Comment

Popular posts from this blog

Export/Backup Ollama Model

The following commands help to export/backup the ollama models which allows to quick restore the models or setup an offline machine using file transfer. Pull a model from registry ollama pull llava:7b Export the model mkdir -p llava_7b_backup cd llava_7b_backup ollama show --modelfile llava:7b > Modelfile cat Modelfile | grep -E '^FROM' | cut -d ' ' -f 2 | xargs -I {} cp {} . cat Modelfile | grep '# FROM' | cut -d ' ' -f 3 | xargs -I {} echo 'ollama create {} -f Modelfile' > readme.txt sed -i '' -E 's/^FROM.*blobs/FROM ./g' Modelfile Import the model  cd llava_7b_backup ollama create llava:7b -f Modelfile Run the model ollama run llava:7b Remove the model ollama rm llava:7b
Read more

Download and run Hugging Face AI models in Ollama

This article explains step by step guide to download an AI model from Hugging Face repository and run it with Ollama. Ollama allows to create a model locally from GGUF format or SafeTensors format and this article will cover both models. Creating from GGUF models is highly reliable as the GGUF file is packaged with model data with template file suitable for the AI model. Running a model using GGUF: Example repository with GGUF formal model https://huggingface.co/TheDrummer/Anubis-Mini-8B-v1-GGUF Create a download directory: mkdir thedrummer-anubis-mini-8b-v1-gguf Download the file into the directory: cd thedrummer-anubis-mini-8b-v1-gguf wget "https://huggingface.co/TheDrummer/Anubis-Mini-8B-v1-GGUF/resolve/main/Anubis-Mini-8B-v1h-Q4_K_M.gguf" -O ./thedrummer-anubis-mini-8b-v1-Q4_K_M.gguf Create an Ollama model: echo 'FROM ./thedrummer-anubis-mini-8b-v1-Q4_K_M.gguf' > Modelfile ollama create thedrummer-anubis-mini-8b-v1-gguf:latest Run and verify the model: curl htt...
Read more

Ubuntu (Linux) Desktop using RDP GUI with Windows WSL

Image
This article explains the steps to setup Ubuntu virtual machine with Desktop environment on Windows using Window's Subsystem for Linux (WSL) and be able to connect to Ubuntu using Window's builtin Remote Desktop Protocol (RDP) app. Youtube video A step by step visual guide of these steps is available in the below youtube link https://youtu.be/F6quxWDX7Hc Enable WSL on Windows: Enabling the features can be done only using administrator user. In corporate laptop or enterprises, this step is performed by Windows Administrator. Through GUI: Start -> Search -> Turn Windows feature on or off Virtual Machine Platform -> Tick Windows Hypervisor Platform -> Tick Window Subsystem for Linux -> Tick Click OK Wait for some installations to complete Restart Windows Through Terminal: dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart dism /online /enable-feature /featurename:HypervisorPlatform /all /norestart dism.exe /online /enable-feature /f...
Read more