Download and run Hugging Face AI models in Ollama

This article explains step by step guide to download an AI model from Hugging Face repository and run it with Ollama.

Ollama allows to create a model locally from GGUF format or SafeTensors format and this article will cover both models.

Creating from GGUF models is highly reliable as the GGUF file is packaged with model data with template file suitable for the AI model.

Running a model using GGUF:

Example repository with GGUF formal model https://huggingface.co/sergbese/gemma-3-isv-translator-v5-gguf-bf16

Create a download directory:

mkdir gemma-3-isv-translator-v5-gguf-bf16

Download the file into the directory

cd gemma-3-isv-translator-v5-gguf-bf16

wget "https://huggingface.co/sergbese/gemma-3-isv-translator-v5-gguf-bf16/resolve/main/gemma-3-finetune-2.BF16.gguf?download=true" -O ./gemma-3-finetune-2.BF16.gguf

Create an Ollama model:

echo 'FROM ./gemma-3-finetune-2.BF16.gguf' > Modelfile

ollama create gemma-3-isv-translator-v5-gguf-bf16:latest

Run and verify the model:

curl http://localhost:11434/api/chat -d '{ "model": "gemma-3-isv-translator-v5-gguf-bf16:latest", "stream": false, "messages": [ { "role": "user", "content": "Translate \"How old are you?\" from English to Tamil" } ] }' | jq .

Running a model using SafeTensors:

In order to create a model from SafeTensors format, we need to download / clone the entire hugging face repository.

Hugging face repository contains large files with size more than 1 GB, hence we need tools like git and git lfs to clone the repository.

Example repository containing SafeTensors models

https://huggingface.co/sergbese/gemma-3-isv-translator-v5

Install git

https://git-scm.com/

Install git lfs

https://git-lfs.com/

Configure git with lfs

git lfs install

Clone the hugging face repository

git clone https://huggingface.co/sergbese/gemma-3-isv-translator-v5

The clone will take longer to complete and it will depend on the internet speed and repository size. Git doesn't show progress of the download, hence the following command will continuously print the directory size and show progress during the download.

cd gemma-3-isv-translator-v5 && while :; do du -sh; sleep 2; done

Create a Ollama model from the cloned repository:

cd gemma-3-isv-translator-v5

echo 'FROM .' > Modelfile

ollama create gemma-3-isv-translator-v5:latest

Run and verify the Ollama Model:

curl http://localhost:11434/api/chat -d '{ "model": "gemma-3-isv-translator-v5:latest", "stream": false, "messages": [ { "role": "user", "content": "Translate \"How old are you?\" from English to Tamil" } ] }' | jq .

Unlike GGUF format, SafeTensors format doesn't contain the built in template for the model. Hence it is a good idea to create the model file along with a template based on the documentation of the base model.

Reference of Ollama Modelfile:

https://github.com/ollama/ollama/blob/main/docs/modelfile.md

Here is a sample Modelfile for base gemma 3 AI Model:

ollama show --modelfile gemma3:4b

FROM .

TEMPLATE """{{- range $i, $_ := .Messages }}

{{- $last := eq (len (slice $.Messages $i)) 1 }}

{{- if or (eq .Role "user") (eq .Role "system") }}<start_of_turn>user

{{ .Content }}<end_of_turn>

{{ if $last }}<start_of_turn>model

{{ end }}

{{- else if eq .Role "assistant" }}<start_of_turn>model

{{ .Content }}{{ if not $last }}<end_of_turn>

{{ end }}

{{- end }}

{{- end }}"""

PARAMETER stop <end_of_turn>

PARAMETER temperature 1

PARAMETER top_k 64

PARAMETER top_p 0.95

Comments

Post a Comment

Popular posts from this blog

JSON with curl and jq

This blog post captures some quick commands to deal with JSON from servers (Http APIs) using curl and jq command line tools About Jq: https://stedolan.github.io/jq/ Install jq: Mac: brew install jq Ubuntu or Debian: sudo apt-get install jq Sample JSON content: Assume the following JSON content will be returned from  URL http://api.example.com/some-json-content {     "user_name": "User 123",     "some_other_field": {         "sub_field": "sub_value"     },     "array_field": [         {             "field1": "value1",             "field2": "value2"         },         {             "field1": "value3",             "field2": "value4"         }     ] } UR...
Read more

Import self signed in Linux for Chrome / Chromium headless testing

Chrome / Chromium browser offers headless mode, which is quite useful to run tests via Selenium WebDriver in CI servers. However under some circumstance your test will hit a server with self signed certificate or invalid certificate or expired certificate. This may happen for developers trying to write tests against a server started on localhost with self signed certificates or testers trying to write tests against non-production servers with expired certificates (invalid certificates). Chrome / Chromium browser ignores the SSL errors when running in non-headless mode. However in CI servers, then it should be running in headless mode and currently we cannot ignore the SSL errors for tests accessing non-localhost websites. This blogs explains the steps for import a certificate into Linux's trust store, which will make the Chrome / Chromium browser to trust the self signed certificate.  Please do not apply these steps in production servers as it will make server vulnerable to ...
Read more

Colima - Drop In replacement for Docker Desktop for Mac and Linux

Colima ( Co ntainer on Li nux Ma chines) is an open source alternative and drop in replacement for Docker Desktop on Mac and Linux. https://github.com/abiosoft/colima It is simple to setup and run containers without the need for sudo or root access. Images can be pulled form both Docker hub ( https://hub.docker.com ) or Amazon's public registry ( https://gallery.ecr.aws ) Amazon's registry is more permissive or friendly with pull rates for both clients and non-clients of AWS. This article show the setup and basics of running containers on mac. Installation command for Linux can vary based on the distribution type, hence please refer to official documentation for up-to-date steps at  https://github.com/abiosoft/colima/blob/main/docs/INSTALL.md Installation on mac: brew install docker  brew install docker-buildx brew install docker-compose brew install jq brew install colima Setup docker compose as docker plugin mkdir -p $HOME/.docker/cli-plugins ln -sfn $(which docker-compose) ...
Read more