Running Cisco Antares-1B Locally (macOS / Apple Silicon)

A working start-to-finish guide. Tested on an M-series MacBook Pro, August 2026.

Read this first: there is one non-obvious step that will waste your afternoon if you miss it — you must convert the weights at BF16, not F16. Details in step 6 and in Troubleshooting. Everything else is routine.

What Antares-1B actually is

An open-weight 1B model from Cisco Foundation AI, built on IBM Granite 4.0 1B, specialized for vulnerability localization. Given a CWE description and a repository, it explores the codebase through a terminal — issuing grep, find, cat — and submits a list of files it believes contain the vulnerability.

Two things to internalize before you start:

It is not a chat assistant. The model card explicitly lists general-purpose chat and instruction following as out-of-scope. Asking it “what are your capabilities” gets you nothing useful even when the setup is perfect.
It is one half of an agent loop. It emits <tool_call> JSON; your harness executes the command and feeds the output back as <tool_response>. It runs up to 15 terminal calls, then terminates via submit_vulnerable_files or submit_no_vulnerability_found.

License is Apache 2.0. The Hugging Face repo is gated behind a contact-info form. Expect ~100 tok/s on Apple silicon. Benchmark is File F1 0.209 on Cisco’s VLoc Bench — useful for triage, not a scanner replacement.

Prerequisites

macOS on Apple Silicon
Homebrew
~15 GB free disk (4 GB weights, 3.7 GB GGUF, plus PyTorch)
A Hugging Face account

1. Accept the model gate

Go to https://huggingface.co/fdtn-ai/antares-1b and complete the access form. Downloads 403 without this. Approval is manual but was quick in practice.

2. Create a Hugging Face token

https://huggingface.co/settings/tokens — read access is enough.

Treat it like a password. If you ever paste one into a chat, ticket, or screen share, revoke it on that page immediately and issue a new one.

3. Install tooling

brew install llama.cpp

This gives you prebuilt llama-cli and llama-server — no compiler needed. You still need the llama.cpp source for the conversion script:

git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp

4. Python environment

Use a venv. The conversion script pulls in PyTorch and you do not want that in your system or pyenv global Python.

cd ~/llama.cpp
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -r requirements.txt

requirements.txt already installs huggingface_hub at a compatible version. Do not run pip install -U huggingface_hub — that pulls the 1.x line, which breaks transformers. If you already did:

pip install "huggingface_hub<1.0"
pip check    # should report no broken requirements

5. Download the weights

hf auth login          # paste your token
mkdir -p ~/models
hf download fdtn-ai/antares-1b --local-dir ~/models/antares-1b

Sanity check the chat template came down — you’ll need to match it later:

cat ~/models/antares-1b/chat_template.jinja

6. Convert to GGUF — BF16, not F16

cd ~/llama.cpp
source .venv/bin/activate
python convert_hf_to_gguf.py ~/models/antares-1b \
  --outfile ~/models/antares-1b-bf16.gguf --outtype bf16

This is the step that matters. Granite 4.0 is a hybrid Mamba/attention architecture. The SSM state updates exceed F16’s dynamic range, saturate, and the model collapses into emitting a single repeated token forever. BF16 has the same bit width but far more exponent headroom and survives fine.

Symptom if you get this wrong: the model loads cleanly, reports ~100 tok/s, and prints @@@@@@@@@@@... in response to anything.

This is also why the community mradermacher / mitkox GGUF conversions on the Hub don’t work — they appear to route through an F16 intermediate. Convert it yourself.

7. Verify with llama.cpp before touching Ollama

llama-cli -m ~/models/antares-1b-bf16.gguf --jinja -no-cnv \
  --temp 0.3 --top-p 1.0 \
  -p "Vulnerability to locate: CWE-78 OS Command Injection."

--jinja uses the template embedded in the GGUF, so there is nothing to hand-write and get wrong. This isolates model correctness from Ollama config.

Expected: coherent reasoning prose, then </think>, then a response. The reply starts mid-thought with no opening <think> tag — that is correct. The template pre-fills it.

If you see @@@@, stop here and go to Troubleshooting. Do not proceed.

8. Import into Ollama

Ollama does not pick up the chat template from this GGUF automatically, so supply it explicitly. Without it Ollama sends raw text with no role markers and no stop token, and you get @@@@ again even from a good BF16 file.

cd ~/models
cat > Modelfile <<'EOF'
FROM ./antares-1b-bf16.gguf

TEMPLATE """<|start_of_role|>system<|end_of_role|>{{ if .System }}{{ .System }}{{ else }}You are a helpful assistant. Please ensure responses are professional, accurate, and safe.{{ end }}<|end_of_text|>
{{ range .Messages }}{{ if eq .Role "user" }}<|start_of_role|>user<|end_of_role|>{{ .Content }}<|end_of_text|>
{{ else if eq .Role "assistant" }}<|start_of_role|>assistant<|end_of_role|>{{ .Content }}<|end_of_text|>
{{ end }}{{ end }}<|start_of_role|>assistant<|end_of_role|><think>
"""

PARAMETER stop "<|end_of_text|>"
PARAMETER stop "<|start_of_role|>"
PARAMETER temperature 0.3
PARAMETER top_p 1.0
PARAMETER num_ctx 32768
EOF

ollama create antares-1b -f Modelfile

Note there is no --quantize flag on the first build. Get it working unquantized, then shrink.

Three details in that template, all derived from the model’s real chat_template.jinja:

A system block is always emitted — the Jinja has no “if system” branch, it injects a default. Making it conditional produces a prompt shape the model never saw in training.
The generation prompt pre-fills <think>\n. The model starts already inside a reasoning block.
Temperature 0.3 / top-p 1.0 are the settings Cisco used for every published evaluation. Ollama’s defaults (0.8 / top-k 40 / repeat-penalty 1.1) are far too loose for an agentic model.

Verify:

ollama show --modelfile antares-1b | head -30

If TEMPLATE shows {{ .Prompt }}, the template didn’t take — rebuild.

ollama run antares-1b "Vulnerability to locate: CWE-78 OS Command Injection."

9. Shrink it (optional)

3.7 GB is large for a 2B model. Step down one level at a time and test each:

ollama create antares-1b:q8 --quantize q8_0 -f Modelfile
ollama run antares-1b:q8 "Vulnerability to locate: CWE-78 OS Command Injection."

ollama create antares-1b:q4 --quantize q4_K_M -f Modelfile
ollama run antares-1b:q4 "Vulnerability to locate: CWE-78 OS Command Injection."

If a level produces @@@@ or degraded reasoning, you’ve found the floor for this architecture — stay one level up. Hybrid SSM models are more quantization-sensitive than plain transformers.

10. Run it as an agent

The REPL is a diagnostic, not a workflow. To see real tool calls, give it the system prompt it was trained against:

ollama run antares-1b

>>> /set system You are a security vulnerability localization agent. You have access to a terminal with the repository mounted at /workspace/repo/. Use shell commands (grep, find, cat, etc.) to explore the codebase and identify files that contain the reported vulnerability. When confident, submit your findings.
>>> Vulnerability to locate: CWE-78: Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').

Success looks like a <tool_call> block containing a shell command. The model is asking you to run it and return the output.

For the real loop, Cisco ships the Antares CLI as a ZIP in the Files tab of the model repo. It implements the full agent loop against any OpenAI-compatible endpoint:

ollama serve        # exposes http://localhost:11434/v1

Configure the CLI with base URL http://localhost:11434/v1, model antares-1b, and any dummy API key. It supports targeted CWE analyses and repo-wide sweeps, with output in human-readable, JSON, or SARIF.

Alternative: skip Ollama entirely and use llama-server, which gives you the same OpenAI-compatible endpoint and tends to get Granite hybrid fixes first:

llama-server -m ~/models/antares-1b-bf16.gguf --jinja \
  -c 32768 --temp 0.3 --top-p 1.0 --port 8080

Sandboxing — not optional

You are executing shell commands the model invents. Per Cisco’s recommendations:

Run the target repo snapshot in a container with --network=none
10 second command timeout
2 CPU cores, 4 GB RAM limit
Destroy the container after each run
Human review before any remediation action

Troubleshooting

@@@@@@@@@ output

Ranked by likelihood:

GGUF converted at F16 instead of BF16 → reconvert (step 6)
Ollama Modelfile missing TEMPLATE → check ollama show --modelfile
Quantized too aggressively → rebuild unquantized and re-test
Using a community GGUF from the Hub → convert it yourself

401: Invalid username or password from ollama pull

Not an auth problem. Hugging Face returns 401 rather than 404 for repos it can’t show you, to avoid leaking whether private repos exist. You almost certainly have the repo name wrong. (mradermacher/Antares-11b-v2-GGUF is a different, unrelated model — an old 11B SOLAR finetune. Nothing to do with Cisco Antares.)

Ollama and gated/private repos

Ollama’s HF integration doesn’t use tokens at all. It authenticates with the SSH key at ~/.ollama/id_ed25519.pub — add that to https://huggingface.co/settings/keys.

pyenv: python: command not found

No pyenv version is active:

pyenv global 3.10.14 && pyenv rehash

If it still fails, your ~/.zshrc is missing eval "$(pyenv init -)".

zsh: bad pattern: ^[[200~

Bracketed-paste escape leaking into zsh. Run exec zsh, or paste one line at a time.

cmake: command not found

You don’t need it — brew install llama.cpp gives prebuilt binaries. Only brew install cmake if you specifically need to build from source (e.g. to pick up a fix newer than the brew formula).

transformers … requires huggingface-hub<1.0

See step 4.

Cross-checking against reference weights

If GGUF misbehaves and you want to know whether the weights themselves are fine:

cd ~/llama.cpp && source .venv/bin/activate && pip install accelerate
python - <<'EOF'
from transformers import AutoTokenizer, AutoModelForCausalLM
import os
p = os.path.expanduser("~/models/antares-1b")
tok = AutoTokenizer.from_pretrained(p)
m = AutoModelForCausalLM.from_pretrained(p, device_map="cpu")
msgs = [{"role":"user","content":"Vulnerability to locate: CWE-78 OS Command Injection."}]
i = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
o = m.generate(**i, max_new_tokens=80, do_sample=True, temperature=0.3, top_p=1.0)
print(tok.decode(o[0][i["input_ids"].shape[-1]:], skip_special_tokens=False))
EOF

Coherent here but garbage from every GGUF = a llama.cpp conversion/runtime gap. Check the llama.cpp issue tracker, or serve with vLLM, which the model card documents as supported.

Quick reference

# Full sequence, assuming gate accepted and brew installed
brew install llama.cpp
git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp
cd ~/llama.cpp && python3 -m venv .venv && source .venv/bin/activate
pip install -U pip && pip install -r requirements.txt
hf auth login
mkdir -p ~/models && hf download fdtn-ai/antares-1b --local-dir ~/models/antares-1b
python convert_hf_to_gguf.py ~/models/antares-1b \
  --outfile ~/models/antares-1b-bf16.gguf --outtype bf16
llama-cli -m ~/models/antares-1b-bf16.gguf --jinja -no-cnv --temp 0.3 -p "test"
# then the Modelfile from step 8
ollama create antares-1b -f ~/models/Modelfile

Known-good settings: BF16 GGUF, temperature 0.3, top-p 1.0, context 32768, explicit Granite <|start_of_role|> template with pre-filled <think>.

April 27, 2024May 11, 2024

Augmenting Network Engineering with RAFT

The practical implementation of the proposed theory and promise of RAFT with Cisco pyATS and LangChain

Introduction

In the field of network engineering, the complexity and dynamism of network architectures present a significant challenge for configuration, troubleshooting, and optimization. This technical blog post delves into a groundbreaking methodology, Retrieval Augmented Fine-Tuning (RAFT), which leverages the power of Langchain pipelines and advanced AI techniques to transform network data handling and question-answering capabilities.

Before we explore the intricate workings of RAFT, let’s break down some key terminology that will be used throughout this discussion:

Large Language Model (LLM): A machine learning model with a vast number of parameters that can understand and generate natural language text.
Small Language Model (SLM): A more compact version of LLMs, with fewer parameters, often optimized for specific, less-complex tasks.
Generative Pretrained Transformer (GPT): A type of LLM known for generating human-like text, trained on a diverse range of internet text.
Retrieval Augmented Generation (RAG): A method that combines the generative capabilities of LLMs with information retrieval to answer questions using external knowledge sources.
RAG-Fusion: An advanced version of RAG that includes multi-query capabilities and reciprocal ranked fusion to improve answer accuracy.
RAPTOR: Tree-Oriented Retrieval, a retrieval technique that structures information in a hierarchical, tree-like manner for more effective data retrieval.
Fine-Tuning (FT): The process of training a pre-existing model on a new, typically smaller, dataset to specialize its understanding or improve its performance on specific tasks.
Retrieval Augmented Fine-Tuning (RAFT): A method that combines RAG with fine-tuning to embed external knowledge directly into an LLM.
Langchain: A software framework designed to create complex applications that combine LLMs with other components like retrieval systems and databases.

*Think of RAG as an Open Book Exam vs a non-fine-tuned standard LLM as a Closed Book Exam. RAFT proposes implementing the Open Book Exam theory to Fine-Tune the model with external, dynamic, automated, inputs.

With these definitions in mind, let’s dive into the RAFT framework and its practical implementation through Langchain.

The RAFT Framework and its Langchain Implementation

The RAFT methodology starts with Cisco’s pyATS, which is used to extract a network’s routing table. PyATS is an automated network testing system that effectively converts complex network information into machine-readable JSON format. This transformation is the first step in preparing our data for the Langchain pipeline.

*Note from John – the code below is very simplified for the sake of the reader; the actual code can be found here on my GitHub:
automateyournetwork/IOS_XE_RAFT: Using the Retrieval Augmented Fine Tuning (RAFT) approach with Cisco IOS XE JSON using pyATS to fine tune open source models (github.com)

** Follow-up note from John – for the sake of transparency and openness I used this code to generate the data set and used openAI chatGPT Fine-Tuning and Playground to prove the data set can fine tune the chatGPT 3.5 model. I am currently testing the second half of the code on my GPU to confirm the same dataset can fine tune the phi-2 model locally. This has yet to yield correct inference responses however on paper it should work with the chatGPT 3.5 model; I believe phi-2 is the issue not the fine tuning code. TBD.

Data Extraction and Transformation

In the initial phase, the routing table data is extracted:

from pyats.topology import loader
testbed = loader.load('testbed.yaml')
device = testbed.devices['router1']
device.connect()
routing_table = device.parse('show ip route')

Once the routing table is captured, pyATS employs a JSONLoader to transform this data into JSON, making it amenable to further processing:

import json
routing_table_json = json.dumps(routing_table, indent=4)
print(routing_table_json)

Text Splitting and Embedding Generation

Next, we engage text splitting techniques to dissect the routing table’s JSON into discrete textual components that can be easily processed:

def split_text(routing_table_json):
    # Splitting logic goes here
    return split_texts

split_routing_table = split_text(routing_table_json)

Each text fragment is then fed into an LLM to generate embeddings, which are high-dimensional representations of the text’s semantic content:

import openai

def generate_embeddings(texts):
    embeddings = [openai.Embedding.create(input=text) for text in texts]
    return embeddings

routing_table_embeddings = generate_embeddings(split_routing_table)

Vector Storage in ChromaDB

These embeddings are stored in ChromaDB, a vector database optimized for similarity searches:

from chromadb import ChromaDB

db = ChromaDB()
for i, embedding in enumerate(routing_table_embeddings):
    db.insert_vector(embedding, metadata={'text': split_routing_table[i]})

Question Generation via LLMs

Utilizing various LLMs, a seed set of questions about the routing table is expanded into a comprehensive questionnaire that explores all facets of the data:

seed_questions = [
    "What is my default route?",
    "What next hop will my default route use?",
    "What is the outgoing interface of my default route?",
    "If I was a packet going to 192.168.1.1 what interface would I use?",
    "If I was a packet going to 192.168.1.1 what is my next hop?"
]

generated_questions = generate_questions(seed_questions, llm='gpt-3.5')

Retrieval Augmented Generation Loop

The RAG loop is then initiated, with each generated question being run through the Langchain pipeline, leveraging the power of an LLM as a retriever to fetch relevant information from ChromaDB and construct an answer:

from langchain.llms import OpenAI

llm = OpenAI()
for question in generated_questions:
    answer = llm.answer(question, retriever=db)
    print(f"Q: {question}\nA: {answer}\n")

Dataset Creation for Fine-Tuning

The Q&A pairs obtained from the RAG loop form a rich dataset, resembling an interactive session between a network engineer and an AI assistant. This dataset is primed for fine-tuning an LLM to encode a deep understanding of network routing logic:

{
    "messages": [
        {"role": "system", "content": "You are a computer networking expert specializing in network routing tables."},
        {"role": "user", "content": "What is the default route in the routing table?"},
        {"role": "assistant", "content": "The default route in the routing table is 0.0.0.0/0. It is active with a metric of 0, a route preference of 1, and it is sourced statically as indicated by the source protocol \"static\" and source protocol codes \"S*\". The next hop for this default route is 10.10.20.254 via the outgoing interface GigabitEthernet1."}
    ]
}

Fine Tuning

Using openAI’s fine-tuning service (for a small fee; publicly hosted) I was able to validate that my data set of 250 System / User / Assistant sets worked!

Visit platform.openai.com

Details of the successful fine-tuning job

More details about the successful fine-tuning job

Then we can head to the Playground to test our new model!

We can test various checkpoints of our fine-tuned model in the Playground

In the playground pick the base chatGPT model on the left, and your pretrained model (either the final model or a checkpoint) and setup your system; in our case a senior network engineer with a lot of experience with routing tables

Setup your user (or assistant) and ask something that you know the answer to about the routing table

Setup your User or Assistant and start testing your fine-tuned model’s knowledge with natural language chat

Now obviously chatGPT 3.5 has no idea how to answer this but will do it’s best to provide, best cast, a general description or conceptual ideas about a default route on a network; and worst case; a hallucination

Default, *pre-trained*, chatGPT 3.5 base model’s response

Now what do you think?

Will the fine tuned model know the answer ?

Here is the JSON to provide you with the correct answer:

Let’s try something else – how about we pick an IP address and see if the fine-tuned model can tell us what interface it would use ya?

10.255.255.9 would use local interface Loopback109

chatGPT 3.5 base model’s best guess without context

Fine-tuned model knows ***exactly*** where this packet would go

After the initial RAG seeding of the data set and fine-tuning of the model – we no longer need RAG / RAPTOR / Agents / external data sources the knowledge is baked into the model!!

What did this cost me?

It cost about $5.50 worth of LLM (generation of 250 questions; RAG for 250 answers) a penny for the embeddings I need for RAG and a whopping 87 cents to make the actual fine-tuned model. Which leads me to my next steps which are already underway and hopefully working by the time you read this

Using Instructor-XL (embeddings) and Microsoft phi-2 (I know phi-3 is out but phi-2 is smaller and I only have 8GB GPU locally) I want to recreate the above.

100% local, private, free RAFT with CUDA / NIVIDIA GPU at home

To save that cost and to be 100% private and local I am literally in the process of fine-tuning phi-2 with the same dataset I used in the cloud. Stay tuned – you might be able to do this at home!

Thank you – I realize I don’t blog much anymore as I like video as my primary source of sharing knowledge. I will be making a follow up YouTube. But for something this important I wanted to break it down in a blog.

I was inspired by the following papers and articles:

RAFT (berkeley.edu)
RAFT (Retrieval Augmented Fine-tuning): A new way to teach LLMs (Large Language Models) to be better at RAG (Retrieval Augmented Generation) (microsoft.com)
[2403.10131] RAFT: Adapting Language Model to Domain Specific RAG (arxiv.org)

And following open source code notebook on how to fine-tune Microsoft phi-2:
notebooks/phi2-finetune-own-data.ipynb at main · brevdev/notebooks (github.com)

John Capobianco
April 27, 2024

Modernizing pyATS MCP: Streamable HTTP, Nine New Tools, and Two Bugs From April 2025 That Nobody Noticed August 23, 2026
pyATS MCP has been live since April 2025. Bringing it to Streamable HTTP and nine new tools meant an isolated venv (after breaking my own other MCP servers first), a real CML lab whose testbed was pointing at the wrong network, two silent bugs from the original release, and an honest benchmark that didn't say […]
John Capobianco
Running Cisco Antares-1B Locally (macOS / Apple Silicon) August 11, 2026
A working start-to-finish guide to running Cisco Foundation AI's Antares-1B vulnerability localization model on Apple Silicon — including the one non-obvious BF16 conversion step that will otherwise waste your afternoon. The post Running Cisco Antares-1B Locally (macOS / Apple Silicon) appeared first on Automate Your Network.
John Capobianco
Twenty-Three Specs in Six Days: What Happened When I Handed NetClaw a Roadmap August 5, 2026
I wrote down twenty-five gaps in what NetClaw could actually reach, ordered them by dependency, and set one rule: one item, one spec, one branch, no batching. Six days later twenty-three specs had merged — twelve new MCP servers, twenty-three new skills, 199 skills to 221 — and reality had corrected my own roadmap four […]
John Capobianco
My Network Agent Lives in My Pocket Now — And Nobody Is in the Middle July 26, 2026
From an emulator Claude Code drove itself, to a real Android phone in someone else's hands, to an iPhone talking to that Android through my own Border — voice, photos, and a live CML lab, all over NCFED. No Slack, no Discord, no Webex, no third party in the path. The post My Network Agent […]
John Capobianco
BISQUE: NetClaw gets a bookshelf — an offline, agentic RAG knowledge base July 18, 2026
Introducing BISQUE — the Base of Ingested Semantic Queries for Unified Embeddings — NetClaw's fully offline, agentic RAG knowledge base. Because the tastiest thing you can make from claws is a bisque. Uploaded a 212-page book from Slack to cited answers in under three minutes, with honest misses instead of hallucinations. The post BISQUE: NetClaw […]
John Capobianco
Claws that prove who they are: certificate security comes to NetClaw federation July 15, 2026
NetClaw federation used to trust a peer's identity on its say-so, in the clear. Claw Certification adds TLS encryption and real cryptographic identity — domain-verified via Let's Encrypt or pinned trust-on-first-use — and we cut the first claw over on a live domain. The post Claws that prove who they are: certificate security comes to […]
John Capobianco
NetClaws, Assemble: introducing N2N Federation and the NCFED protocol July 12, 2026
NetClaw's mesh could always tell you who was online. Now it tells you what they can do — and lets you use it. Introducing N2N Federation and NCFED, a new protocol for federating AI network engineers safely. The post NetClaws, Assemble: introducing N2N Federation and the NCFED protocol appeared first on Automate Your Network.
John Capobianco
Giving NetClaw Eyes and Hands: From Browser Control to Full Desktop Control July 8, 2026
Most of what NetClaw does, it does through APIs — gNMI, REST, NETCONF, GraphQL. That’s the right way to automate a network: structured, auditable, repeatable. But real networks are messy, and not everything worth touching has a clean API behind it. Some of the most useful information lives behind a login screen in a vendor’s […]
John Capobianco
I Rebuilt a Real Cisco Network as a 3D World — Inside NetClaw’s New Unreal Engine 5.8 Integration July 3, 2026
NetClaw can now clone a live Cisco Modeling Labs topology — gathered with pyATS — straight into a 3D world inside Unreal Engine 5.8, over the Model Context Protocol. Here's the achievement, the vision, and (for the technically curious) every hurdle it took to get there. The post I Rebuilt a Real Cisco Network as […]
John Capobianco
NetClaw + IP Fabric: Network Assurance Meets AI-Powered Automation June 27, 2026
Announcing the integration of IP Fabric's network assurance platform into NetClaw through 10 new MCP tools for health assessment, path analysis, visual diagrams, and API discovery. A collaboration with Daren Fulwell spanning nearly a decade. The post NetClaw + IP Fabric: Network Assurance Meets AI-Powered Automation appeared first on Automate Your Network.
John Capobianco