Let's be honest — I hit my first real wall when I tried to build my own AI assistant. Every tutorial pointed me toward OpenAI's API, Anthropic, or similar services. Then came the bills. Small, almost invisible ones at first, but they added up fast. So I decided to flip the script and see if there was a way to build something genuinely useful without handing over my credit card to a big tech company.

Spoiler: there is. And it's actually pretty exhilarating.

Over the past few months, I've been experimenting with open-source models, local inference engines, and agent frameworks that run entirely on my machine. The results surprised me. Some of these models, running locally, can hold genuine conversations, execute code, browse the web, and even manage files on my computer — all without a single API call to a cloud provider.

In this guide, I'll walk you through exactly how I did it, step by step. If you're tired of paying per-token for every little thing, this might just change your workflow.

Why Go Fully Local?

Before we dive into the how, let me tell you why I made this shift in the first place.

There are three reasons that matter most to me, and honestly, they probably matter to you too:

Cost. This isn't just about feeling cheap — it's about predictability. With local models, my electricity bill is my only variable cost. There's no surprise $47 charge at the end of the month because some script went haywire and fired off a million tokens. For hobby projects and personal tools, that matters.

Privacy. When you send data to an API, you're trusting a third party with whatever you're asking. That could be sensitive documents, personal notes, code you haven't pushed yet, or anything else. Running locally means your data stays local. Period.

Control. Cloud APIs change pricing, availability, and policies overnight. Local models are yours. You decide when to update them, which versions to run, and how to customize them. That freedom is worth a bit more setup effort.

Now, I'm not saying local models are perfect. They lag behind the top-tier cloud models in raw capability, especially for complex reasoning and long-context tasks. But for everyday automation — summarizing your emails, managing your to-do list, writing code snippets, organizing files — they're shockingly good, and the gap is closing fast.

Setting Up Your Local LLM Engine

The foundation of any local AI agent is the model engine — the piece of software that actually runs the language model on your hardware. After trying several options, I landed on Ollama as my go-to choice, and here's why it deserves your attention too.

Ollama is incredibly simple to set up. You download it, run a command, and you're serving models from your own machine. No Docker complexities, no Kubernetes configurations, no cloud accounts to create.

Here's what I did on my macOS machine (the process is nearly identical on Linux and Windows):

# Install Ollama via their official installer
curl -fsSL https://ollama.com/install.sh | sh

# Verify the installation
ollama --version

Once that's running, you can pull your first model. I started with Llama 3.2 3B, a compact but surprisingly capable model from Meta. It fits comfortably on most modern laptops and handles conversational tasks well.

# Pull the Llama 3.2 3B model
ollama pull llama3.2:3b

# Test it immediately
ollama run llama3.2:3b "Explain how REST APIs work in simple terms"

You'll see the model start downloading. Depending on your internet connection, this might take a few minutes. Once it finishes, you can interact with it right in your terminal.

I also keep Mistral 7B and Phi-3.5 in my toolkit for different use cases. Mistral is a bit heavier but handles more complex reasoning, while Phi-3.5 is Microsoft's lightweight champion that punches above its weight class.

A quick note on hardware: if you're running this on a laptop without a dedicated GPU, don't panic. Ollama uses CPU inference as a fallback, and while it won't be lightning fast, it's perfectly usable for personal automation tasks. My MacBook Air M1 handles Llama 3.2 3B at about 20-30 tokens per second, which is more than enough for most workflows.

Building the Agent Framework

Now that we have a model running locally, it's time to build the actual agent — the piece that ties the model to tools and gives it the ability to do things beyond just chatting.

I explored several frameworks before finding the right fit. Options like LangChain and LlamaIndex are powerful but come with significant complexity. After some experimentation, I landed on SmolAgents by Hugging Face. It's lightweight, well-documented, and designed specifically for building practical agents without the overhead.

Here's how I set it up:

# Create a fresh virtual environment
python -m venv .venv
source .venv/bin/activate

# Install SmolAgents and dependencies
pip install smolagents[ollama] python-dotenv

Now, let me show you the actual agent I built. This one handles daily task management — it can create tasks, mark them complete, summarize your day, and organize your notes.

import os
from dotenv import load_dotenv
from smolagents import CodeAgent, OllamaClient

load_dotenv()

# Connect to our local Ollama instance
llm = OllamaClient(
    model_id="llama3.2:3b",
    url="http://localhost:11434"
)

# Define the tools the agent can use
tools = [
    TaskManagerTool(),
    NoteOrganizerTool(),
    WebSearchTool(),
]

# Create the agent with a clear system prompt
agent = CodeAgent(
    tools=tools,
    llm=llm,
    max_steps=10,
    system_prompt="""You are a helpful personal assistant 
    running locally on the user's machine. You can manage tasks, 
    organize notes, and search the web. Always be concise and 
    practical."""
)

# Run the agent with a natural language request
result = agent.run("What do I have on my todo list? Summarize my latest notes and suggest what I should work on first.")
print(result)

The magic here is in the CodeAgent class. Unlike simpler chatbots, a CodeAgent writes and executes Python code to accomplish tasks. This means it can literally interact with your filesystem, call APIs, and manipulate data — all through generated code that runs safely on your machine.

One thing I want to flag: start with simple prompts. When I first tried giving my agent complex multi-step instructions, it would sometimes get stuck in loops or hallucinate tool calls. Breaking down requests into smaller, clearer steps made a dramatic difference.

Connecting Real Tools

An agent without tools is just a chatbot with extra steps. The real power comes from giving your agent access to things it can actually do — search the web, read and write files, run commands, and so on.

Let me show you the tools I built for my personal agent and how they work under the hood.

Task Management Tool

import json
import os
from pathlib import Path
from smolagents import tool

TASKS_FILE = Path.home() / ".local_agent" / "tasks.json"

@tool
def add_task(title: str, priority: str = "medium") -> str:
    """Add a new task to your local task list."""
    data = load_tasks()
    task_id = len(data) + 1
    data.append({
        "id": task_id,
        "title": title,
        "priority": priority,
        "completed": False
    })
    save_tasks(data)
    return f"Added task #{task_id}: {title} (priority: {priority})"

@tool
def list_tasks() -> str:
    """List all tasks, showing pending and completed separately."""
    data = load_tasks()
    pending = [t for t in data if not t["completed"]]
    completed = [t for t in data if t["completed"]]
    
    result = "=== Pending Tasks ===\n"
    for t in pending:
        result += f"  #{t['id']} [{t['priority']}] {t['title']}\n"
    result += f"\n=== Completed ({len(completed)}) ===\n"
    for t in completed:
        result += f"  ✓ #{t['id']}: {t['title']}\n"
    return result if result.strip() else "No tasks yet!"

def load_tasks():
    data = json.loads(TASKS_FILE.read_text()) if TASKS_FILE.exists() else []
    return data

def save_tasks(data):
    TASKS_FILE.parent.mkdir(parents=True, exist_ok=True)
    TASKS_FILE.write_text(json.dumps(data, indent=2))

Web Search Tool

For web search, I use a clever trick: since many search APIs require keys and payment, I built a tool that leverages browser automation through a headless browser, or I use free search endpoints that don't require authentication.

import requests
from smolagents import tool

@tool
def search_web(query: str, max_results: int = 5) -> str:
    """Search the web for information using a free search API."""
    # Using a free public search API
    response = requests.get(
        f"https://api.duckduckgo.com/?q={query}&format=json&no_html=1",
        timeout=10
    )
    data = response.json()
    
    results = []
    abstract = data.get("AbstractText", "No summary available.")
    related = data.get("RelatedTopics", [])[:max_results]
    
    results.append(f"Summary: {abstract}\n")
    for item in related:
        if "Text" in item:
            results.append(f"- {item['Text']}")
    
    return "\n".join(results) if results else "No results found."

Pro Tip: Start Small

When I first connected these tools, I tried to make them all work simultaneously. The agent got confused and started calling tools incorrectly. My fix was simple: enable one tool at a time, verify it works, then add the next one. Your agent will thank you.

Putting It All Together

Here's the complete, working agent that I use daily. I'll walk through the key parts so you understand what's happening.

""" Local AI Agent - A fully offline personal assistant Run with: python agent.py """

import json import os from pathlib import Path from datetime import datetime from smolagents import CodeAgent, tool from smolagents.agents import ToolCall import requests

── Configuration ──────────────────────────────────────

AGENT_DIR = Path.home() / ".local_agent" TASKS_FILE = AGENT_DIR / "tasks.json" NOTES_FILE = AGENT_DIR / "notes.json" LOG_FILE = AGENT_DIR / "agent.log"

── Ensure directories exist ───────────────────────────

AGENT_DIR.mkdir(parents=True, exist_ok=True)

── Tool Definitions ───────────────────────────────────

@tool def add_task(title: str, deadline: str = None, priority: str = "medium") -> str: """Add a task. Priority options: low, medium, high.""" tasks = load_json(TASKS_FILE, []) task_id = len(tasks) + 1 task = { "id": task_id, "title": title, "priority": priority, "deadline": deadline, "created": datetime.now().isoformat(), "completed": False } tasks.append(task) save_json(TASKS_FILE, tasks) log(f"Added task #{task_id}: {title}") return f"✓ Added task #{task_id}: '{title}' (priority: {priority})"

@tool def complete_task(task_id: int) -> str: """Mark a task as completed.""" tasks = load_json(TASKS_FILE, []) for task in tasks: if task["id"] == task_id and not task["completed"]: task["completed"] = True task["completed_at"] = datetime.now().isoformat() save_json(TASKS_FILE, tasks) log(f"Completed task #{task_id}") return f"✓ Marked task #{task_id} as complete" return f"Task #{task_id} not found or already completed."

@tool def list_tasks(status: str = "pending") -> str: """List tasks. Use 'pending' or 'completed'.""" tasks = load_json(TASKS_FILE, []) if status == "pending": filtered = [t for t in tasks if not t["completed"]] label = "Pending" else: filtered = [t for t in tasks if t["completed"]] label = "Completed"

if not filtered: return f"No {label.lower()} tasks."

result = f"=== {label} Tasks ===\n" for t in filtered: deadline_info = f" | Due: {t.get('deadline', 'No deadline')}" if t.get("deadline") else "" result += f" #{t['id']} [{t['priority']}] {t['title']}{deadline_info}\n" return result

@tool def add_note(topic: str, content: str) -> str: """Save a note with a topic label.""" notes = load_json(NOTES_FILE, []) note = { "topic": topic, "content": content, "created": datetime.now().isoformat() } notes.append(note) save_json(NOTES_FILE, notes) log(f"Added note: {topic}") return f"✓ Saved note under '{topic}'"

@tool def search_web(query: str) -> str: """Search the web using DuckDuckGo.""" try: resp = requests.get( f"https://api.duckduckgo.com/?q={query}&format=json&no_html=1", timeout=15 ) data = resp.json() parts = [] if data.get("AbstractText"): parts.append(f"Summary: {data['AbstractText']}") for topic in data.get("RelatedTopics", [])[:3]: if "Text" in topic: parts.append(f"- {topic['Text']}") return "\n".join(parts) if parts else "No results found." except Exception as e: return f"Search failed: {e}"

@tool def read_file(path: str) -> str: """Read and summarize a text file.""" try: full_path = Path(path).expanduser() if not full_path.exists(): return f"File not found: {path}" content = full_path.read_text(encoding="utf-8", errors="ignore") lines = content.split("\n") summary = f"File: {full_path.name} ({len(lines)} lines)\n---\n" summary += content[:2000] if len(content) > 2000: summary += "\n... [truncated]" return summary except Exception as e: return f"Error reading file: {e}"

── Helper Functions ───────────────────────────────────

def load_json(filepath, default=None): if filepath.exists(): return json.loads(filepath.read_text()) return default if default is not None else []

def save_json(filepath, data): filepath.write_text(json.dumps(data, indent=2))

def log(message: str): timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") log_entry = f"[{timestamp}] {message}\n" LOG_FILE.write_text(log_entry, mode="a")

── Agent Initialization ───────────────────────────────

llm_config = { "model_id": "llama3.2:3b", "url": "http://localhost:11434" }

agent_tools = [ add_task, complete_task, list_tasks, add_note, search_web, read_file, ]

agent = CodeAgent( tools=agent_tools, llm=llm_config, max_steps=15, system_prompt="""You are a helpful local AI assistant. You can manage tasks, save notes, search the web, and read files — all running offline on this machine. Be practical, concise, and always verify your actions before reporting them back.""" )

── Interactive Mode ───────────────────────────────────

if __name__ == "__main__": print("=" 50) print(" Local AI Agent — Running on Ollama (llama3.2:3b)") print(" Type 'quit' to exit, 'clear' to reset context") print("=" 50)

while True: try: user_input = input("\nYou: ").strip() if user_input.lower() in ("quit", "exit", "q"): print("Goodbye!") break if user_input.lower() == "clear": agent = CodeAgent( tools=agent_tools, llm=llm_config, max_steps=15, system_prompt=agent.system_prompt ) print("Context cleared.") continue if not user_input: continue print("\nAgent: ", end="", flush=True) response = agent.run(user_input) print(response) except KeyboardInterrupt: print("\nGoodbye!") break except Exception as e: print(f"\nError: {e}") log(f"Error

Tech Master Hub

How to Build a Local AI Agent Without Spending a Dime on API Keys