Navigate back to the previous page in browser history. Requires browser_navigate to be called first.

Click on an element identified by its ref ID from the snapshot (e.g., '@e5'). The ref IDs are shown in square brackets in the snapshot output. Requires browser_navigate and browser_snapshot to be called first.

The element reference from the snapshot (e.g., '@e5', '@e12')

Close the browser session and release resources. Call this when done with browser tasks to free up Browserbase session quota.

Get browser console output and JavaScript errors from the current page. Returns console.log/warn/error/info messages and uncaught JS exceptions. Use this to detect silent JavaScript errors, failed API calls, and application warnings. Requires browser_navigate to be called first.

If true, clear the message buffers after reading

Get a list of all images on the current page with their URLs and alt text. Useful for finding images to analyze with the vision tool. Requires browser_navigate to be called first.

Navigate to a URL in the browser. Initializes the session and loads the page. Must be called before other browser tools. For simple information retrieval, prefer web_search or web_extract (faster, cheaper). Use browser tools when you need to interact with a page (click, fill forms, dynamic content).

The URL to navigate to (e.g., 'https://example.com')

Press a keyboard key. Useful for submitting forms (Enter), navigating (Tab), or keyboard shortcuts. Requires browser_navigate to be called first.

Key to press (e.g., 'Enter', 'Tab', 'Escape', 'ArrowDown')

Scroll the page in a direction. Use this to reveal more content that may be below or above the current viewport. Requires browser_navigate to be called first.

Direction to scroll

Get a text-based snapshot of the current page's accessibility tree. Returns interactive elements with ref IDs (like @e1, @e2) for browser_click and browser_type. full=false (default): compact view with interactive elements. full=true: complete page content. Snapshots over 8000 chars are truncated or LLM-summarized. Requires browser_navigate first.

If true, returns complete page content. If false (default), returns compact view with interactive elements only.

Type text into an input field identified by its ref ID. Clears the field first, then types the new text. Requires browser_navigate and browser_snapshot to be called first.

The element reference from the snapshot (e.g., '@e3')

The text to type into the field

Take a screenshot of the current page and analyze it with vision AI. Use this when you need to visually understand what's on the page - especially useful for CAPTCHAs, visual verification challenges, complex layouts, or when the text snapshot doesn't capture important visual information. Returns both the AI analysis and a screenshot_path that you can share with the user by including MEDIA:<screenshot_path> in your response. Requires browser_navigate to be called first.

What you want to know about the page visually. Be specific about what you're looking for.

If true, overlay numbered [N] labels on interactive elements. Each [N] maps to ref @eN for subsequent browser commands. Useful for QA and spatial reasoning about page layout.

clarify

Ask the user a question when you need clarification, feedback, or a decision before proceeding. Supports two modes:

1. **Multiple choice** — provide up to 4 choices. The user picks one or types their own answer via a 5th 'Other' option.
2. **Open-ended** — omit choices entirely. The user types a free-form response.

Use this tool when:
- The task is ambiguous and you need the user to choose an approach
- You want post-task feedback ('How did that work out?')
- You want to offer to save a skill or update memory
- A decision has meaningful trade-offs the user should weigh in on

Do NOT use this tool for simple yes/no confirmation of dangerous commands (the terminal tool handles that). Prefer making a reasonable default choice yourself when the decision is low-stakes.

The question to present to the user.

Choices:
Up to 4 answer choices. Omit this parameter entirely to ask an open-ended question. When provided, the UI automatically appends an 'Other (type your answer)' option.

cronjob

Manage scheduled cron jobs with a single compressed tool.

Use action='create' to schedule a new job from a prompt or one or more skills.
Use action='list' to inspect jobs.
Use action='update', 'pause', 'resume', 'remove', or 'run' to manage an existing job.

Jobs run in a fresh session with no current-chat context, so prompts must be self-contained.
If skill or skills are provided on create, the future cron run loads those skills in order, then follows the prompt as the task instruction.
On update, passing skills=[] clears attached skills.

NOTE: The agent's final response is auto-delivered to the target. Put the primary
user-facing content in the final response. Cron jobs run autonomously with no user
present — they cannot ask questions or request clarification.

Important safety rule: cron-run sessions should not recursively schedule more cron jobs.

One of: create, list, update, pause, resume, remove, run

Required for update/pause/resume/remove/run

For create: the full self-contained prompt. If skill or skills are also provided, this becomes the task instruction paired with those skills.

For create/update: '30m', 'every 2h', '0 9 * * *', or ISO timestamp

Optional human-friendly name

Optional repeat count. Omit for defaults (once for one-shot, forever for recurring).

Delivery target: origin, local, telegram, discord, slack, whatsapp, signal, matrix, mattermost, homeassistant, dingtalk, feishu, wecom, email, sms, or platform:chat_id or platform:chat_id:thread_id for Telegram topics. Examples: 'origin', 'local', 'telegram', 'telegram:-1001234567890:17585', 'discord:#engineering'

Optional per-job model override used when the cron job runs

Optional per-job provider override used when resolving runtime credentials

Optional per-job base URL override paired with provider/model routing

For list: include paused/completed jobs

Optional single skill name to load before executing the cron prompt

Optional ordered list of skills to load before executing the cron prompt. On update, pass an empty array to clear attached skills.

Optional pause reason

Spawn one or more subagents to work on tasks in isolated contexts. Each subagent gets its own conversation, terminal session, and toolset. Only the final summary is returned -- intermediate tool results never enter your context window.

TWO MODES (one of 'goal' or 'tasks' is required):
1. Single task: provide 'goal' (+ optional context, toolsets)
2. Batch (parallel): provide 'tasks' array with up to 3 items. All run concurrently and results are returned together.

WHEN TO USE delegate_task:
- Reasoning-heavy subtasks (debugging, code review, research synthesis)
- Tasks that would flood your context with intermediate data
- Parallel independent workstreams (research A and B simultaneously)

WHEN NOT TO USE (use these instead):
- Mechanical multi-step work with no reasoning needed -> use execute_code
- Single tool call -> just call the tool directly
- Tasks needing user interaction -> subagents cannot use clarify

IMPORTANT:
- Subagents have NO memory of your conversation. Pass all relevant info (file paths, error messages, constraints) via the 'context' field.
- Subagents CANNOT call: delegate_task, clarify, memory, send_message, execute_code.
- Each subagent gets its own terminal session (separate working directory and state).
- Results are always returned as an array, one entry per task.

What the subagent should accomplish. Be specific and self-contained -- the subagent knows nothing about your conversation history.

Background information the subagent needs: file paths, error messages, project structure, constraints. The more specific you are, the better the subagent performs.

Toolsets to enable for this subagent. Default: inherits your enabled toolsets. Common patterns: ['terminal', 'file'] for code work, ['web'] for research, ['terminal', 'file', 'web'] for full-stack tasks.

Task goal

Task-specific context

Toolsets for this specific task

Batch mode: up to 3 tasks to run in parallel. Each gets its own subagent with isolated context and terminal session. When provided, top-level goal/context/toolsets are ignored.

Max tool-calling turns per subagent (default: 50). Only set lower for simple tasks.

Run a Python script that can call Hermes tools programmatically. Use this when you need 3+ tool calls with processing logic between them, need to filter/reduce large tool outputs before they enter your context, need conditional branching (if X then Y else Z), or need to loop (fetch N pages, process N files, retry on failure).

Use normal tool calls instead when: single tool call with no processing, you need to see the full result and apply complex reasoning, or the task requires interactive user input.

Available via `from hermes_tools import ...`:

  web_search(query: str, limit: int = 5) -> dict
    Returns {"data": {"web": [{"url", "title", "description"}, ...]}}
  web_extract(urls: list[str]) -> dict
    Returns {"results": [{"url", "title", "content", "error"}, ...]} where content is markdown
  read_file(path: str, offset: int = 1, limit: int = 500) -> dict
    Lines are 1-indexed. Returns {"content": "...", "total_lines": N}
  write_file(path: str, content: str) -> dict
    Always overwrites the entire file.
  search_files(pattern: str, target="content", path=".", file_glob=None, limit=50) -> dict
    target: "content" (search inside files) or "files" (find files by name). Returns {"matches": [...]}
  patch(path: str, old_string: str, new_string: str, replace_all: bool = False) -> dict
    Replaces old_string with new_string in the file.
  terminal(command: str, timeout=None, workdir=None) -> dict
    Foreground only (no background/pty). Returns {"output": "...", "exit_code": N}

Limits: 5-minute timeout, 50KB stdout cap, max 50 tool calls per script. terminal() is foreground-only (no background or pty). If the session uses a cloud sandbox backend, treat it as resumable task state rather than a durable always-on machine.

Print your final result to stdout. Use Python stdlib (json, re, math, csv, datetime, collections, etc.) for processing between tool calls.

Also available (no import needed — built into hermes_tools):
  json_parse(text: str) — json.loads with strict=False; use for terminal() output with control chars
  shell_quote(s: str) — shlex.quote(); use when interpolating dynamic strings into shell commands
  retry(fn, max_attempts=3, delay=2) — retry with exponential backoff for transient failures

Python code to execute. Import tools with `from hermes_tools import web_search, terminal, ...` and print your final result to stdout.

memory

Save durable information to persistent memory that survives across sessions. Memory is injected into future turns, so keep it compact and focused on facts that will still matter later.

WHEN TO SAVE (do this proactively, don't wait to be asked):
- User corrects you or says 'remember this' / 'don't do that again'
- User shares a preference, habit, or personal detail (name, role, timezone, coding style)
- You discover something about the environment (OS, installed tools, project structure)
- You learn a convention, API quirk, or workflow specific to this user's setup
- You identify a stable fact that will be useful again in future sessions

PRIORITY: User preferences and corrections > environment facts > procedural knowledge. The most valuable memory prevents the user from having to repeat themselves.

Do NOT save task progress, session outcomes, completed-work logs, or temporary TODO state to memory; use session_search to recall those from past transcripts.
If you've discovered a new way to do something, solved a problem that could be necessary later, save it as a skill with the skill tool.

TWO TARGETS:
- 'user': who the user is -- name, role, preferences, communication style, pet peeves
- 'memory': your notes -- environment facts, project conventions, tool quirks, lessons learned

ACTIONS: add (new entry), replace (update existing -- old_text identifies it), remove (delete -- old_text identifies it).

SKIP: trivial/obvious info, things easily re-discovered, raw data dumps, and temporary task state.

The action to perform.

Which memory store: 'memory' for personal notes, 'user' for user profile.

The entry content. Required for 'add' and 'replace'.

Short unique substring identifying the entry to replace or remove.

Route a hard problem through multiple frontier LLMs collaboratively. Makes 5 API calls (4 reference models + 1 aggregator) with maximum reasoning effort — use sparingly for genuinely difficult problems. Best for: complex math, advanced algorithms, multi-step analytical reasoning, problems benefiting from diverse perspectives.

The complex query or problem to solve using multiple AI models. Should be a challenging problem that benefits from diverse perspectives and collaborative reasoning.

patch

Targeted find-and-replace edits in files. Use this instead of sed/awk in terminal. Uses fuzzy matching (9 strategies) so minor whitespace/indentation differences won't break it. Returns a unified diff. Auto-runs syntax checks after editing.

Replace mode (default): find a unique string and replace it.
Patch mode: apply V4A multi-file patches for bulk changes.

Edit mode: 'replace' for targeted find-and-replace, 'patch' for V4A multi-file patches

File path to edit (required for 'replace' mode)

Text to find in the file (required for 'replace' mode). Must be unique in the file unless replace_all=true. Include enough surrounding context to ensure uniqueness.

Replacement text (required for 'replace' mode). Can be empty string to delete the matched text.

Replace all occurrences instead of requiring a unique match (default: false)

V4A format patch content (required for 'patch' mode). Format:
*** Begin Patch
*** Update File: path/to/file
@@ context hint @@
 context line
-removed line
+added line
*** End Patch

process

Manage background processes started with terminal(background=true). Actions: 'list' (show all), 'poll' (check status + new output), 'log' (full output with pagination), 'wait' (block until done or timeout), 'kill' (terminate), 'write' (send raw stdin data without newline), 'submit' (send data + Enter, for answering prompts).

Action to perform on background processes

Process session ID (from terminal background output). Required for all actions except 'list'.

Text to send to process stdin (for 'write' and 'submit' actions)

Max seconds to block for 'wait' action. Returns partial output on timeout.

Line offset for 'log' action (default: last 200 lines)

Max lines to return for 'log' action

Read a text file with line numbers and pagination. Use this instead of cat/head/tail in terminal. Output format: 'LINE_NUM|CONTENT'. Suggests similar filenames if not found. Use offset and limit for large files. Reads exceeding ~100K characters are rejected; use offset and limit to read specific sections of large files. NOTE: Cannot read images or binary files — use vision_analyze for images.

Path to the file to read (absolute, relative, or ~/path)

Line number to start reading from (1-indexed, default: 1)

Maximum number of lines to read (default: 500, max: 2000)

Search file contents or find files by name. Use this instead of grep/rg/find/ls in terminal. Ripgrep-backed, faster than shell equivalents.

Content search (target='content'): Regex search inside files. Output modes: full matches with line numbers, file paths only, or match counts.

File search (target='files'): Find files by glob pattern (e.g., '*.py', '*config*'). Also use this instead of ls — results sorted by modification time.

Regex pattern for content search, or glob pattern (e.g., '*.py') for file search

'content' searches inside file contents, 'files' searches for files by name

Directory or file to search in (default: current working directory)

Filter files by pattern in grep mode (e.g., '*.py' to only search Python files)

Maximum number of results to return (default: 50)

Skip first N results for pagination (default: 0)

Output format for grep mode: 'content' shows matching lines with line numbers, 'files_only' lists file paths, 'count' shows match counts per file

Number of context lines before and after each match (grep mode only)

Search your long-term memory of past conversations, or browse recent sessions. This is your recall -- every past session is searchable, and this tool summarizes what happened.

TWO MODES:
1. Recent sessions (no query): Call with no arguments to see what was worked on recently. Returns titles, previews, and timestamps. Zero LLM cost, instant. Start here when the user asks what were we working on or what did we do recently.
2. Keyword search (with query): Search for specific topics across all past sessions. Returns LLM-generated summaries of matching sessions.

USE THIS PROACTIVELY when:
- The user says 'we did this before', 'remember when', 'last time', 'as I mentioned'
- The user asks about a topic you worked on before but don't have in current context
- The user references a project, person, or concept that seems familiar but isn't in memory
- You want to check if you've solved a similar problem before
- The user asks 'what did we do about X?' or 'how did we fix Y?'

Don't hesitate to search when it is actually cross-session -- it's fast and cheap. Better to search and confirm than to guess or ask the user to repeat themselves.

Search syntax: keywords joined with OR for broad recall (elevenlabs OR baseten OR funding), phrases for exact match ("docker networking"), boolean (python NOT java), prefix (deploy*). IMPORTANT: Use OR between keywords for best results — FTS5 defaults to AND which misses sessions that only mention some terms. If a broad OR query returns nothing, try individual keyword searches in parallel. Returns summaries of the top matching sessions.

Search query — keywords, phrases, or boolean expressions to find in past sessions. Omit this parameter entirely to browse recent sessions instead (returns titles, previews, timestamps with no LLM cost).

Optional: only search messages from specific roles (comma-separated). E.g. 'user,assistant' to skip tool outputs.

Max sessions to summarize (default: 3, max: 5).

Manage skills (create, update, delete). Skills are your procedural memory — reusable approaches for recurring task types. New skills go to ~/.hermes/skills/; existing skills can be modified wherever they live.

Actions: create (full SKILL.md + optional category), patch (old_string/new_string — preferred for fixes), edit (full SKILL.md rewrite — major overhauls only), delete, write_file, remove_file.

Create when: complex task succeeded (5+ calls), errors overcome, user-corrected approach worked, non-trivial workflow discovered, or user asks you to remember a procedure.
Update when: instructions stale/wrong, OS-specific failures, missing steps or pitfalls found during use. If you used a skill and hit issues not covered by it, patch it immediately.

After difficult/iterative tasks, offer to save as a skill. Skip for simple one-offs. Confirm with user before creating/deleting.

Good skills: trigger conditions, numbered steps with exact commands, pitfalls section, verification steps. Use skill_view() to see format examples.

create patch edit delete write_file remove_file

The action to perform.

Skill name (lowercase, hyphens/underscores, max 64 chars). Must match an existing skill for patch/edit/delete/write_file/remove_file.

Full SKILL.md content (YAML frontmatter + markdown body). Required for 'create' and 'edit'. For 'edit', read the skill first with skill_view() and provide the complete updated text.

Text to find in the file (required for 'patch'). Must be unique unless replace_all=true. Include enough surrounding context to ensure uniqueness.

Replacement text (required for 'patch'). Can be empty string to delete the matched text.

For 'patch': replace all occurrences instead of requiring a unique match (default: false).

Optional category/domain for organizing the skill (e.g., 'devops', 'data-science', 'mlops'). Creates a subdirectory grouping. Only used with 'create'.

Path to a supporting file within the skill directory. For 'write_file'/'remove_file': required, must be under references/, templates/, scripts/, or assets/. For 'patch': optional, defaults to SKILL.md if omitted.

Content for the file. Required for 'write_file'.

Skills allow for loading information about specific tasks and workflows, as well as scripts and templates. Load a skill's full content or access its linked files (references, templates, scripts). First call returns SKILL.md content plus a 'linked_files' dict showing available references/templates/scripts. To access those, call again with file_path parameter.

The skill name (use skills_list to see available skills)

OPTIONAL: Path to a linked file within the skill (e.g., 'references/api.md', 'templates/config.yaml', 'scripts/validate.py'). Omit to get the main SKILL.md content.

List available skills (name + description). Use skill_view(name) to load full content.

Optional category filter to narrow results

terminal

Execute shell commands on a Linux environment. Filesystem usually persists between calls.

Do NOT use cat/head/tail to read files — use read_file instead.
Do NOT use grep/rg/find to search — use search_files instead.
Do NOT use ls to list directories — use search_files(target='files') instead.
Do NOT use sed/awk to edit files — use patch instead.
Do NOT use echo/cat heredoc to create files — use write_file instead.
Reserve terminal for: builds, installs, git, processes, scripts, network, package managers, and anything that needs a shell.

Foreground (default): Commands return INSTANTLY when done, even if the timeout is high. Set timeout=300 for long builds/scripts — you'll still get the result in seconds if it's fast. Prefer foreground for everything that finishes.
Background: ONLY for long-running servers, watchers, or processes that never exit. Set background=true to get a session_id, then use process(action="wait") to block until done — it returns instantly on completion, same as foreground. Use process(action="poll") only when you need a progress check without blocking.
Do NOT use background for scripts, builds, or installs — foreground with a generous timeout is always better (fewer tool calls, instant results).
Working directory: Use 'workdir' for per-command cwd.
PTY mode: Set pty=true for interactive CLI tools (Codex, Claude Code, Python REPL).

Do NOT use vim/nano/interactive tools without pty=true — they hang without a pseudo-terminal. Pipe git output to cat if it might page.
Important: cloud sandboxes may be cleaned up, idled out, or recreated between turns. Persistent filesystem means files can resume later; it does NOT guarantee a continuously running machine or surviving background processes. Use terminal sandboxes for task work, not durable hosting.

The command to execute on the VM

ONLY for servers/watchers that never exit. For scripts, builds, installs — use foreground with timeout instead (it returns instantly when done).

Max seconds to wait (default: 180). Returns INSTANTLY when command finishes — set high for long tasks, you won't wait unnecessarily.

Working directory for this command (absolute path). Defaults to the session working directory.

Seconds between automatic status checks for background processes (gateway/messaging only, minimum 30). When set, I'll proactively report progress.

Run in pseudo-terminal (PTY) mode for interactive CLI tools like Codex, Claude Code, or Python REPL. Only works with local and SSH backends. Default: false.

Convert text to speech audio. Returns a MEDIA: path that the platform delivers as a voice message. On Telegram it plays as a voice bubble, on Discord/WhatsApp as an audio attachment. In CLI mode, saves to ~/voice-memos/. Voice and provider are user-configured, not model-selected.

The text to convert to speech. Keep under 4000 characters.

Optional custom file path to save the audio. Defaults to ~/.hermes/audio_cache/<timestamp>.mp3

todo

Manage your task list for the current session. Use for complex tasks with 3+ steps or when the user provides multiple tasks. Call with no parameters to read the current list.

Writing:
- Provide 'todos' array to create/update items
- merge=false (default): replace the entire list with a fresh plan
- merge=true: update existing items by id, add any new ones

Each item: {id: string, content: string, status: pending|in_progress|completed|cancelled}
List order is priority. Only ONE item in_progress at a time.
Mark items completed immediately when done. If something fails, cancel it and add a revised item.

Always returns the full current list.

Task items to write. Omit to read current list.

Unique item identifier

Task description

Current status

true: update existing items by id, add new ones. false (default): replace the entire list.

Analyze images using AI vision. Provides a comprehensive description and answers a specific question about the image content.

Image URL (http/https) or local file path to analyze.

Your specific question or request about the image to resolve. The AI will automatically provide a complete image description AND answer your specific question.

Extract content from web page URLs. Returns page content in markdown format. Also works with PDF URLs (arxiv papers, documents, etc.) — pass the PDF link directly and it converts to markdown text. Pages under 5000 chars return full markdown; larger pages are LLM-summarized and capped at ~5000 chars per page. Pages over 2M chars are refused. If a URL fails or times out, use the browser tool to access it instead.

List of URLs to extract content from (max 5 URLs per call)

Search the web for information on any topic. Returns up to 5 relevant results with titles, URLs, and descriptions.

The search query to look up on the web

Write content to a file, completely replacing existing content. Use this instead of echo/cat heredoc in terminal. Creates parent directories automatically. OVERWRITES the entire file — use 'patch' for targeted edits.

Path to the file to write (will be created if it doesn't exist, overwritten if it does)

Complete content to write to the file


Me podrías decir si el Plan always free de oracle sería suficiente para correr mi propia instancia de algún modelo local que te parece podes investigar??!


llama-cpp

---
name: llama-cpp
description: Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.
version: 1.0.0
author: Orchestra Research
license: MIT
dependencies: [llama-cpp-python]
metadata:
  hermes:
    tags: [Inference Serving, Llama.cpp, CPU Inference, Apple Silicon, Edge Deployment, GGUF, Quantization, Non-NVIDIA, AMD GPUs, Intel GPUs, Embedded]

---

# llama.cpp

Pure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non-NVIDIA hardware.

## When to use llama.cpp

**Use llama.cpp when:**
- Running on CPU-only machines
- Deploying on Apple Silicon (M1/M2/M3/M4)
- Using AMD or Intel GPUs (no CUDA)
- Edge deployment (Raspberry Pi, embedded systems)
- Need simple deployment without Docker/Python

**Use TensorRT-LLM instead when:**
- Have NVIDIA GPUs (A100/H100)
- Need maximum throughput (100K+ tok/s)
- Running in datacenter with CUDA

**Use vLLM instead when:**
- Have NVIDIA GPUs
- Need Python-first API
- Want PagedAttention

## Quick start

### Installation

```bash
# macOS/Linux
brew install llama.cpp

# Or build from source
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

# With Metal (Apple Silicon)
make LLAMA_METAL=1

# With CUDA (NVIDIA)
make LLAMA_CUDA=1

# With ROCm (AMD)
make LLAMA_HIP=1
```

### Download model

```bash
# Download from HuggingFace (GGUF format)
huggingface-cli download \
    TheBloke/Llama-2-7B-Chat-GGUF \
    llama-2-7b-chat.Q4_K_M.gguf \
    --local-dir models/

# Or convert from HuggingFace
python convert_hf_to_gguf.py models/llama-2-7b-chat/
```

### Run inference

```bash
# Simple chat
./llama-cli \
    -m models/llama-2-7b-chat.Q4_K_M.gguf \
    -p "Explain quantum computing" \
    -n 256  # Max tokens

# Interactive chat
./llama-cli \
    -m models/llama-2-7b-chat.Q4_K_M.gguf \
    --interactive
```

### Server mode

```bash
# Start OpenAI-compatible server
./llama-server \
    -m models/llama-2-7b-chat.Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -ngl 32  # Offload 32 layers to GPU

# Client request
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-2-7b-chat",
    "messages": [{"role": "user", "content": "Hello!"}],
    "temperature": 0.7,
    "max_tokens": 100
  }'
```

## Quantization formats

### GGUF format overview

| Format | Bits | Size (7B) | Speed | Quality | Use Case |
|--------|------|-----------|-------|---------|----------|
| **Q4_K_M** | 4.5 | 4.1 GB | Fast | Good | **Recommended default** |
| Q4_K_S | 4.3 | 3.9 GB | Faster | Lower | Speed critical |
| Q5_K_M | 5.5 | 4.8 GB | Medium | Better | Quality critical |
| Q6_K | 6.5 | 5.5 GB | Slower | Best | Maximum quality |
| Q8_0 | 8.0 | 7.0 GB | Slow | Excellent | Minimal degradation |
| Q2_K | 2.5 | 2.7 GB | Fastest | Poor | Testing only |

### Choosing quantization

```bash
# General use (balanced)
Q4_K_M  # 4-bit, medium quality

# Maximum speed (more degradation)
Q2_K or Q3_K_M

# Maximum quality (slower)
Q6_K or Q8_0

# Very large models (70B, 405B)
Q3_K_M or Q4_K_S  # Lower bits to fit in memory
```

## Hardware acceleration

### Apple Silicon (Metal)

```bash
# Build with Metal
make LLAMA_METAL=1

# Run with GPU acceleration (automatic)
./llama-cli -m model.gguf -ngl 999  # Offload all layers

# Performance: M3 Max 40-60 tokens/sec (Llama 2-7B Q4_K_M)
```

### NVIDIA GPUs (CUDA)

```bash
# Build with CUDA
make LLAMA_CUDA=1

# Offload layers to GPU
./llama-cli -m model.gguf -ngl 35  # Offload 35/40 layers

# Hybrid CPU+GPU for large models
./llama-cli -m llama-70b.Q4_K_M.gguf -ngl 20  # GPU: 20 layers, CPU: rest
```

### AMD GPUs (ROCm)

```bash
# Build with ROCm
make LLAMA_HIP=1

# Run with AMD GPU
./llama-cli -m model.gguf -ngl 999
```

## Common patterns

### Batch processing

```bash
# Process multiple prompts from file
cat prompts.txt | ./llama-cli \
    -m model.gguf \
    --batch-size 512 \
    -n 100
```

### Constrained generation

```bash
# JSON output with grammar
./llama-cli \
    -m model.gguf \
    -p "Generate a person: " \
    --grammar-file grammars/json.gbnf

# Outputs valid JSON only
```

### Context size

```bash
# Increase context (default 512)
./llama-cli \
    -m model.gguf \
    -c 4096  # 4K context window

# Very long context (if model supports)
./llama-cli -m model.gguf -c 32768  # 32K context
```

## Performance benchmarks

### CPU performance (Llama 2-7B Q4_K_M)

| CPU | Threads | Speed | Cost |
|-----|---------|-------|------|
| Apple M3 Max | 16 | 50 tok/s | $0 (local) |
| AMD Ryzen 9 7950X | 32 | 35 tok/s | $0.50/hour |
| Intel i9-13900K | 32 | 30 tok/s | $0.40/hour |
| AWS c7i.16xlarge | 64 | 40 tok/s | $2.88/hour |

### GPU acceleration (Llama 2-7B Q4_K_M)

| GPU | Speed | vs CPU | Cost |
|-----|-------|--------|------|
| NVIDIA RTX 4090 | 120 tok/s | 3-4× | $0 (local) |
| NVIDIA A10 | 80 tok/s | 2-3× | $1.00/hour |
| AMD MI250 | 70 tok/s | 2× | $2.00/hour |
| Apple M3 Max (Metal) | 50 tok/s | ~Same | $0 (local) |

## Supported models

**LLaMA family**:
- Llama 2 (7B, 13B, 70B)
- Llama 3 (8B, 70B, 405B)
- Code Llama

**Mistral family**:
- Mistral 7B
- Mixtral 8x7B, 8x22B

**Other**:
- Falcon, BLOOM, GPT-J
- Phi-3, Gemma, Qwen
- LLaVA (vision), Whisper (audio)

**Find models**: https://huggingface.co/models?library=gguf

## References

- **[Quantization Guide](references/quantization.md)** - GGUF formats, conversion, quality comparison
- **[Server Deployment](references/server.md)** - API endpoints, Docker, monitoring
- **[Optimization](references/optimization.md)** - Performance tuning, hybrid CPU+GPU

## Resources

- **GitHub**: https://github.com/ggerganov/llama.cpp
- **Models**: https://huggingface.co/models?library=gguf
- **Discord**: https://discord.gg/llama-cpp

To view linked files, call skill_view(name, file_path) where file_path is e.g. 'references/api.md' or 'assets/config.yaml'


Explanation:
**Researching Oracle specs**

I’m thinking about researching the current specs for Oracle's Always Free offerings, especially the Ampere A1 and E2 instances. I could start with a web search that focuses on the Oracle page for their latest updates. Maybe I’ll consider using parallel tools to streamline my search and find benchmarks, possibly looking at llama.cpp or relevant discussions on Reddit. I want to make sure I gather accurate, up-to-date information!

Question: Oracle Cloud Always Free Ampere A1 4 OCPU 24 GB RAM specifications

Question: Oracle Cloud Always Free local LLM llama.cpp Ampere A1 benchmark 7B 2025

Question: Oracle Cloud Always Free ARM local LLM Qwen 2.5 3B 7B llama.cpp

Question: Oracle Cloud Always Free resources AMD micro instance specs


Setup Forever Free Oracle 24 GB RAM ARM-based Ampere Cloud

This video demonstrates how to create a free instance in Oracle Cloud using the Ampere processor, providing 24GB of RAM.

Oracle Cloud Always Free Tier Ampere A1 Instance 4C 24GB RAM ...

Oracle Cloud Always Free Tier Ampere A1 Instance 4C 24GB RAM Details. Oracle Cloud Always Free Tier Ampere A1 Instance 4C 24GB RAM Details.

How long can I safely run an Always Free ARM (A1) instance with 4 ...

# How long can I safely run an Always Free ARM (A1) instance with 4 OCPUs & 24 GB RAM? r/oraclecloud - How long can I safely run an Always Free ARM (A1) instance with 4 OCPUs & 24 GB RAM? r/oraclecloud - How long can I safely run an Always Free ARM (A1) instance with 4 OCPUs & 24 GB RAM? I’m using Oracle Cloud Infrastructure and running an **OCI Ampere A1 (ARM) instance with 4 OCPUs and 24 GB RAM**, which is advertised as *Always Free* (3,000 OCPU hours and 18,000 GB hours per month). However, I’ve noticed a pattern where:. * The instance runs free at the start of the month. * Around the **middle of the month**, billing suddenly starts. * This has happened more than once, even though I didn’t exceed the documented monthly limits and didn’t change the configuration. Reddit Rules    Privacy Policy    User Agreement    Accessibility    Impressum   Reddit, Inc.

Free Tier VM.Standard.A1.Flex 4OCPU/24GB RAM - VPSBenchmarks

Yabs.sh benchmarks results for Oracle Corporation (ORACLE Cloud) - Free Tier VM.Standard.A1.Flex 4OCPU/24GB RAM | AD-3 FD-1 Frankfurt.

Always Free Resources - Oracle Help Center

Using the Always Free resources, you can provision a virtual machine (VM) instance, an Oracle Autonomous Database, and the networking, load balancing, and storage resources needed to support the applications that you want to build. All tenancies get a set of Always Free resources in the Compute service for creating compute virtual machine (VM) instances. Your account comes with 200 GB of Always Free block volume storage which you use to create the boot volumes for your compute instances. Many combinations are possible, depending on how you allocate your block storage and OCI Ampere A1 Compute OCPUs. See Details of the Always Free compute instances for more information on allocating OCPU and memory resources when creating OCI Ampere A1 Compute instances. All tenancies receive a total of 200 GB of Block Volume storage, and five volume backups included in the Always Free resources. Also, because the minimum boot volume size allowed for compute instances is 50 GB, launching four instances will use all your Always Free Block Volume resources.


Performance of llama.cpp on Apple Silicon M-series #4167 - GitHub

As an example, here is how the same test compares over time on M2 Ultra: | | BW [GB/s] | GPU Cores | F16 PP [t/s] | F16 TG [t/s] | Q8\_0 PP [t/s] | Q8\_0 TG [t/s] | Q4\_0 PP [t/s] | Q4\_0 TG [t/s] | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 2023 Nov 21 | | M2 Ultra 8e672ef | 800 | 76 | 1401.85 | 41.02 | 1248.59 | 66.64 | 1238.48 | 94.27 | | 2024 Nov 12 | | M2 Ultra 86ed72d + FA | 800 | 76 | 1525.95 | 43.15 | 1368.18 | 73.11 | 1391.78 | 108.80 | | 2025 Aug 02 | | M2 Ultra 5c0eb5e + FA | 800 | 76 | 1561.35 | 43.24 | 1386.97 | 73.35 | 1412.42 | 109.41 | --- M1 Pro, 8+2 CPU, 16 GPU (@ggerganov) ✅ | model | size | params | backend | ngl | test | t/s | | --- | --- | --- | --- | --- | --- | --- | | llama 7B mostly F16 | 12.55 GiB | 6.74 B | Metal | 99 | pp 512 | 302.14 ± 0.07 | | llama 7B mostly F16 | 12.55 GiB | 6.74 B | Metal | 99 | tg 128 | 12.75 ± 0.00 | | llama 7B mostly Q8\_0 | 6.67 GiB | 6.74 B | Metal | 99 | pp 512 | 270.37 ± 0.02 | | llama 7B mostly Q8\_0 | 6.67 GiB | 6.74 B | Metal | 99 | tg 128 | 22.34 ± 0.00 | | llama 7B mostly Q4\_0 | 3.56 GiB | 6.74 B | Metal | 99 | pp 512 | 266.25 ± 0.07 | | llama 7B mostly Q4\_0 | 3.56 GiB | 6.74 B | Metal | 99 | tg 128

Ampera A1 for Llama : r/LocalLLaMA - Reddit

Today, I was running llama2 7B Q2_k using llama.cpp and got around 5 tokens/s. I plan on testing bigger models over the next few days.

Running LLAMA.CPP on Ampere Altra Hardware - Medium

[Sign in](https://medium.com/m/signin?operation=login&redirect=https%3A%2F%2Fmedium.com%2F%40ravigupta1989%2Frunning-llama-cpp-on-ampere-altra-hardware-459d05c3dc51&source=post_page---top_nav_layout_nav-----------------------global_nav------------------). [Sign in](https://medium.com/m/signin?operation=login&redirect=https%3A%2F%2Fmedium.com%2F%40ravigupta1989%2Frunning-llama-cpp-on-ampere-altra-hardware-459d05c3dc51&source=post_page---top_nav_layout_nav-----------------------global_nav------------------). [](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fvote%2Fp%2F459d05c3dc51&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40ravigupta1989%2Frunning-llama-cpp-on-ampere-altra-hardware-459d05c3dc51&user=Ravi_Gupta&userId=29e52c21e9c9&source=---header_actions--459d05c3dc51---------------------clap_footer------------------). [](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fbookmark%2Fp%2F459d05c3dc51&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40ravigupta1989%2Frunning-llama-cpp-on-ampere-altra-hardware-459d05c3dc51&source=---header_actions--459d05c3dc51---------------------bookmark_footer------------------). ![Image 3](https://medium.com/@ravigupta1989/running-llama-cpp-on-ampere-altra-hardware-459d05c3dc51). ![Image 4](https://medium.com/@ravigupta1989/running-llama-cpp-on-ampere-altra-hardware-459d05c3dc51). ![Image 5](https://medium.com/@ravigupta1989/running-llama-cpp-on-ampere-altra-hardware-459d05c3dc51). ![Image 6](https://medium.com/@ravigupta1989/running-llama-cpp-on-ampere-altra-hardware-459d05c3dc51). ![Image 7](https://medium.com/@ravigupta1989/running-llama-cpp-on-ampere-altra-hardware-459d05c3dc51). llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from ./models/mistral-7b-instruct-v0.1.Q4_0.gguf (version GGUF V2). llama_model_loader: - kv 1: general.name str = mistralai_mistral-7b-instruct-v0.1. llama_model_loader: - kv 4: llama.block_count u32 = 32. llama_model_loader: - kv 12: tokenizer.ggml.model str = llama. llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<... llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1. llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 2. llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0. Model metadata: {'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '2', 'general.architecture': 'llama', 'llama.rope.freq_base': '10000.000000', 'llama.context_length': '32768', 'general.name': 'mistralai_mistral-7b-instruct-v0.1', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '32', 'llama.attention.head_count_kv': '8', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '2'}. [Llama Cpp](https://medium.com/tag/llama-cpp?source=post_page-----459d05c3dc51---------------------------------------). [](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fvote%2Fp%2F459d05c3dc51&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40ravigupta1989%2Frunning-llama-cpp-on-ampere-altra-hardware-459d05c3dc51&user=Ravi_Gupta&userId=29e52c21e9c9&source=---footer_actions--459d05c3dc51---------------------clap_footer------------------). [](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fvote%2Fp%2F459d05c3dc51&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40ravigupta1989%2Frunning-llama-cpp-on-ampere-altra-hardware-459d05c3dc51&user=Ravi_Gupta&userId=29e52c21e9c9&source=---footer_actions--459d05c3dc51---------------------clap_footer------------------). [](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fbookmark%2Fp%2F459d05c3dc51&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40ravigupta1989%2Frunning-llama-cpp-on-ampere-altra-hardware-459d05c3dc51&source=---footer_actions--459d05c3dc51---------------------bookmark_footer------------------).

Running LLMs locally on Ampere A1 Linux VM: Comparing options

Finally, I will benchmark Docker Model Runner, Ollama and Ampere optimized Ollama, as well as llama.cpp and Ampere optimized llama.cpp using the same `qwen2.5:3B` model with Q4_K_M quantization. It is a user-friendly wrapper around llama.cpp with a lightweight tool for model downloading, quantization, running, and managing LLMs locally. For Ampere, run the optimized Docker container, and from there you can pull and run models with a single command (`ollama run qwen2.5:3B`) which gives you an interactive prompt to start chatting. Nevertheless, llama.cpp edges out others in raw speed due to its C++ implementation and optimization for running LLMs on CPUs. llama.cpp’s quantization reduces model size and speeds up inference without needing GPU tensors. *   **Ampere optimized llama.cpp for performance and advanced customization**: If you are an advanced user who need to maximize performance and control every single inference parameter, the **optimized llama.cpp container** is a fantastic runner-up. | Feature | Ampere Optimized Ollama | Docker Model Runner | llama.cpp | vLLM | HF Transformer |.

LLM inferencing with Arm-based OCI Ampere A1 Compute in OCI ...

# LLM inferencing with Arm-based OCI Ampere A1 Compute in OCI Data Science AI Quick Actions. Oracle Cloud Infrastructure (OCI) Data Science AI Quick Actions now supports inferencing of models in GPT-Generated Unified Format (GGUF) with OCI Ampere A1 CPU shapes. For more information on using A1 Ampere in generative AI (GenAI) applications, read the blog posts, From inference to RAG: Choosing CPUs for efficient generative AI application deployments and Introducing Meta Llama 3 on OCI Ampere A1: A testament to CPU-Based Model Inference. ## GGUF Models in AI Quick Actions. Oracle and Ampere have partnered to optimize llama.cpp to work with Ampere Arm 64 CPUs. In AI Quick Actions, you can perform inferencing of GGUF models with our service-managed container running the OCI Ampere optimized llama.cpp with A1 shapes. Fig 1: A1 configuration options for working with a GGUF model in AI Quick Actions. llama.cpp is in the service managed container for GGUF models in AI Quick Actions.


Self-hosting Qwen2.5-3B for a production app - what's your setup?

Oracle Cloud free tier (4 ARM cores, 24GB RAM) · llama.cpp with Q4_K_M quantization · ~10-15 t/s should be fine for my use case.

llama.cpp - Qwen - Read the Docs

To use llama.cpp means that you use the llama.cpp library in your own program, like writing the source code of Ollama, LM Studio, GPT4ALL, llamafile etc. Instead, here we introduce how to use the `llama-cli` example program, in the hope that you know that llama.cpp does support Qwen2.5 models and how the ecosystem of llama.cpp generally works. In this guide, we will show how to “use” llama.cpp to run models on your local machine, in particular, the `llama-cli` and the `llama-server` example program, which comes with the library. llama.cpp supports Qwen3 and Qwen3MoE from version `b5092`. 2. Second, you can install the pre-built binaries, `llama-cli` and `llama-server` included, with a single command:. llama-cli is a console program which can be used to chat with LLMs. Simple run the following command where you place the llama.cpp programs:. * **Model**: llama-cli supports using model files from local path, remote URL, or Hugging Face hub.

Running qwen 2.5:7b llama 3.2:3b locally? - Facebook

Ollama (LOCAL) Anyone successfully running qwen 2.5:7b + llama 3.2:3b? 👀 I'm running locally just as a last line of defence fallback.

Run FREE Local LLMs in Claude Code with Ollama (Llama, Qwen ...

In this video, I walk you through how to set up Ollama with Claude Code CLI so you can run powerful open-source models like Llama 3.2, Qwen,

Run Qwen 3.5 27B locally with llama.cpp and opencode - YouTube

agentic coding and other agentic use cases, including OpenClaw. To have enough free ... I Tested the Cheapest Path to 96GB of VRAM. Alex


Always Free tier clarification (4x Ampere + 1x Micro) : r/oraclecloud

I think they increased the minimum boot size to 50GB. So if I understood correctly, I can set up 1VM 4xAmpere, 24GB ram, 150GB boot volume + 1

oracle-cloud-free-tier-guide - GitHub Gist

x2 x86 instances (2core/1g) · x2 ampere instances (with 4core/24g spread between them) · 200GB total boot volume space across all intances (

Breaking down the free tier of Oracle Cloud Infrastructure

Oracle Cloud Infrastructure, or OCI for short, has a very generous free tier, but I see a lot of people on the oraclecloud and selfhosted subreddits being confused about exactly what's included, so here is a straightforward breakdown of every service and allotment in the free tier. Different product from the above, this service provides automated traffic distribution from one entry point to multiple servers reachable from your virtual cloud network (VCN), of which you get two in the free tier as you’ll see further below. *Always Free: Up to 10 GB per month shared across OCI Logging services*. *Always Free: One standalone HeatWave instance in OCI in your home region, along with up to 50 GB of storage and up to 50 GB of backup storage*. The free tier includes 200 GB of combined boot volume and block volume storage, across all VMs, whether they are E2 Micro and Ampere A1.

Getting Started with Oracle Cloud Free Tier: Always Free Services ...

Always Free Oracle Cloud Infrastructure services include a virtual machine instance, an Oracle Autonomous Database, and the networking, load


Explanation:
**Extracting Oracle docs**

I need to focus on extracting Oracle documents, but I might also want to grab an article about Ampere optimized benchmarks or possibly a blog. I think I can use a web extraction tool to get these documents along with the relevant blog or article. It seems there's a lot of information to consider, so I'll aim to gather the most useful resources that relate to these benchmarks.


# Oracle Cloud Infrastructure: Always Free Resources Summary

Oracle Cloud Infrastructure (OCI) provides a suite of "Always Free" resources available to all accounts (Free and Paid) for the life of the account. These resources must be provisioned in the tenancy's **home region**.

---

## 🏗 Infrastructure & Compute

### Compute Instances
Users can run up to 4 instances simultaneously depending on configuration.
*   **AMD (Micro):** 2 instances using `VM.Standard.E2.1.Micro` shape.
    *   1/8 OCPU, 1 GB RAM, 50 Mbps network bandwidth.
*   **ARM (Ampere A1):** 3,000 OCPU hours and 18,000 GB hours per month.
    *   Equivalent to **4 OCPUs and 24 GB RAM** total (can be split across multiple VMs).
    *   Requires at least 8 GB RAM for "Oracle Linux Cloud Developer" image.

> [!IMPORTANT]
> **Reclamation of Idle Instances:** Oracle may reclaim Always Free instances if, over 7 days:
> * CPU, Network, and Memory (ARM only) utilization for the 95th percentile are all **less than 20%**.

### Storage
*   **Block Volume:** 200 GB total (combined boot and block volumes) and 5 volume backups.
    *   *Note:* Minimum boot volume is 47-50 GB; launching 4 instances uses the full 200 GB allotment.
*   **Object Storage:** 20 GB total (Standard, Infrequent Access, and Archive tiers combined).
    *   Includes 50,000 API requests per month.
    *   *Warning:* If a Free Trial expires and usage exceeds 20 GB, all objects will be deleted.

### Security & Automation
*   **Certificates:** 5 Certificate Authorities (CAs) and 150 certificates.
*   **Vault:** 20 HSM-protected master key versions, 150 secrets, and unlimited software-protected keys.
*   **Resource Manager:** Automate provisioning via Terraform.
    *   Limits: 100 Stacks, 100 Templates, 2 Concurrent Jobs.

---

## 🗄 Database Services

*   **Oracle Autonomous Database:** 2 instances (1 OCPU and 20 GB storage each).
    *   Workloads: Transaction Processing, Data Warehouse, APEX, or JSON.
*   **Oracle NoSQL:** 3 tables, 25 GB storage per table, 133M reads/writes per month.
*   **Oracle MySQL HeatWave:** 1 standalone DB system (single node), 50 GB storage + 50 GB backup storage.

---

## 🌐 Networking

*   **Load Balancing:**
    *   1 Flexible Load Balancer (10 Mbps fixed bandwidth).
    *   1 Network Load Balancer.
*   **VCN & Connectivity:**
    *   Up to 2 Virtual Cloud Networks (VCNs).
    *   **Site-to-Site VPN:** Up to 50 IPSec connections.
    *   **Data Transfer:** 10 TB per month of outbound data transfer.
*   **Bastion:** Free SSH access to private target resources.

---

## 📊 Observability & Management

| Service | Always Free Limit |
| :--- | :--- |
| **Monitoring** | 500M ingestion / 1B retrieval data points |
| **Notifications** | 1M HTTPS / 1,000 Email notifications per month |
| **Email Delivery** | 3,000 emails per month |
| **Logging** | 10 GB per month (shared across services) |
| **Dashboards** | 100 dashboards per tenancy |
| **Connector Hub** | 2 connectors |
| **Fleet App Mgmt** | First 25 resources (Compute/DB) managed for free |

---

## 🛠 Troubleshooting & Capacity
*   **"Out of host capacity" Error:** This indicates a temporary shortage of Always Free shapes in a specific Availability Domain.
    *   *Action:* Try a different Availability Domain or wait to retry.
*   **Upgrading:** Upgrading to a "Pay As You Go" account does not remove Always Free benefits; you are only charged for usage exceeding the limits.
*   **Checking Limits:** Navigate to **Governance & Administration** → **Tenancy Management** → **Limits, Quotas and Usage** in the OCI Console.

# LLM Inferencing with Arm-based OCI Ampere A1 Compute

Oracle Cloud Infrastructure (OCI) Data Science **AI Quick Actions** now supports inferencing for Large Language Models (LLMs) in **GPT-Generated Unified Format (GGUF)** using **OCI Ampere A1 CPU shapes**. This integration provides a cost-effective alternative to GPUs for LLM inferencing and Retrieval-Augmented Generation (RAG).

## Key Technologies
*   **GGUF:** A file format designed to compress 16-bit floating-point model weights, optimizing computational resource use and simplifying model loading/storage.
*   **Llama.cpp:** An inferencing framework optimized by Oracle and Ampere specifically for Arm 64 CPUs to provide minimal setup and high performance.
*   **OCI Ampere A1:** Arm-based compute shapes that support up to 80 OCPUs, with performance scaling linearly with core count.

## Benchmarking Results
Testing was conducted using the **Phi-3-mini-4k-instruct-q4.gguf** model on an **A1 shape with 40 OCPUs** (Llama.cpp python version 0.2.78).

| Processing Architecture | Prompt Tokens | Output Tokens | Inference Time (Sec) | Tokens Per Second (TPS) |
| :--- | :--- | :--- | :--- | :--- |
| Arm CPU A1 (40 OCPUs) | 10 | 50 | 1.257 | **39.77** |
| Arm CPU A1 (40 OCPUs) | 10 | 100 | 2.51 | **39.84** |
| Arm CPU A1 (40 OCPUs) | 10 | 200 | 5.119 | **39.07** |
| Arm CPU A1 (40 OCPUs) | 20 | 50 | 1.263 | **39.9** |
| Arm CPU A1 (40 OCPUs) | 30 | 200 | 5.227 | **38.27** |

## Implementation Options in AI Quick Actions

Users can deploy GGUF models via two primary methods:

### 1. Service-Cached Models
Oracle provides pre-configured GGUF models that are ready for immediate deployment without downloading artifacts:
*   `Phi-3-mini-4k-instruct-q4.gguf`
*   `Phi-3-mini-4k-instruct-fp16.gguf`

### 2. Bring Your Own Model (BYOM)
Users can import any GGUF model from Hugging Face:
1.  **Register** the model in AI Quick Actions.
2.  **Deploy** by selecting the `llama.cpp` container under the **Inference Container** option in the **Advanced Options** section.

> "Enabling the use of A1 Ampere CPU shapes for (LLM) inferencing and retrieval-augmented generation (RAG) provides more options for our customers to work with LLMs for their use cases."

## Additional Resources
*   [OCI Data Science Sample Repository (GitHub)](https://github.com/oracle-samples/oci-data-science-ai-samples/)
*   [AI Quick Actions Documentation](https://docs.oracle.com/en-us/iaas/data-science/using/ai-quick-actions.htm)
*   **Contact:** ask-oci-data-science_grp@oracle.com

# Running LLMs on Ampere A1 Linux (Oracle Cloud)

This guide evaluates options for running Large Language Models (LLMs) on **Oracle Cloud Infrastructure (OCI) Ampere A1 Flex** instances (specifically 4 ARM-based cores and 24GB RAM).

## 🎯 The Goal & Hardware Constraints
The objective is to achieve efficient, headless local inference on a Linux VM for a Python web app backend.
*   **Hardware:** 4 Ampere A1 Cores, 24GB RAM (No GPU).
*   **Target Performance:** >5 tokens per second (TPS) to match human reading speed.
*   **Memory Availability:** ~20-22GB usable RAM after OS overhead.

---

## 🏆 The Verdict: Ampere-Optimized Ollama
**Ampere-Optimized Ollama** is the recommended choice for most users. It balances simplicity with significant performance gains specifically tuned for the Ampere Altra architecture.

> "It offers the best balance of simplicity (one-command install, automatic model handling), solid CPU performance on ARM, and manageable resource consumption."

### Key Benefits:
*   **Performance:** Specifically compiled to leverage Ampere Altra CPU architectural strengths.
*   **Ease of Use:** Single command to pull and run models (`ollama run qwen2.5:3B`).
*   **Integration:** Built-in OpenAI-compatible API on port `11434` and Python bindings.
*   **Model Management:** Supports "hot swapping" models and uses `Modelfiles` for inference tweaks.

---

## 📊 Comparison of Inference Options

| Tool | Best For | Pros | Cons |
| :--- | :--- | :--- | :--- |
| **Ampere Ollama** | **General Use** | Easiest setup; optimized for A1; model registry. | Higher abstraction overhead than llama.cpp. |
| **llama.cpp** | **Max Performance** | Lowest RAM usage; fastest raw speed; highly customizable. | Requires manual build; manual GGUF conversion. |
| **Docker Model Runner** | **Docker Workflows** | Native host performance; OCI artifact support. | Limited fine-grained control; newer/less tested. |
| **vLLM** | **High Throughput** | PagedAttention; great for concurrent users. | Overkill for single-user; CPU backend is slower. |
| **HF Transformers** | **Research/Dev** | Access to entire HF ecosystem; flexible. | High Python overhead; slowest CPU inference. |

---

## 💡 Model Selection & Performance
To maintain a responsive experience (>5 TPS), quantization is mandatory (specifically **GGUF Q4_K_M** format).

### Recommended Model Sizes:
*   **7B - 8B Models (e.g., Llama 3.1, Mistral, Qwen2.5):** **The Sweet Spot.** Uses ~4-8GB RAM. High speed and plenty of room for context.
*   **13B Models (e.g., Code Llama):** **Good Balance.** Uses ~7-10GB RAM. Better reasoning, still fits comfortably in 24GB.
*   **34B+ Models:** **Not Recommended.** Uses ~15-25GB RAM. Leaves no room for OS/context; extremely slow TTFT (Time to First Token).
*   **70B Models:** **Infeasible.** Exceeds the 24GB RAM limit.

### Performance Estimates (4 Cores):
*   **7B Model:** ~12–25 tokens/sec.
*   **13B Model:** ~8–15 tokens/sec.
*   **TTFT (Time to First Token):** 0.5s – 2.0s for optimized tools.

---

## 🛠 Implementation Snippets

### Running Ampere-Optimized Ollama (Docker)
```bash
# Pull and run the optimized container
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama amperecomputingai/ollama

# Interact via API
curl http://localhost:11434/api/generate -d '{
  "model": "llama3",
  "prompt": "Write a haiku about Ollama"
}'
```

### Running Hugging Face Transformers (Python)
```python
from transformers import pipeline;
pipe = pipeline('text-generation', model='meta-llama/Llama-2-7b-hf', device='cpu')
print(pipe("Write a haiku about Transformers"))
```

### vLLM (ARM64 Build)
```bash
# Start an OpenAI-compatible server
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-7b-hf \
    --tensor-parallel-size 1 \
    --dtype float16
```

---

## 📌 Key Takeaways
1.  **Quantization is Essential:** Use GGUF format with 4-bit quantization to fit models into RAM and speed up CPU math.
2.  **Avoid vLLM for Single Users:** While powerful for production, its CPU backend underperforms compared to `llama.cpp`-based tools on low-core counts.
3.  **Monitor Resources:** Use `htop` or `free -h` to ensure the model doesn't trigger "swapping," which causes performance to collapse.


Sí, **te alcanza**, pero con una salvedad importante: **Oracle Always Free sirve para correr modelos chicos/medianos cuantizados en CPU, no para “un ChatGPT propio” pesado**.

## Qué te da Oracle Always Free
Según la doc de OCI, el tier gratis incluye aprox.:

- **Hasta 4 OCPU ARM (Ampere A1)**
- **24 GB de RAM**
- **200 GB de storage total**
- sin GPU

Eso significa:

- **bien** para inferencia local en **CPU**
- **mal** para modelos grandes o muchas requests concurrentes
- **obligatorio** usar modelos **quantized/GGUF** y runners tipo **llama.cpp** u **Ollama optimizado para Ampere**

## Mi conclusión corta
**Sí, es suficiente si tu objetivo es:**

- tener una API local tuya
- usar un modelo para chat, automatizaciones, clasificación, extracción, RAG liviano
- uno o pocos usuarios
- tolerar velocidades tipo “usable” pero no brutales

**No es suficiente si querés:**

- modelos grandes tipo **70B**
- mucha concurrencia
- latencia muy baja estilo SaaS comercial
- servir algo “pro” a varios usuarios a la vez

---

## Qué modelos sí veo razonables ahí

### Recomendados
Estos me parecen los más sensatos para ese hardware:

- **Qwen2.5 3B Instruct**
- **Qwen2.5 7B Instruct**
- **Llama 3.2 3B**
- **Phi-3 Mini / 3.8B**
- **Mistral 7B Instruct**
- **Gemma 2 2B / 9B** (9B ya más justo, pero posible)

En **GGUF Q4_K_M** o parecido.

### Posibles pero ya más al límite
- **13B-ish** cuantizados
  Entran, pero ya empezás a pagar con:
  - más latencia
  - peor tiempo al primer token
  - menos margen de RAM para contexto y servicios auxiliares

### No
- **34B+**: mala idea
- **70B**: directamente **no**
- cualquier cosa sin cuantizar: tampoco

---

## Rendimiento esperable
Encontré referencias bastante alineadas con esto para un A1 4 cores / 24 GB:

- **7B quantized:** aprox. **12–25 tok/s**
- **13B quantized:** aprox. **8–15 tok/s**
- target de “usable para un humano”: arriba de **5 tok/s**

Eso, en la práctica, significa que un **3B o 7B bien elegido** puede quedar **bastante decente** para uso personal.

Igual te diría algo importante:
**para tareas de agente / tools / automatización, muchas veces un 3B–7B bien afinado rinde mejor de lo que uno imagina** si el prompting está bien y no le pedís magia.

---

## Lo que más te conviene correr ahí
Si yo tuviera que elegir hoy para Oracle Free, haría esto:

### Opción 1 — la más sensata
**Qwen2.5 7B Instruct Q4_K_M** con **llama.cpp** o **Ollama optimizado para Ampere**

Por qué:
- buen equilibrio entre calidad y RAM
- bastante versátil
- razonable para chat, scripting, extracción, tareas backend

### Opción 2 — si querés más velocidad
**Qwen2.5 3B** o **Llama 3.2 3B**

Por qué:
- más rápido
- menos RAM
- mejor para automatizaciones o fallback local

### Opción 3 — si querés coding liviano
**Qwen2.5-Coder 7B** cuantizado
Puede andar, pero no esperes nivel Claude/GPT en programación compleja.

---

## Tooling: qué usaría yo
### Mejor opción
- **llama.cpp**
  - más eficiente en CPU
  - ideal para GGUF
  - muy buena opción en ARM

### Más cómodo
- **Ollama optimizado para Ampere**
  - más simple de instalar y usar
  - API fácil
  - un poco más de overhead, pero práctico

### Lo que evitaría en ese contexto
- **vLLM** en CPU para un solo usuario: no me parece la mejor apuesta
- Transformers “pelado” en Python: más overhead, peor experiencia

---

## Limitaciones reales del Always Free
Hay varias que te conviene tener en cuenta:

### 1. Es ARM, no x86
No todo corre igual de cómodo.
Tenés que elegir tooling compatible con **ARM64**.

### 2. Oracle puede reclamar instancias idle
La doc dice que pueden recuperar instancias Always Free si quedan muy ociosas durante varios días.

### 3. Storage finito
Con 200 GB estás bien, pero no para bajar media Hugging Face.
Un par de modelos GGUF sí; una colección enorme, no.

### 4. Cero GPU
Todo depende de CPU + quantization + paciencia.

---

## Veredicto honesto
**Para uso personal / experimental / backend chico: sí, re sirve.**
**Para reemplazar un servicio comercial potente: no.**

Si me pedís una recomendación concreta:

> **Sí vale la pena** montar una instancia local en Oracle Always Free si querés correr **Qwen2.5 3B/7B, Llama 3.2 3B o Mistral 7B cuantizados** con **llama.cpp/Ollama**.

Es probablemente el **mejor uso posible** de ese tier gratis para LLM local.

Si querés, en el próximo mensaje te hago una **shortlist exacta de 3 modelos** para tu caso de uso:
1. **chat general**
2. **coding**
3. **automatizaciones / agentes**
y te digo cuál elegiría yo en cada uno.


Salió Gemma 4 Hoy


Explanation: The user says "Salió Gemma 4 Hoy" (Gemma 4 came out today). I should check for recent news about Gemma 4 to verify and provide relevant details like model sizes, capabilities, and how it might run on the Oracle Cloud free tier we just discussed.

Question: Gemma 4 announcement April 2026 model sizes specs

Question: Gemma 4 Google release 2026


Google announces Gemma 4 open AI models, switches to Apache ...

# Google announces Gemma 4 open AI models, switches to Apache 2.0 license. Gemma 4 brings the first major update to Google’s open models in a year. Like past versions of its open-weight models, Google has designed Gemma 4 to be usable on local machines. Google says Gemma 31B will debut at number three on the Arena list of top open AI models, behind GLM-5 and Kimi 2.5. Based on the same underlying technology as Google’s Gemini 3 closed models, Gemma 4 offers improved reasoning, math, and instruction-following. Google Pixels and a few other phones run local AI models known as Gemini Nano. A Google representative notes that Gemini Nano has always been derived from Gemma models, but that’s especially true of the next-gen update to Gemini Nano 4. The current Gemini Nano 3 running on Pixel phones is based on Gemma 3n, but Google confirmed to Ars Technica that the next-gen Nano 4 will have 2B and 4B variants based on Gemma 4 E2B and E4B.

Google releases Gemma 4, a family of open models built ... - Engadget

[News](https://www.engadget.com/news/). [Reviews](https://www.engadget.com/reviews/). [Buying guides](https://www.engadget.com/best-tech/). [Gaming](https://www.engadget.com/gaming/). [Big tech](https://www.engadget.com/big-tech/). [Deals](https://www.engadget.com/deals/). [Entertainment](https://www.engadget.com/entertainment/). [More](https://www.engadget.com/ai/google-releases-gemma-4-a-family-of-open-models-built-off-of-gemini-3-160000332.html). *   [New York Auto Show 2026: EV roundup](https://www.engadget.com/transportation/evs/the-hottest-evs-from-the-2026-new-york-auto-show-plus-one-brawny-concept-120000557.html). *   [What’s going on with Donut Lab'?](https://www.engadget.com/transportation/evs/whats-going-on-with-donut-lab-173007121.html). *   [Apple AirPods Max 2 review](https://www.engadget.com/audio/headphones/apple-airpods-max-2-review-better-late-than-never-130000982.html). *   [WWDC 2026: What to expect](https://www.engadget.com/big-tech/what-to-expect-from-wwdc-2026-110000086.html). *   [March Madness 2026](https://www.engadget.com/entertainment/streaming/march-madness-2026-how-to-watch-the-sweet-16-and-elite-eight-this-weekend-154903032.html). 1.   [](https://www.engadget.com/ai/google-releases-gemma-4-a-family-of-open-models-built-off-of-gemini-3-160000332.html#). 2.   [AI](https://www.engadget.com/ai/). [Igor Bonifacic](https://www.engadget.com/about/editors/igor-bonifacic/). [Add Engadget on Google](https://www.google.com/preferences/source?q=engadget.com). When Google released [Gemini 3 Pro](https://www.engadget.com/ai/googles-new-gemini-3-model-arrives-in-ai-mode-and-the-gemini-app-160054273.html) at the end of last year, it was a significant step forward for the company's proprietary large language models. *   [Privacy Dashboard](https://guce.engadget.com/privacy-dashboard?locale=en-US). [Engadget masthead](https://www.engadget.com/about/). [About our ads](https://www.engadget.com/about-our-advertising/). [Advertise](https://www.engadget.com/about/advertise/). [Licensing](https://info.wrightsmedia.com/engadget-licensing). [FAQ](https://www.engadget.com/about-faq-194611880.html). [RSS feed](https://www.engadget.com/rss.xml). [Reviews](https://www.engadget.com/reviews/). [Gear](https://www.engadget.com/gear/). [Gaming](https://www.engadget.com/gaming/). [Entertainment](https://www.engadget.com/entertainment/). [Tomorrow](https://www.engadget.com/tomorrow/). [Buying guides](https://www.engadget.com/best-tech/). [Video](https://www.engadget.com/videos/). [Podcasts](https://www.engadget.com/podcasts/). [Deals](https://www.engadget.com/deals/). [Comment guidelines](https://www.engadget.com/2017-05-01-engadget-commenting-policy.html). [Support](https://www.engadget.com/support/). [Best laptop](https://www.engadget.com/computing/laptops/best-laptops-120008636.html). [The best iPad](https://www.engadget.com/mobile/tablets/best-ipads-how-to-pick-the-best-apple-tablet-for-you-150054066.html). [Best Bluetooth speaker](https://www.engadget.com/audio/speakers/best-portable-bluetooth-speakers-133004551.html). [Best E Ink tablets](https://www.engadget.com/mobile/tablets/best-e-ink-tablet-130037939.html). [Best wireless earbuds](https://www.engadget.com/audio/headphones/best-wireless-earbuds-120058222.html). [Best power banks](https://www.engadget.com/computing/accessories/best-power-bank-143048526.html). [Best gaming handhelds](https://www.engadget.com/gaming/best-handheld-gaming-system-140018863.html). [Comment guidelines](https://www.engadget.com/2017-05-01-engadget-commenting-policy.html). [Support](https://www.engadget.com/support/). [Best laptop](https://www.engadget.com/computing/laptops/best-laptops-120008636.html). [The best iPad](https://www.engadget.com/mobile/tablets/best-ipads-how-to-pick-the-best-apple-tablet-for-you-150054066.html). [Best Bluetooth speaker](https://www.engadget.com/audio/speakers/best-portable-bluetooth-speakers-133004551.html). [Best E Ink tablets](https://www.engadget.com/mobile/tablets/best-e-ink-tablet-130037939.html). [Best wireless earbuds](https://www.engadget.com/audio/headphones/best-wireless-earbuds-120058222.html). [Best power banks](https://www.engadget.com/computing/accessories/best-power-bank-143048526.html). [Best gaming handhelds](https://www.engadget.com/gaming/best-handheld-gaming-system-140018863.html). [](https://www.facebook.com/Engadget/)[](https://twitter.com/engadget/)[](https://www.instagram.com/engadget/). [About Us](https://legal.yahoo.com/us/en/yahoo/privacy/guce/faq/index.html)[Reprints and Permissions](https://info.wrightsmedia.com/engadget-licensing)[](https://www.engadget.com/ai/google-releases-gemma-4-a-family-of-open-models-built-off-of-gemini-3-160000332.html)[Trademarks](https://legal.aol.com/trademarks/)[Advertise](https://www.engadget.com/about/advertise/)[About Our Ads](https://legal.yahoo.com/us/en/yahoo/privacy/adinfo/index.html). [Terms](https://guce.engadget.com/terms?locale=en-US). [Privacy Policy](https://guce.engadget.com/privacy-policy?locale=en-US).

Gemma 4 - Models - Google DeepMind

[![Image 7](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__genie-3__large_OKty371.svg) Genie 3 Generate and explore interactive worlds](https://deepmind.google/models/genie/)[![Image 8](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__gemini-robotics__large.svg) Gemini Robotics Perceive, reason, use tools and interact](https://deepmind.google/models/gemini-robotics/). [![Image 9](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__gemma__large.svg) Gemma Build responsible AI applications at scale](https://deepmind.google/models/gemma/). [![Image 10](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__sima-2__large.svg) SIMA 2 An agent that plays, reasons, and learns with you](https://deepmind.google/blog/sima-2-an-agent-that-plays-reasons-and-learns-with-you-in-virtual-3d-worlds/)[![Image 11](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__genie-3__large_OKty371.svg) Genie 3 Generate and explore interactive worlds](https://deepmind.google/models/genie/)[![Image 12](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__alphago__large.svg) AlphaGo Mastering the game of Go](https://deepmind.google/research/alphago/)[![Image 13](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__gemini-robotics__large.svg) Gemini Robotics Perceive, reason, use tools and interact](https://deepmind.google/models/gemini-robotics/). [![Image 17](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__alphaearth__large.svg) AlphaEarth Foundations Map our planet in unprecedented detail](https://deepmind.google/blog/alphaearth-foundations-helps-map-our-planet-in-unprecedented-detail/)[![Image 18](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__weathernext__large.svg) WeatherNext Fast and accurate AI weather forecasting](https://deepmind.google/science/weathernext/)[![Image 19](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__weather-lab__large.svg) Weather Lab Test our experimental weather models](https://deepmind.google.com/science/weatherlab/?utm_source=deepmind.google&utm_medium=referral&utm_campaign=gdm&utm_content=). [![Image 26](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__genie-3__large_OKty371.svg) Genie 3 Generate and explore interactive worlds](https://deepmind.google/models/genie/)[![Image 27](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__gemini-robotics__large.svg) Gemini Robotics Perceive, reason, use tools and interact](https://deepmind.google/models/gemini-robotics/). [![Image 28](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__gemma__large.svg) Gemma Build responsible AI applications at scale](https://deepmind.google/models/gemma/). [![Image 29](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__sima-2__large.svg) SIMA 2 An agent that plays, reasons, and learns with you](https://deepmind.google/blog/sima-2-an-agent-that-plays-reasons-and-learns-with-you-in-virtual-3d-worlds/)[![Image 30](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__genie-3__large_OKty371.svg) Genie 3 Generate and explore interactive worlds](https://deepmind.google/models/genie/)[![Image 31](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__alphago__large.svg) AlphaGo Mastering the game of Go](https://deepmind.google/research/alphago/)[![Image 32](https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/original_images/nav__dm__gemini-robotics__large.svg) Gemini Robotics Perceive, reason, use tools and interact](https://deepmind.google/models/gemini-robotics/). [Try in Google AI Studio](https://aistudio.google.com/prompts/new_chat?model=gemma-4-31b-it&utm_source=deepmind.google&utm_medium=referral&utm_campaign=gdm&utm_content=).

Gemma 4 model card | Google AI for Developers

# Gemma 4 model card. multimodal, handling text and image input (with audio supported on small models). window, while the medium models support 256K. ### Dense Models. Gemma 4 models handle a broad range of tasks across text, vision, and audio. You can use all Gemma 4 models with the latest version of Transformers. MODEL_ID = "google/gemma-4-E2B-it". Once the model is loaded, you can start generating output:. outputs = model.generate(**inputs, max_new_tokens=1024). Compared to Gemma 3, the models use standard `system`, `assistant`, and `user`. All models support image inputs and can process videos as frames whereas the E2B. and E4B models also support audio inputs. ## **Model Data**. Data used for model training and how the data was processed. Gemma 4 models were developed in partnership with internal safety and. content safety relative to previous Gemma models. significantly outperform Gemma 3 and 3n models in improving safety, while. Multimodal models (capable of processing vision, language, and/or audio) have a.

Google releases Gemma 4 under Apache 2.0 - VentureBeat

As some Chinese AI labs (most notably Alibaba’s latest Qwen models, Qwen3.5 Omni and Qwen 3.6 Plus) have begun pulling back from fully open releases for their latest models, Google is moving in the opposite direction — opening up its most capable Gemma release yet while explicitly stating the architecture draws from its commercial **Gemini 3** research. The "workstation" tier includes a **31B-parameter dense model** and a **26B A4B Mixture-of-Experts model** — both supporting text and image input with 256K-token context windows. For teams building voice-first applications that need to keep data local — think healthcare, field service, or multilingual customer interaction — running ASR, translation, reasoning, and function calling in a single model on a phone or edge device is a genuine architectural simplification. What distinguishes Gemma 4 is less any single benchmark and more the combination: strong reasoning, native multimodality across text, vision, and audio, function calling, 256K context, and a genuinely permissive license — all in a single model family with deployment options from edge devices to cloud serverless.


Google unveils Gemma 4 models, aimed at advanced reasoning ...

Book with Early 2026 Deals and stay until April. Sponsored. In ... Google releases Gemma 4, a family of open models built off of Gemini 3.

The latest AI news we announced in March 2026 - Google Blog

[Skip to main content](https://blog.google/innovation-and-ai/technology/ai/google-ai-updates-march-2026/#jump-content). *   [Gemini models](https://blog.google/innovation-and-ai/models-and-research/gemini-models/). *   [See all](https://blog.google/innovation-and-ai/models-and-research/). *   [Gemini app](https://blog.google/innovation-and-ai/products/gemini-app/). *   [See all](https://blog.google/innovation-and-ai/products/). *   [See all](https://blog.google/innovation-and-ai/infrastructure-and-cloud/). [See all AI updates](https://blog.google/innovation-and-ai/). *   [Gemini models](https://blog.google/innovation-and-ai/models-and-research/gemini-models/). *   [Gemini app](https://blog.google/innovation-and-ai/products/gemini-app/). [See all](https://blog.google/innovation-and-ai/products/ "See all Products articles"). [See all AI updates](https://blog.google/innovation-and-ai/). *   [Health](https://blog.google/innovation-and-ai/technology/health/). *   [Health](https://blog.google/innovation-and-ai/technology/health/). *   [How is Gemini changing Maps?](https://blog.google/innovation-and-ai/technology/ai/google-ai-updates-march-2026/#). *   [Gemini models](https://blog.google/innovation-and-ai/models-and-research/gemini-models/). *   [See all](https://blog.google/innovation-and-ai/models-and-research/). *   [Gemini app](https://blog.google/innovation-and-ai/products/gemini-app/). *   [See all](https://blog.google/innovation-and-ai/products/). *   [See all](https://blog.google/innovation-and-ai/infrastructure-and-cloud/). [See all AI updates](https://blog.google/innovation-and-ai/). *   [Health](https://blog.google/innovation-and-ai/technology/health/). 3.   [Technology](https://blog.google/innovation-and-ai/technology/). [**We introduced tools to help people switch to Gemini**](https://blog.google/innovation-and-ai/products/gemini-app/switch-to-gemini-app/). [**We released new AI features for your devices in the March Pixel Drop**](https://blog.google/products-and-platforms/devices/pixel/march-2026-pixel-drop/). *   [Health](https://blog.google/innovation-and-ai/technology/health/). [![Image 13](https://storage.googleapis.com/gweb-uniblog-publish-prod/images/gemma-4_blog_keyword_header-dark.width-300.format-webp.webp) Developer tools #### Gemma 4: Byte for byte, the most capable open models By Clement Farabet & Olivier Lacombe Apr 02, 2026](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/). [![Image 14](https://storage.googleapis.com/gweb-uniblog-publish-prod/images/cost_reliability_Gemini_API-socia.width-300.format-webp.webp) Developer tools #### New ways to balance cost and reliability in the Gemini API By Lucia Loher & Hussein Hassan Harrirou Apr 02, 2026](https://blog.google/innovation-and-ai/technology/developers-tools/introducing-flex-and-priority-inference/). By Trey Nguyen Apr 01, 2026](https://blog.google/innovation-and-ai/technology/developers-tools/gemini-api-docsmcp-agent-skills/).

Google's Gemma 4 model goes fully open-source and ... - ZDNET

# Google's Gemma 4 model goes fully open-source and unlocks powerful local AI - even on phones. Google's Gemma 4 model goes fully open-source and unlocks powerful local AI - even on phones. Google announced today that its DeepMind AI research division is releasing Gemma 4, its latest generation of open large language models. The models are being released under the Apache 2.0 license, making them truly open source compared to the permissive but still controlled license of earlier Gemma generations. The difference is that Gemini is a subscription-based closed product, whereas Gemma is an open model that can be downloaded and run locally for free. Instead, they're licensing Gemma 4 under the Apache 2.0 license, which means users and developers can use and distribute the model in *any* way they want without restrictions. Not only can we expect to see Gemma 4 adopted in more projects, but it's also now legitimately possible to bundle the AI with products, services, and devices that can benefit from a powerful on-board model.

Google's Gemma 4 Surfaces Online Ahead of Official Release

Google DeepMind's next open-source AI model — Gemma 4 — has been spotted in LMSYS Chatbot Arena before an official announcement.


Explanation: Let me get more details on the specific Gemma 4 model sizes to assess which ones would fit on the Oracle Always Free tier (24GB RAM, 4 ARM cores).

Question: Gemma 4 E2B E4B 31B model sizes parameters details 2026


# Google Announces Gemma 4: Open Models & Apache 2.0 License

Google has unveiled **Gemma 4**, the first major update to its open-weight model family in over a year. This release marks a significant shift in strategy, moving away from restrictive custom licenses toward the industry-standard **Apache 2.0 license**.

## 1. The Gemma 4 Model Lineup
Gemma 4 is designed for local execution across a range of hardware, from high-end GPUs to mobile devices.

| Model Variant | Size/Type | Target Hardware | Key Feature |
| :--- | :--- | :--- | :--- |
| **31B Dense** | 31B Parameters | 80GB Nvidia H100 (unquantized) | Highest quality; optimized for fine-tuning. |
| **26B MoE** | 26B (3.8B active) | 80GB Nvidia H100 (unquantized) | High tokens-per-second via Mixture of Experts. |
| **Effective 4B (E4B)** | 4B Parameters | Mobile, Raspberry Pi, Jetson Nano | Balanced performance for edge devices. |
| **Effective 2B (E2B)** | 2B Parameters | Mobile, Raspberry Pi, Jetson Nano | Lowest memory/battery usage; "near-zero" latency. |

> **Note on Hardware:** While the larger models are designed for the $20,000 H100 in bfloat16 format, they can fit on consumer GPUs if quantized to lower precision.

## 2. Performance & Capabilities
Gemma 4 is built on the same technology as the closed **Gemini 3** models, offering improvements in reasoning, math, and instruction-following.

*   **Benchmark Ranking:** The 31B model is expected to debut at **#3 on the Arena list** of open models, trailing only GLM-5 and Kimi 2.5 (both of which are significantly larger).
*   **Context Windows:**
    *   **Edge Models (2B/4B):** 128k tokens.
    *   **Large Models (26B/31B):** 256k tokens.
*   **Agentic Workflows:** Native support for function calling, structured JSON output, and API instructions.
*   **Multimodal & Coding:** Optimized for offline code generation, OCR, chart understanding, and native speech recognition.
*   **Language Support:** Works in over 140 languages.

## 3. The Licensing Shift: Apache 2.0
In a major win for developers, Google has abandoned its restrictive custom license in favor of **Apache 2.0**.

*   **Previous Issues:** The Gemma 3 license allowed Google to update terms unilaterally, restricted commercial use, and had ambiguous clauses regarding synthetic data ownership.
*   **Apache 2.0 Benefits:** Permissive, familiar to developers, no overbearing terms of use, and no commercial restrictions.

## 4. Future of Mobile: Gemini Nano 4
Google confirmed that **Gemini Nano 4** (the on-device AI for Pixel phones) will be derived directly from the Gemma 4 E2B and E4B models.
*   **Forward Compatibility:** Developers can prototype agentic workflows in the **AI Core Developer Preview** using Gemma E2B/E4B today; these will be compatible with Nano 4 upon its official launch.

## 5. Availability
The models are available immediately through the following channels:
*   **Testing:** [AI Studio](https://aistudio.google.com/) (31B and 26B MoE) and [AI Edge Gallery](https://ai.google.dev/edge) (E4B and E2B).
*   **Downloads:** [Hugging Face](https://huggingface.co/collections/google/gemma-4), [Kaggle](https://www.kaggle.com/models/google/gemma-4), and [Ollama](https://ollama.com/library/gemma4).
*   **Cloud:** Available for deployment on Google Cloud (paid).

# Gemma 4 Model Card Summary

Gemma 4 is a family of multimodal open models developed by **Google DeepMind**. Built on both Dense and Mixture-of-Experts (MoE) architectures, these models handle text and image inputs (with audio on smaller variants) and feature a context window of up to **256K tokens** with support for **140+ languages**.

## **Key Architectural Advancements**
*   **Reasoning:** Configurable "thinking modes" for complex problem-solving.
*   **Multimodality:** Native support for Text and Image (all models), plus Video and Audio (E2B and E4B models).
*   **Hybrid Attention:** Interleaves local sliding window attention with global attention for speed and memory efficiency.
*   **Per-Layer Embeddings (PLE):** Used in smaller models to maximize parameter efficiency for on-device deployment.
*   **Native System Prompt:** Introduces the `system` role for structured conversation control.

---

## **Model Variants & Specifications**

### **Dense Models**
| Property | E2B | E4B | 31B Dense |
| :--- | :--- | :--- | :--- |
| **Effective Params** | 2.3B (5.1B total) | 4.5B (8B total) | 30.7B |
| **Context Length** | 128K tokens | 128K tokens | 256K tokens |
| **Modalities** | Text, Image, Audio | Text, Image, Audio | Text, Image |
| **Target Device** | High-end phones | Laptops | Workstations/Servers |

### **Mixture-of-Experts (MoE) Model**
*   **Model:** **26B A4B MoE**
*   **Active Parameters:** 3.8B (runs at the speed of a 4B model).
*   **Total Parameters:** 25.2B.
*   **Experts:** 8 active / 128 total + 1 shared.
*   **Context:** 256K tokens.

---

## **Benchmark Performance Highlights**
Gemma 4 shows significant leaps over Gemma 3, particularly in reasoning and coding.

| Metric | Gemma 4 31B | Gemma 4 26B A4B | Gemma 3 27B |
| :--- | :--- | :--- | :--- |
| **MMLU Pro** | 85.2% | 82.6% | 67.6% |
| **AIME 2026 (no tools)** | 89.2% | 88.3% | 20.8% |
| **LiveCodeBench v6** | 80.0% | 77.1% | 29.1% |
| **Codeforces ELO** | 2150 | 1718 | 110 |
| **MATH-Vision** | 85.6% | 82.4% | 46.0% |

---

## **Implementation & Code Snippets**

### **Installation**
```bash
pip install -U transformers torch accelerate
```

### **Basic Usage (Text Generation)**
```python
import torch
from transformers import AutoProcessor, AutoModelForCausalLM

MODEL_ID = "google/gemma-4-E2B-it"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16, device_map="auto")

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write a short joke about saving RAM."},
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=True)
inputs = processor(text=text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024)
```

---

## **Best Practices & Configurations**

### **1. Sampling Parameters**
For optimal performance, use these standardized settings:
*   `temperature=1.0`
*   `top_p=0.95`
*   `top_k=64`

### **2. Thinking Mode**
To enable reasoning, set `enable_thinking=True`. The model uses specific control tokens:
*   `<|think|>`
*   `<|channel>thought\n`
*   `<channel|>`

### **3. Multimodal Constraints**
*   **Audio:** Maximum length of **30 seconds**.
*   **Video:** Maximum length of **60 seconds** (processed at 1 frame per second).
*   **Image Resolution:** Supports variable resolution via a "visual token budget" to balance detail vs. compute.

---

## **Ethics, Safety, and Data**
*   **Training Data:** Diverse collection of web docs, code, images, and audio with a cutoff of **January 2025**.
*   **Safety:** Undergoes the same rigorous evaluations as Gemini models. Gemma 4 shows "major improvements" in content safety and lower unjustified refusal rates compared to Gemma 3.
*   **License:** [Apache 2.0](https://ai.google.dev/gemma/docs/gemma_4_license).
*   **Limitations:** Users should be aware of potential hallucinations, the need for task-specific fine-tuning, and inherent biases in large-scale datasets.


What Is Google Gemma 4? Architecture, Benchmarks, and Why It ...

Google Gemma 4 is the most capable open model family from DeepMind yet, shipping four sizes under Apache 2.0 with multimodal input, native reasoning, and on-device deployment down to a Raspberry Pi. #gemma-4  #google  #open-source  #multimodal  #on-device-ai. On April 2, 2026, Google DeepMind released **Gemma 4** — four open-weight models built from the same research behind Gemini 3, now available under the **Apache 2.0 license**. On LMArena’s text-only leaderboard, the 31B ranks #3 globally among open models with an ELO of ~1452. Every Gemma 4 model supports multimodal input out of the box:. * **Hugging Face**: `google/gemma-4-31B-it`, `google/gemma-4-26B-A4B-it`, `google/gemma-4-E4B-it`, `google/gemma-4-E2B-it`. Gemma 4 represents a serious move from Google in the open model space. The 31B dense model competes with models many times its size on reasoning and coding benchmarks. ### daVinci-MagiHuman: The Open-Source Model That Just Crushed Every Digital Human Generator### Introducing daVinci MagiHuman Image-to-Video on WaveSpeedAI### Introducing daVinci MagiHuman Text-to-Video on WaveSpeedAIIntroducing Google Lyria 3 Pro on WaveSpeedAI.

Google's Gemma 4 is now available with Apache 2.0 licensing for ...

**Google is releasing Gemma 4, its most capable open model family yet. Gemma 4 comes in four sizes: Effective 2B (E2B), Effective 4B (E4B), a 26B Mixture-of-Experts (MoE) model, and a 31B Dense model. The 31B model currently sits at 3rd place among all open models worldwide on the Arena AI Text Leaderboard, while the 26B MoE model ranks 6th. Google says Gemma 4 outperforms models 20 times its size. The 26B MoE model only activates 3.8 billion of its parameters during inference, which should make for especially fast token generation. Independent benchmarks from Artificial Analysis back up the numbers for the larger Gemma 4 models. According to Artificial Analysis, that's the second-best result among all open models with fewer than 40 billion parameters, just behind Qwen3.5 27B at 85.8 percent. Google AI Studio supports the 31B and 26B models, while Google AI Edge Gallery handles the E4B and E2B variants.

Gemma 4 - How to Run Locally | Unsloth Documentation

Run Google’s new Gemma 4 models locally, including E2B, E4B, 26B A4B, and 31B. Gemma 4 is Google DeepMind’s new family of open models, including **E2B**, **E4B**, **26B-A4B**, and **31B.** These multimodal, hybrid-thinking models support 140+ languages, up to **256K context**, and come in both dense and MoE variants. Gemma 4 can now be run and fine-tuned in Unsloth Studio, our new open-source web UI for local AI. export LLAMA_CACHE="unsloth/gemma-4-26B-A4B-GGUF" export LLAMA_CACHE="unsloth/gemma-4-26B-A4B-GGUF" ./llama.cpp/llama-cli \ ./llama.cpp/llama-cli \  -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL \  -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL \  --temp 1.0 \  --temp 1.0 \  --top-p 0.95 \  --top-p 0.95 \  --top-k 64 --top-k 64. hf download unsloth/gemma-4-26B-A4B-it-GGUF \ hf download unsloth/gemma-4-26B-A4B-it-GGUF \  --local-dir unsloth/gemma-4-26B-A4B-it-GGUF \  --local-dir unsloth/gemma-4-26B-A4B-it-GGUF \  --include "*mmproj-BF16*" \  --include "*mmproj-BF16*" \  --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit. ./llama.cpp/llama-cli \ ./llama.cpp/llama-cli \  --model unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \  --model unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \  --mmproj unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf \  --mmproj unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf \  --temp 1.0 \  --temp 1.0 \  --top-p 0.95 \  --top-p 0.95 \  --top-k 64 --top-k 64.

Google's Gemma 4 Arrives in Four Sizes With a License Developers ...

The new Gemma models are available immediately in AI Studio (31B and 26B MoE) and AI Edge Gallery (E4B and E2B). Full model weights can be

Gemma 4 - Models - Google DeepMind


Sí, salió el 2 de abril. Está buenísimo. Resumen:

## Gemma 4 — Lo importante

Google DeepMind lanzó **4 variantes**, y cambió la licencia a **Apache 2.0** (antes era restrictiva), ahora es *realmente* open source:

| Modelo | Tipo | Parámetros activos | Contexto | Pensado para |
|---|---|---|---|---|
| **E2B** | Dense | 2.3B effective (5.1B total) | 128K | Celulares / edge |
| **E4B** | Dense | 4.5B effective (8B total) | 128K | Laptops |
| **26B A4B** | **MoE** | **3.8B activos** (25.2B total) | 256K | Workstation |
| **31B** | Dense | 30.7B | 256K | Servidor / GPU |

Benchmarks respetables: el 31B es **#3 abierto en LMSYS**, y el **26B MoE** corre a velocidad de modelo 4B porque solo activa 128 expertos (usa ~3.8B parámetros por token). Soporta texto, imagen, audio (E2B/E4B), reasoning mode nativo, function calling, y **140+ idiomas**.

---

## ¿Y en tu Oracle Always Free (4 ARM, 24 GB)?

Acá es donde se pone interesante:

### E2B / E4B → ✅ Sí, perfecto
En GGUF cuantizado (Q4_K_M o Q8):
- **~2-4 GB de RAM**
- **buena velocidad** en CPU ARM
- ideales para chat, agente local, automatizaciones
- son **nativamente multimodales** (texto + imagen + audio en E2B/E4B)

### 26B A4B (MoE) → ✅ Es la verdadera joya
Acá la clave: es **Mixture of Experts**, o sea que en inferencia **solo corre con 3.8B parámetros activos**, pero tiene la "inteligencia" de 26B guardados.

En GGUF Q4_K_M:
- **~14-16 GB de RAM** (entran cómodos en 24 GB)
- **rendimiento tipo 4B en velocidad** pero calidad mucho mayor
- **256K contexto**
- sería el **mejor modelo por lejos** que podés correr en Oracle Always Free

El único caveat: MoE en CPU puro va a ser más lento que un dense 4B en tokens/s, pero sigue siendo usable. Y en la práctica, la calidad del 26B A4B **debería aplastar** a cualquier 7B dense actual.

### 31B Dense → 🟡 Posible pero justo
En Q4_K_M: **~17-19 GB de RAM**. Entra, pero deja menos margen para contexto. Y al ser dense, toda la computación pesa más. No me parece el mejor uso del hardware.

---

## Mi recomendación para tu Oracle Free

**1. Gemma 4 E4B** — para uso general, rápido, estable
**2. Gemma 4 26B A4B** — para calidad máxima con MoE, es la verdadera joya del release para tu hardware
**3. Gemma 4 E2B** — como fallback ultra liviano

Yo pondría mi energía en el **26B A4B quantizado en llama.cpp**. Es el que más juego da: calidad de modelo grande con requerimientos que un CPU de 4 cores puede manejar razonablemente.

¿Querés que lo instalemos en esta máquina para probarlo? Ya tenés el server ARM montado, le tiramos Ollama o llama.cpp y lo bajamos.

Explanation: Now I have the full picture on Gemma 4. Let me compile a response about how it fits with Oracle Always Free.


Nah solo de curioso. Por ahora seguiré usando providers.


Perfecto, tiene todo el sentido. Los providers te dan modelos potentes sin tener que pelear con RAM y tok/s en CPU.

Avísame cuando quieras meterle mano a algo, Chicho 👋

Explanation:
The user is saying they're just curious for now and will keep using API providers instead of self-hosting.
They're not asking for anything else to be done.
I should acknowledge this briefly and move on.
No need for tools. Just a short, direct response per their preferences.


Che estoy tratando de que hables  en estr canal privado de discord pero no se si no te llegan mis msjs o si no funcionas en canales privados o no se https://discord.com/channels/1483873336718135336/1489503144290422914


Probablemente es eso: **yo no puedo “ver” ni leer canales privados de Discord por mi cuenta**, aunque me pases el link.

Solo puedo responder en:
- **este chat donde me estás escribiendo ahora**, o
- algún canal/grupo **si la integración realmente me entrega los mensajes de ese lugar**

O sea, el problema no es “canal privado sí/no” en abstracto, sino **si Hermes/Pipo está conectado a ese canal y recibe eventos de ahí**.

## Lo importante
- Ese link **no me da acceso**
- Yo **no puedo abrir Discord internamente** ni revisar historial/canales
- Si escribís en un canal privado y no me llega acá, entonces **esa conversación no está conectada a mi runtime**

## Cómo probarlo rápido
Hacé una de estas:

1. **Mandame un mensaje ahí mencionándome explícitamente**
   - algo tipo: `@Pipo decí hola`

2. Si no responde:
   - revisá si el bot/integración **está agregado a ese canal privado**
   - revisá permisos:
     - ver canal
     - leer mensajes
     - enviar mensajes
     - leer historial
     - usar menciones / responder en threads si aplica

3. Si querés aislar si el problema es el canal:
   - probá en **otro canal donde sabés que sí tiene acceso**
   - o mandame desde ese privado algo muy simple y fijate si aparece

## Mi sospecha
Lo más probable es:
- **el bot no fue agregado a ese canal privado**, o
- **no tiene permisos para verlo/responder**, o
- esa superficie de Discord **no está conectada** a Hermes

Si querés, te ayudo a diagnosticarlo paso a paso y armamos una checklist corta de permisos/config para ese canal privado.

Explanation:
**Clarifying limitations**

I need to provide a direct answer to the user about their question regarding a private channel. I should mention that I can't access Discord’s specific APIs or search through channel history, so I can only read messages sent directly to me. It's also important to clarify that just providing a link doesn't grant me access. Maybe I could suggest the user test it by trying to mention or invite the bot/app in their DM. I should be careful in how I phrase this!