The idea
Hermes is my answer to the disposable AI chat: a JARVIS of my own that remembers what matters and helps with real day-to-day jobs. I built and run it end to end on my own infrastructure, so research, reminders, personal services and urgent alerts all meet in one private conversation.
Technical design goal
The challenge was to make an agent useful beyond one chat while keeping its data, actions and running costs under my control. That led to a self-hosted, message-first design with durable memory, isolated jobs and deliberate model routing.
What runs under the hood
Six layers sit between a message and an answer: the messaging interface, the agent core that holds memory and skills, the scheduler, a research pipeline, personal-ops integrations, and the infrastructure layer that keeps the whole thing alive.
Interface & Messaging
- Matrix-first control: A self-hosted Synapse server is the primary channel, holding long-term session context so the conversation never resets.
- Voice in and out: Groq transcribes voice notes for free; Piper TTS speaks answers back when reading is inconvenient.
- ntfy push: Time-sensitive alerts and attachments are pushed straight to my phone through a self-hosted ntfy instance.
- Subagent delegation: Heavy tasks run in isolated worker sessions, keeping the main conversation's context clean.
Agent Core & Memory
- Persistent memory vault: Searchable Markdown notes with YAML frontmatter are pushed to a personal git server - durable long-term memory, not just chat history.
- Self-healing skills: 80+ reusable procedure files that auto-patch when a step drifts, so a skill that breaks once gets fixed for good.
- Model Context Protocol: MCP integrations bridge the agent to external tools and private homelab services behind a clean interface.
Scheduled Automation
- Isolated cron sessions: Each scheduled job runs in a clean session - collect, summarize, deliver - with no tangled state machines.
- Cost-aware routing: Cheap models handle routine watchdog checks; frontier models are reserved strictly for deep synthesis.
- Chained pipelines: A notifier job consumes the structured summary of a collector job, so workflows compose without rewrites.
Research & Knowledge
- Autonomous deep research: Hermes rewrites the query, fetches pages, and synthesizes a cited answer instead of dumping links.
- Beyond plain web: Extractors pull YouTube/TikTok transcripts and Twitter threads into the same analysis.
- Results that last: Every report lands in the structured Markdown vault, referenceable from any later session.
Personal Operations
- Inbox & calendar: Hermes checks email, parses events, and turns natural-language requests into reminders and tasks.
- Real transport, not a demo: Reverse-engineered Mobiliteit.lu endpoints return live route options for the commutes I actually take.
Infrastructure, Runtime & Recovery
- Multi-node watchdog: Monitors CPU, RAM, temperature and container state, with deterministic health scoring so an LLM never invents an outage.
- Security triage: Correlates CrowdSec logs to surface what's worth my attention from the noise.
- Contained & recoverable: Runs in Docker with persistent volumes and least-privilege tooling; git history makes any agent action inspectable and revertible.
Why I built it
Honestly, it started with curiosity and a sci-fi itch - I wanted a JARVIS of my own. Seeing a friend self-host his own agent was the nudge: the timing was right, so I rented a box and put Hermes on it to see how far I could take it.
Hermes isn't my coding setup - I have plenty of agents for that. It fills a different gap: instead of opening a throwaway ChatGPT tab that forgets me by tomorrow, I just ask Hermes. It remembers either way, holds context across days, and can act on my own services.
What matters most is that it's mine - not a rented product from Claude or any other provider. I own the data, the infrastructure and the skills, and I decide what it's allowed to do. The AI model is the only piece I can't self-host yet, and that doesn't change whose agent this is.
Then one uncompacted session loop on pay-as-you-go cost me €107 in a single week. That bill taught me cost-routing and session hygiene the hard way - and turned model routing into a core discipline of the project.
What it taught me
Memory is the moat
A chatbot without memory is a search engine with extra steps. Persistent context is what makes Hermes feel like an assistant.
Cheap beats clever
Don't spend a frontier model on "is Docker up?". Cost control beats raw model quality for everyday work.
Skills compound
A one-shot prompt is disposable. A skill that gets patched when it breaks gets better every time it runs.
Own the whole stack
Self-hosting means no managed service to blame - you learn the infrastructure, networking and secrets for real.
Isolate heavy work
Pushing big tasks into worker subagents keeps the main conversation focused and stops context from flooding.
Keep cron boring
Collect, summarize, deliver. The most reliable automation pipelines are the ones with no clever state machine.