Self-hosting a Linux server can involve many repetitive and time-consuming tasks. When I first set up my Ubuntu server, I would spend much of the day searching Stack Overflow for various bash commands. Then LLMs came onto the scene. They made this task much more ergonomic. Still, there was friction. Part of this was due to early LLMs being less reliable, but most of the friction came from the LLM not having the context it needed. So each chat would start with a litany of system info, apps that were installed, troubleshooting steps that had already been tried, etc. My approach has now evolved to using a structured folder documentation repo. This started by coming up with a schema that basically became something like:

HomeServer/
├── README.md                 # hostname, IP, ports, service index
├── server/
│   └── README.md             # host: hardware, network, firewall, cron
├── services/
│   ├── apache/
│   ├── ssh/
│   ├── mattermost/
│   └── …                     # one folder + README per service + optional service-related scripts
├── tools/
│   ├── ssh/                  # PowerShell helpers for remote access
│   └── …                     # scripts / small utilities
└── docs/
    └── ai-agent-runbook.md   # how agents should work against the box

This confers multiple advantages that aren’t immediately intuitive. My first instinct was to create a simplistic file system mirror; this way agents would be able to instantly see “what’s there.” I decided that this is the wrong level of abstraction for several reasons. Telling an agent what is there is often not as important as telling an agent why something is there. Grouping documents and scripts by services is much more natural.

Sessions often revolve around a single service at a time, so the goal is then to communicate context as efficiently as possible without information loss. Documenting a service at a time allows you to succinctly communicate why you are using the service and how it interacts with other services in your particular scenario, which is information completely unique to your use case. This primes the agent’s world model to be aligned with the eccentricities of your server, as opposed to the general distribution of home servers it has seen in training.

It also provides the perfect place to record every time something went wrong — whether it’s unexpected agent behavior or syntax that agents often get wrong. Tracking problems saves time and creates more reliable agent behavior. This naturally evolves into the tools folder, which provides a collection of safe and proven scripts agents can use to repeatedly accomplish tasks with less hand-holding. Combining all of these techniques, we can then write detailed runbooks that incorporate all this knowledge to reliably execute multi-step actions.

Upgrading Mattermost

Mattermost is a Slack alternative and upgrading is a relatively straightforward process, but it would often take several hours running backups, reading release docs, checking for breaking changes, etc. To address this, I created a detailed runbook for an agent with links to documentation, explicit commands to run on the server, and expectations for what those commands would return. Crucially, the runbook also encodes explicit checkpoints where the agent should pause, present a summary of their current status and findings, and ask the user whether they should continue or abort. This has turned a multiple-hour process into a pain-free, simple command with a few confirmation steps built in.

Limitations

It should be said that this approach is not for everyone. The largest issue is that the guardrails and conventions are not binding. The agent can ignore them. Without responsible supervision, it’s still possible for the agent to wreak havoc. The obvious next step here would be to introduce a harness to validate commands agents attempt to run on the server. Another problem is that there is no current mechanism for keeping the service documentation in sync with what is actually on the server, so it is possible for documentation to drift. While this is somewhat handled by having agents update documentation as they go, it is a good opportunity to define another runbook that can crawl the server and cross-check our documentation to make sure they are in sync.