AutOps is a workflow-based system rather than a standalone application. Its logic lives in n8n workflows kept under version control, which means the whole system is auditable, forkable and modifiable without recompiling anything. This page describes how the pieces fit together and why they are arranged this way.
AutOps is designed to be dropped into an existing server environment, not to replace it. It interacts with six things: the monitored servers themselves, whatever monitoring tools are already collecting metrics, the n8n workflow engine at the centre, the AI analysis module, the messaging platform used for approvals, and the administrators who make the decisions.
Four constraints shaped every decision that follows. The system must express its logic as n8n workflows, which bounds what integrations are possible. It depends on external monitoring tools, AI services and messaging platforms, each of which can impose API limits or go down. It must remain portable and well-documented, because it is open source. And every automated action must pass through a human, which rules out the simpler fully autonomous designs.
Before any of the internals matter, it is worth being precise about what AutOps is and is not responsible for. Five components sit inside the system boundary. Everything else — the servers being watched, the messaging platform, the inference API and the vector store — is a dependency the project consumes but does not own, which is exactly where its failure modes and its API limits come from.
Responsibilities are separated so that no single module can both decide and act. Detection cannot execute. Reasoning cannot execute. Only the execution module touches the server, and only when the decision module has recorded an approval.
Initiates the workflow, either from a scheduled cron node or a manual invocation. It opens a
secure SSH session, runs docker ps -a, and reconstructs the complete state of
every container including ones that have crashed or exited.
Because Docker's output uses variable-width whitespace, a JavaScript node tokenises it with
a /\s{2,}/ split to extract container names and states reliably. State-aware
filtering then whitelists known-ephemeral containers — the certbot renewal container,
for example, whose exited state is normal — so they never raise false positives. If
every remaining container is Up, the workflow terminates silently.
A LangChain agent node gives the model genuine tool access rather than static chat behaviour. On receiving a trigger, a data-ingestion component gathers relevant data from the state store, the monitoring layer and the host; a context builder assembles it into a situational snapshot; and the reasoning engine determines what happened.
The agent's toolset lets it:
It cannot run anything that changes state. That restriction is enforced through a structured system prompt and reinforced by the module boundary — the analysis module has no path to the execution node.
Receives the proposal and routes it through policy evaluation. A policy manager decides whether an action may proceed automatically or requires human intervention; for state-changing commands the answer is always the latter. The proposal is persisted and the workflow halts pending an administrator's explicit decision.
Formats the incident, the diagnosis and the exact proposed command into a rich report and
delivers it to the configured administrative channel. The agent is programmed to emit a
strict template, placing the remediation command under a
Proposed Action Required heading followed by the explicit
!approve / !deny prompt, so the message is machine-parseable as
well as human-readable.
Receives the approved action, validates its safety, and runs it. The command is loaded into an SSH execution node that authenticates to the host with cryptographic keys.
The five modules fall into three privilege zones, and the boundaries between them are the whole safety argument. A module in the observe zone has no state-changing tool available to it. A module in the decide zone can write a record but has no connector to the execution node. Only the act zone can reach the host, and it will only run a record that already carries an approval. An attacker who fully controls the model’s output still lands in zone two.
Execution is not fire-and-forget. The module captures the complete output of the command
— both stdout and stderr — and evaluates it to confirm
the command ran without syntax errors or runtime failures before reporting back. What you
receive is a verified result, not an assumption that the command probably worked.
Docker routes containerised application logs through stderr, not
stdout. An early version of the !logs command captured only
stdout and delivered empty messages while appearing to succeed. Reading both
streams is what makes log diagnostics work at all — and it materially improved the
AI's accuracy, because the model was finally seeing the actual error text.
The module view says who does what; it does not show the one property that shapes the rest of the design. Detection is continuous and machine-paced, while authorisation is human-paced and arrives whenever somebody reads their phone. The sequence below makes that gap explicit.
pending_actions exists at all.Everything awkward about the implementation follows from the band in the middle of that diagram. Because the workflow cannot stay resident while it waits, the incident has to be written down somewhere durable, the approval has to be matched back to it later, and the match has to be revalidated at the moment of execution rather than trusted from when the alert was sent.
Continuous background telemetry and asynchronous human authorisation are decoupled, so the system needs durable state to connect a detection to the approval that arrives minutes or hours later. This is held in n8n's native data tables rather than an external database, avoiding an extra dependency.
A pending_actions table tracks the lifecycle of each detected anomaly with
explicit fields: the target container's identifier, the AI-formulated remediation command,
the current alert state, and a detection timestamp. When an approval arrives over Discord,
the execution module queries this table to retrieve the exact validated parameters.
A five-minute polling cycle against a container that stays down for two hours would naively produce twenty-four duplicate incident records. Ingestion therefore uses an upsert rather than an insert: if an unresolved entry already exists for that container, only the detection timestamp is refreshed; a new row is appended only when no prior record exists. This keeps the table clean, keeps queries fast, and prevents duplicated alerts at the source.
pending_actions lifecycle. The loop at the top is the idempotent upsert: a container down for two hours produces one row, not twenty-four.
A Command Router node classifies inbound messages before any inference happens,
which is both a cost and a latency optimisation.
!approve,
!deny or !logs skips the inference engine entirely. Direct
administrative commands do not need an LLM, and routing them around it cuts both API cost
and response latency.
The first prototype used a community-built n8n Discord node. It failed for an architectural reason rather than a bug: the node maintains a stateful, long-lived WebSocket connection to Discord's gateway, while n8n executes on a stateless, trigger-based model. The connections experienced silent timeouts and dropped payloads without raising errors — the worst possible failure mode for a monitoring system, because it fails quietly.
The fix was a middleware daemon. A containerised Python service built on
discord.py runs continuously alongside the workflow engine. It natively holds
the stateful gateway connection Discord expects, subscribes to the administrative channel,
and converts inbound chat events into structured JSON delivered to n8n as ordinary HTTP
webhooks. Outbound replies go the other way, through an HTTP request node against Discord's
REST API.
The result is a protocol translation layer: stateful where Discord requires it, stateless where n8n requires it, with neither side compromised.
The entire ecosystem is described by a single docker-compose.yml acting as a
declarative blueprint — infrastructure as code rather than a sequence of manual
installation steps. It defines the network topology, persistent volumes, environment
variables and inter-service dependencies needed to instantiate the system.
Bringing the stack up provisions, in the correct order:
Every service is configured with restart: always. After a host failure, a
service crash or a scheduled VPS reboot, the Docker daemon restores the whole stack without
manual intervention — which matters more for a monitoring tool than for most software,
since a monitor that dies silently is worse than no monitor at all.
| Resource | Reference prototype | Recommended |
|---|---|---|
| CPU | 1 vCPU | 4 cores |
| Memory | 2 GB RAM | 8 GB RAM |
| Storage | VPS default | 50 GB |
| Network | Public IPv4 | 100 Mbps minimum, 1 Gbps preferred |
| Operating system | Ubuntu 24.04 LTS | Ubuntu Server LTS |
| GPU | None — inference is hosted | Optional, only for self-hosted heavy models |
The prototype's 1 vCPU / 2 GB allocation was determined empirically to be sufficient for the orchestration engine, the messaging bridge and the diagnostic test applications without hitting bottlenecks. Inference runs on a hosted API, which is why the compute footprint stays this small.
Every choice below had a cheaper or simpler alternative. They are recorded with what was given up, because a decision with no cost attached to it usually means the alternative was never seriously examined.
| Decision | Alternative considered | Why, and what it costs |
|---|---|---|
| Logic lives in n8n workflows | A standalone application in Python or Go | Auditable, forkable and modifiable without recompiling anything. The cost is that the set of possible integrations is bounded by what n8n nodes exist. |
| A human approves every state change | Fully autonomous remediation | A wrong diagnosis can never become a wrong action. The cost is real latency and the requirement that somebody is reachable. |
| A custom Python Discord daemon | The community n8n Discord node | The node held a stateful WebSocket inside a stateless engine and timed out silently. The cost is one more container to run and maintain. |
| n8n native data tables for state | An external Postgres or Redis instance | Avoids a dependency and a second backup story for what is one small table. It would not hold up under high write volume. |
| Hosted inference | A self-hosted open-weight model | Keeps the footprint at 1 vCPU and 2 GB with no GPU. The cost is an external API dependency, a per-token bill, and sending log text off the host. |
| Upsert on ingestion | An insert for every detection | One row per unresolved incident instead of one per poll. It requires a well-defined uniqueness rule per target to be correct. |
| A whitelist in the execution module | Trusting the model’s proposed command | Bounds the blast radius even if the model is fully compromised. Every legitimate new command needs an explicit addition. |
| Prefix bypass for direct commands | Routing every message through the LLM | Removes both cost and latency from the most frequent operations. It adds a second code path that has to stay in step with the first. |
Three issues consumed disproportionate effort and are recorded here because each has a non-obvious cause.
Community Discord nodes held WebSocket connections that silently timed out under n8n's stateless execution model. Resolved by introducing the persistent Python daemon described above, which converts gateway events into webhooks.
Approvals stopped matching pending records. The cause was a parsing quirk in the data
table module, which prepended = to dynamic string variables — storing
=website instead of website and making every strict-equality
lookup return null. A sanitisation node now strips leading equals signs and whitespace
before any query.
The !logs command reported success and delivered nothing, because the
execution node captured only stdout while Docker routes application logs
through stderr. Rewritten to capture, concatenate and return both streams.
Long diagnostic payloads exceeded Discord's message limit and returned HTTP 400. Outbound messages are now intercepted, truncated to roughly 1,900 characters and marked with an ellipsis so delivery always succeeds.
Testing followed a bottom-up strategy: verify each data source before verifying the consumers that depend on it. All of it ran on production-grade VPS hardware, with a dedicated Docker Compose test profile provisioning isolated ephemeral containers that shadow live service configurations without port or network conflicts.
Each requirement maps to the component that realises it and to the test that demonstrates it. Where a test found a defect, the defect is named rather than smoothed over.
| Requirement | Realised by | Evidence |
|---|---|---|
| Detect a container failure in under a minute | Trigger module — cron plus docker ps -a over SSH |
Failure-resilience runs against isolated ephemeral containers; detection logic completes in seconds, with the polling interval setting the ceiling |
| Do not alert on a normal exit | State-aware filtering with an ephemeral-container whitelist | The certbot renewal container, whose exited state is expected, raises no incident |
| Ground the diagnosis in this deployment, not a generic one | Pinecone retrieval inside the AI agent’s toolset | Retrieval-accuracy test: embeddings generated from the inbound query and matching context returned before any diagnosis |
| Diagnose from the actual error text | Execution and log capture across both output streams | !logs regression: the empty-message defect closed once stderr was read, and accuracy improved measurably |
| Never change state without a human | Decision module, approval gate, and the three privilege zones in Figure 3 | Negative security testing with malicious payloads; the whitelist blocks and logs every one |
| Match an approval to the right open incident | pending_actions plus the validation gateway in Figure 6 |
Concurrency test: ten simultaneous executions commit without lock errors, corruption or duplicates |
| Keep discussing the same incident across messages | Session-bound conversational memory | Session-continuity test: memory binds to the correct session identifier |
| Survive a reboot or a crash unattended | restart: always on every Compose service |
Host reboot and service-crash scenarios; the stack returns without manual intervention |
| Always reach the operator, even on a bad payload | Outbound truncation to roughly 1,900 characters, plus a fallback notification | Long-payload test: the HTTP 400 from Discord’s 2,000-character ceiling no longer occurs; SSH timeouts and refusals still deliver an alert |
| Run on modest hardware | Hosted inference, so no GPU and no local model weights | Reference deployment on 1 vCPU and 2 GB serves the engine, the bridge and the test applications without bottlenecking |
The workflows, system prompts and Compose configuration are all in the public repository. If you want to read the actual implementation rather than a description of it, start there.