Watching the AUR Continuously¶
trustsight full-aur --watch turns the tool into a scanner: every cycle
refreshes the AUR metadata snapshot, diffs what moved, analyses the changed
packages, and reports new alert clusters. A cluster is announced the first
time it is seen and then counted, not re-announced, so a quiet night prints
nothing.
What a cycle costs¶
The AUR holds about 120,000 packages. The first run is a bootstrap: two
requests per package (PKGBUILD and .SRCINFO), throttled to 5 requests per
second with exponential backoff, so a from-scratch pass takes most of a day
and must be asked for with --bootstrap. Every cycle is capped
(limits.corpus_max_per_cycle) and resumes automatically, so the bootstrap
advances in gentle chunks across as many cycles as it needs.
After that, a cycle is nearly free: one conditional GET for the metadata (zero bytes when the snapshot has not moved) plus two requests per changed package. That is the whole load on the AUR, at any watch interval.
Choosing an interval¶
trustsight full-aur --watch # one cycle per hour (default)
trustsight full-aur --watch --interval 1800
The interval sets alert latency, not politeness: the load is the same
metadata GET plus actual churn either way. Below about 15 minutes you are
mostly re-downloading a snapshot the AUR has not regenerated; the floor is 60
seconds (limits.watch_min_interval), because a shorter interval only spins.
At startup the watch syncs rules.toml to the shipped ruleset the safe way
(adds missing rules, repairs a match_target/severity/category/scope
that moved, adopts a superseded shipped pattern, never touches a pattern you
edited) and logs what it changed, so an unattended watcher never analyses with
a stale ruleset. Set TRUSTSIGHT_NO_SYNC_RULES to skip it.
Replaying the past with --since¶
A fresh watcher only sees what changes from now on. To scan for a campaign that already happened, replay history instead:
Each cycle analyses the packages whose AUR LastModified falls in one day
of AUR time, empty days are skipped inside the cycle, and the replay joins
the live delta stream when it catches up. The clock is the AUR's own
timestamps, not wall time, so a replay of June 2026 walks June 2026 no
matter when you start it. The cursor persists in the database: interrupt
it, restart the container, and the replay continues from the day it
reached.
Two things to expect on a replay. It alerts through the same
over_threshold bar as the live stream; and the sweep and the adoption
feed stay out of replay cycles, because they model the live stream and two
years of history would only distort their baselines. A package that was
cleaned up after the campaign scores by its current recipe: what
persists is what gets flagged.
Alerting over a webhook¶
A watcher nobody reads is a log file. Give it a push channel and new alert clusters arrive as one HTTPS POST of a JSON document per cycle:
or permanently in config.toml:
A second, independent sink posts the same document to the TrustSight feed, which ingests it directly instead of polling ntfy:
Every feed POST is signed with an Ed25519 key
(TRUSTSIGHT_FEED_KEY, base64 PKCS#8; TRUSTSIGHT_FEED_KEY_ID names it), and
the feed holds only the public half, so a leaked feed secret cannot forge a
document. ntfy and the feed are independent: one failing never stops the
other, and with no key the feed sink is skipped.
The document carries the cycle counts and the alerts. Every
over_threshold entry says what happened, not only that it did: the
version transition, the AUR change date, the current and previous
maintainer and the rules that fired. Each alerts entry names the
package and rule, and an ownership rule (H044/H026) also carries old
and new; a campaign_cluster carries its stable campaign_id and
members_detail with the same ownership and version context for each member,
so a reader can see which maintainer adopted which package and follow the
campaign across cycles. An alert may also carry pkgbuild_sha256,
previous_pkgbuild_sha256 and a bounded diff of the exact recipe change;
when the diff exceeds the cap, diff_truncated and diff_sha256 remain. While a --since replay runs, day
names the AUR day being analysed. The alerting bar is --over-threshold
N (the over_threshold argument on refresh_corpus/watch); the
default is 30, the benign corpus's measured 95th percentile
(calibration.benign_p95), and a package alerts when its score is
strictly above it.
{
"event": "trustsight.alerts",
"tool": "trustsight",
"title": "TrustSight: hot-pkg scored 45 (H001) (2026-05-01)",
"priority": "default",
"day": "2026-05-01",
"cycle": {"added": 3, "changed": 41, "removed": 2, "processed": 44},
"alerts": [
{"package": "adopted-pkg", "rule_id": "H044",
"maintainer": "newm", "old": "oldm", "new": "newm"}
],
"over_threshold": [
{
"package": "hot-pkg",
"score": 45,
"aur": "https://aur.archlinux.org/packages/hot-pkg",
"version": "1.0 -> 1.1",
"last_modified": "2026-05-01 12:40 UTC",
"maintainer": "hotm",
"previous_maintainer": "oldhotm",
"rules": ["H001", "R001"]
}
],
"campaign_cluster": [
{
"rule_id": "G003",
"members": ["adopted-pkg", "other-pkg"],
"count": 2,
"members_detail": [
{"package": "adopted-pkg", "maintainer": "newm",
"previous_maintainer": "oldm", "version": "1.0 -> 2.0"}
]
}
],
"ioc_matches": []
}
The corpus path downloads each recipe's snapshot tarball (the GitHub AUR
mirror first, the AUR's cgit second) rather than cloning, so there is no
commit id to report; the AUR LastModified and the version transition are
the anchors that carry the same information. title lands in the notification header on ntfy, with a tag for the
level. Alerts never carry a priority: an over-threshold package is worth a
quiet read in the morning, not a page at night; only the daily heartbeat
sets one (Priority 2, low).
Once a day the watcher also sends a low-priority heartbeat
(trustsight.heartbeat, ntfy Priority: 2) with the cycle counts and,
while replaying, the day it has reached. Silence then has only one
meaning: nothing found, not "the watcher died".
Any receiver that accepts a JSON POST works. ntfy accepts the same POST and
shows the document as the message text, so --notify https://ntfy.sh/mytopic
is a working phone notification with no further setup. A dead receiver is
logged and swallowed: a notification must never kill the watch loop. Cycles
with no new alerts send nothing. A package matching an IOC baseline entry
is listed in the document's ioc_matches whatever it scored: the IOC tier
reports outside the heuristic score, so it would otherwise never cross the
bar, and the match is a fact to read rather than a higher priority.
Single-shot cycles (a bootstrap chunk, a plain full-aur run) notify like
watch cycles.
Health and delivery¶
A failed cycle is not a dead watcher, and a quiet channel is not proof of
life. trustsight full-aur health reads the recorded last start, last success,
consecutive failures, outbox and database size; --json is the machine form.
The container healthcheck uses full-aur health --optional, so a database with
no watcher state is healthy and a stale or repeatedly failing watcher is not.
Alerts are written to a durable outbox before any POST. If ntfy or the feed is
down, the document stays queued and is retried with capped exponential backoff
on later cycles; an alert that cannot be delivered within 24 hours is parked
and health reports it. This is at-least-once delivery: ntfy may show a
duplicate after a crash between the POST and the delivery mark, while the feed
dedupes on the persisted webhook-id.
SIGTERM stops the loop between packages, saves the resume state and
checkpoints the WAL. Only one watcher may own a data directory; a second
start fails with the lock path instead of interleaving cycles.
For a container, mount the feed key as a secret and point
TRUSTSIGHT_FEED_KEY_FILE at it rather than putting the private key in the
environment. The shipped compose file runs read-only with a tmpfs for /tmp
and the cache, drops all capabilities, sets no-new-privileges, and permits
the parser workers' unshare and mount operations with seccomp=unconfined
and apparmor=unconfined (the workers still apply their own seccomp
allowlist and namespace/mount floor).
As a systemd service¶
# ~/.config/systemd/user/trustsight-watch.service
[Unit]
Description=TrustSight AUR watcher
After=network-online.target
[Service]
ExecStart=/usr/bin/trustsight full-aur --watch --interval 1800
Restart=on-failure
RestartSec=300
KillSignal=SIGTERM
TimeoutStopSec=120
[Install]
WantedBy=default.target
State is durable at every cycle boundary (the snapshot and the resume file
are saved before a cycle returns), so Restart=on-failure loses at most the
cycle in flight. Enable with:
systemctl --user enable --now trustsight-watch.service
loginctl enable-linger "$USER" # keep it running without a login session
As a Docker container¶
The image is published to the GitHub Container Registry on every release:
docker run -d --name trustsight-watch \
--user 1000:1000 \
--security-opt seccomp=unconfined \
--security-opt apparmor=unconfined \
-v trustsight-data:/home/ts/.local/share/trustsight \
-v $PWD/trustsight-config:/home/ts/.config/trustsight \
--restart unless-stopped \
ghcr.io/emiliano-go/trustsight:latest \
full-aur --watch --interval 1800
The volume holds the database, the corpus state and the resume file, so a
container restart (or a new image) continues where the last cycle ended.
Mount a config directory carrying your config.toml (and [notify] webhook)
to keep settings out of the image; it is mounted writable because the startup
ruleset sync writes the synced rules.toml (and any missing default file)
back into it (the directory must be writable by the container's ts user,
uid 1000). A read-only or unwritable mount logs ruleset sync skipped:
Permission denied and keeps the rules the volume was created with, unless you
set TRUSTSIGHT_NO_SYNC_RULES=1 to skip the sync deliberately. The
container runtime must allow unprivileged user namespaces: if it does not, the
parser workers refuse and the package is reported as not vetted rather than
analysed with weaker confinement. With Docker, also pass --security-opt
seccomp=unconfined --security-opt apparmor=unconfined (or custom profiles
that permit unshare and mount), because the built-in profiles deny the
namespace and mount syscalls. The process needs no
privileges. See
Reducing Blast Radius for the boundary around the
auditor and what it does not cover.
A compose file with the same settings - a fixed container name, log
rotation, the timestamped log format, a read-only root filesystem, dropped
capabilities, resource limits and the watcher healthcheck - ships as
docker-compose.watcher.yml:
mkdir -p trustsight-config
docker compose -f docker-compose.watcher.yml up -d
docker compose -f docker-compose.watcher.yml logs -f trustsight-watch
Reading the logs¶
The watcher logs every cycle to stderr, so docker logs -f trustsight-watch
(or the compose command above) is the watcher's whole history. A watch opens
with a banner naming the build, the database path, the interval and whether an
alert webhook is configured; each cycle then reports what moved, the packages
over the alerting bar, and any new cluster alert. The output is unbuffered
(PYTHONUNBUFFERED=1 in the image), so lines appear as they are written.
TRUSTSIGHT_LOG_TIMESTAMPS=1 prefixes each line with a timestamp and level -
useful when the container is the only record. TRUSTSIGHT_LOG_LEVEL raises or
lowers verbosity (DEBUG, INFO, WARNING, ERROR). Log records are rendered
inert like every other terminal surface: a maintainer name, URL, path or
exception message cannot inject control sequences or forge an extra log line.
The image runs rootless and ships no pacman, which makes it the cold-environment reference: every measurement inside it is the reproducible one.
Feeding the watcher the baselines¶
The watcher is sharpest with the signed channel data in place:
trustsight seed fetch # known URLs, maintainers, dependency names
trustsight ioc update # federated known-bad indicators
Both verify against the pinned distribution key. Neither refreshes itself: re-run both on your own schedule (a cron entry next to the service is fine; they are cheap conditional fetches).
Reading what it found¶
New alerts also land in the log and in --json output (one JSON object per
cycle). For the history of what moved, trustsight corpus pivot and
trustsight history <package> read the same database the watcher writes.