Skip to content

Tuning False Positives

No rule is perfect. Some may fire on patterns that are normal for your package set but would be suspicious elsewhere. This guide explains how to identify over-firing rules and what to do about them.

Step 1: Check fire rates

Start from the measured corpus fire rates in Fire Rates, then check how often the rule fires across your own reviewed packages:

trustsight review --verbose

For each rule, note the fire rate: the percentage of your reviewed packages where it triggered. Re-derive the corpus rates with scripts/rebaseline.py after changing rules or weights.

The >30% heuristic: A rule firing on more than 30% of your packages is not detecting anomalies; it is describing a property of your package set. That rule is now a census, not a signal.

Step 2: Understand why

Inspect a few packages where the rule fired:

trustsight inspect <package>

Look for patterns:

  • H004/R010/R011 (command-structure rules): H004 requires sudo at command position; R010 and R011 are restricted to function bodies. If they are still over-firing, the package may use unconventional but legitimate helper functions.
  • H001 (checksum removal): some AUR packages legitimately skip checksums for binary blobs.
  • H002 (checksum array emptied): inspect why a recipe changed sha256sums to an empty array.
  • H003 (insecure download protocol): inspect an added http:// source and whether the same change added or updated checksum backing.

Scope constraints already applied

R010 and R011 are scoped to function_body context to reduce false positives on top-level variable assignments and sourced library files. H004 is code-emitted and requires sudo at command position. If they still over-fire, your further options are:

  1. Demote the severity to INFO.
  2. Disable the rule entirely (not recommended; you lose signal).
  3. Add the false-positive pattern to a local allow-list.

Step 3: Fix : demote, disable, or constrain

Demote severity (preferred):

## config.toml
[rules.R010]
weight_override = 5  # was LOW/5; still fires, but you can raise or lower it

Disable the rule:

## config.toml
[rules.R010]
enabled = false

Only disable a TOML-defined, non-FATAL rule if you are certain the pattern it detects is never malicious in your context. Revisit this decision periodically; the threat landscape changes.

Constrain scope (where supported):

scope is a field of the rule definition in rules.toml, not a config.toml override. Edit it on the rule itself:

[[rules]]
id = "R010"
# ...existing fields...
scope = ["function_body"]  # already the default

Step 4: Re-baseline

After any change, re-run against your corpus:

trustsight review

Score changes: the demoted/disabled rule contributes less. Verify that the packages that were false positives now score where you expect them. See configuring rules and weights and the re-baselining guide.

Step 5: Validate with benchmarks

TrustSight's current test suite has a zero-rate of 70.3% (benign packages scoring 0). After tuning, re-run:

uv run --with pytest python -m pytest tests/

Ensure CRITICAL recall stays at 100%: every known malicious pattern must still fire. The corpus benchmarks in the explanation section define the expected p5/p95 separations:

Metric Value
CRITICAL recall 100%
CRITICAL p5 60
Benign p95 30

If demoting a rule drops CRITICAL recall below 100%, you have gone too far. Restore the rule and find another approach.

When not to tune

  • First-seen novelty scores (5-15) are not false positives. They are honest uncertainty that resolves as the database-wide maturity gate accumulates observations.
  • C-series rules (C001-C013) are structural invariants. They cannot be disabled through config. If they fire, they are detecting a real property of the PKGBUILD: investigate before suppressing.
  • INCONCLUSIVE verdicts from a cold database are not rule false positives. Let the maturity gate accumulate 50 observations before judging.

See also