The Leak Check Only Grepped Half the Package

The short version

We had built the leak check for a paid distribution package out of a fixed grep word list, and it had a hole no amount of vocabulary could close: the paths the check grepped were only part of the package.

• The installer script shipped to buyers with an internal note left in it
• The file bundled as the terms of use was a draft, from before legal review
• The grep check passed every time. Both files were outside the directory being inspected

Adding words would not prevent a recurrence, so we revisited the inspected paths themselves.

What it looks like

We run a build script distributing an internal skill collection as a paid package. It gathers artefacts into an output directory with rsync, greps for internal-specific terms (company names, personal names, fragments of local paths) to check none remain, and then zips. The check was green every time; as far as CI went, all you saw was “no internal-specific text”.

Then a buyer’s review pointed out two things.

• The bundled installer script had shipped with an internal comment to the effect that “this repository is the internal source of the port, not something a purchaser is expected to use day to day”
• The file bundled as the “terms of use” was a draft, from before internal legal review

Both exist as text and both read sensibly. The problem is that grep had never inspected these files at all.

Why

Revisiting the check command, grep’s target path was not the whole output directory but only the subdirectory holding the skills.

The installer script and the “terms of use” file, meanwhile, were copied outside that subdirectory — directly into the output root. They were never in grep’s target paths to begin with.

However many internal terms you pile into the word list, a file outside the paths grep reads has nothing to match against. This finding did also surface genuine gaps in the vocabulary (terms indicating a pre-legal-review stage, and a name referring to another internal project, were unregistered), but fixing only that would have left the same hole. As long as the inspected paths cover half the package, no enrichment of the vocabulary ever reaches the files outside them.

Fixing it

We made it two-stage.

1. Widen grep’s target paths to the whole package, root included.

# before: inspected only the skills subdirectory
grep -rInE "$WORDS" "$OUT/skills/"

# after: covers the whole package, including the installer and docs copied to the root
grep -rInE "$WORDS" "$OUT" --include="*.md" --include="*.sh"

2. Add a sanitising step before grep.

grep is a mechanism for finding only the words on the list, so it is powerless against a new leak pattern not on it. Instead of chasing individual words forever, we put a step first that mechanically handles the categories that must always be dropped at distribution time. Provenance comments (statements of which repository something came from) and internal-only config values are rewritten with sed just before distribution.

for f in "$OUT/install.sh" "$OUT/uninstall.sh"; do
  sed -i '' '/<pattern naming the internal repository>/d' "$f"
  sed -i '' 's/<internal-only config value>/<default for distribution>/' "$f"
done
bash -n "$OUT/install.sh" && bash -n "$OUT/uninstall.sh" \
  || { echo "sanitising broke the script"; exit 1; }

sed rewriting can break a script’s syntax, so a bash -n syntax check runs immediately after sanitising. The order is now drop it mechanically in sanitising, and let grep inspect the whole thing as the last line of defence.

Making it less likely to recur

A denylist check readily produces a sense of safety with every word added — “that one is covered now”. But the reality here was that the paths, not the vocabulary, were covering half. A check returning green every time is not proof that what it inspects is right.

The order to review should be reversed. Doubt what the check command actually reads first, confirm it covers the whole package, and only then work on enriching the vocabulary. In a case like this one, where the output directory’s structure changes partway (more files copied to the root), revisiting the inspection scope falls behind updating the vocabulary easily.

The same shape of “the assumed scope of a check and its real scope diverge” appears in A staging Default in env Deleted Production Images too. In both, the check is running and what it looks at is narrower than anyone thought.