Glean
A local-LLM OSINT pipeline that turns scattered recon into one prioritised, provenance-tracked brief.
Open-source OSINT automation is strong at collection and weak at judgment. Tools like SpiderFoot, Amass, and theHarvester can gather thousands of reconnaissance findings, but they arrive as a flat pile: no reliable prioritisation, no provenance surfaced to the analyst, and no synthesis into something a human can act on. A single scan of one low-value domain can return around 960 findings whose headline correlations are a handful of low-priority notes about unrelated shared-hosting neighbours. Collection succeeds; judgment does not.
Glean closes that gap. It runs a curated set of maintained FOSS tools, normalises their output into a single provenance-tracked entity model, and correlates entities deterministically in code rather than in the model. A small, locally-run LLM through Ollama then produces a one-page prioritised brief a human can action in under two minutes. The tool is the deliverable, but the research question sits underneath it: can small local models faithfully prioritise and preserve provenance when synthesising structured, multi-tool intelligence?
That question is largely unmeasured. Existing security-LLM faithfulness work, from CTIBench and RAGIntel to general frameworks like RAGAS and FaithBench, evaluates text-to-text tasks over prose. Glean's input is not prose but normalised entity records, and its output is not an answer but a ranked judgment with provenance. Measuring faithfulness, prioritisation quality, and provenance retention over that structured artifact is the gap the project is built to close.
Version one is deliberately narrow: one entity domain (infrastructure and domain reconnaissance), four to six curated tools, deterministic correlation, and an evaluation harness that reports three numbers against a hand-built ground-truth set. It is CLI-first and for authorised research only, with passive and active reconnaissance clearly separated and every claim traceable to its source tool. The package name is reserved on PyPI as glean-osint, and the project is in early development.
What it involved
- Unified entity model: findings from multiple FOSS recon tools normalised into one schema, with the source tool preserved as a first-class provenance field
- Deterministic correlation: dedup and entity-linking handled in code, never delegated to the LLM
- Local-LLM synthesis: a small model through Ollama produces a prioritised, one-page brief a human can action in under two minutes
- Evaluation as the research: a harness that measures faithfulness, prioritisation quality, and provenance retention against a hand-built ground-truth set
- Scoped and ethical: infrastructure OSINT first, authorised targets only, with passive and active reconnaissance clearly separated