Cluster before embed, accept soft duplicates

Constraint

Meridian's daily briefing had to stay readable. The scanner pulls the same story from three outlets with different titles. If every variant lands in Weaviate as a fresh chunk, the briefing reads like a press-release echo chamber. Deduping had to happen before the briefing agent wrote prose.

Options

1. Exact-match on normalized title/URL only (fast, misses rewrites).
2. Embed first, then drop near-neighbors above a cosine threshold before briefing.
3. Cluster on cheap lexical features first, embed one centroid per cluster, accept that some soft duplicates survive.

Chose

Option 3. The tradeoff: embedding every raw item was too slow and too expensive for the scan cadence I wanted. Lexical clustering is cheap and catches the obvious reprints. Soft duplicates with divergent wording still slip through. I accepted a noisier briefing over a slower, more expensive pipeline. That leak is still open.