services/ssr.py walks JSON-LD and Next.js __NEXT_DATA__ for article-like
objects (headline, or title+url, or name with article signals), recovering
JS-rendered sites without a browser. prepare_headlines favors embedded JSON
over regex link scraping. Forbes is re-added as a live proof.
- Lower `SIMILARITY_THRESHOLD` to 0.68 and raise `MIN_SOURCES` to 3 for stricter story clustering.
- Expand `sources.txt` with new international and regional RSS feeds, including RT, The Jakarta Post, Philstar, and others.
- Add a new "Miscellaneous" section for diverse investigatory sources like The Intercept, OCCRP, and Bellingcat.
Add English-language international feeds: DW, Le Monde (English ed.),
France24, Euronews, Der Spiegel, El Pais (EN), The Japan Times, SCMP,
ABC News Australia, and CBC. Nikkei offers no clean public RSS, so Japan
Times + SCMP cover Japan/Asia instead.
Replace JS-heavy/paywalled HTML sources with verified RSS feeds (CNN,
Fox, NBC, NPR, CBS, NYT, Time, Axios). Drop Reuters, AP, Forbes, MSN and
US News, which offer no public/working RSS and are not reliably scrapable.
- services/feeds.py parses RSS 2.0/1.0 and Atom into (title, date) entries
- prepare_headlines auto-detects feeds by content and parses them directly
- strip feeds./rss./moxie. subdomains so feeds collapse to the outlet domain
- add 7 verified RSS feeds (BBC, Guardian, Al Jazeera, WaPo, The Hill, Vox, CNBC)
- get_sources skips blank lines and # comments
- broaden excluded-phrase list (skip links, share/follow, newsletter, utility)
Add resources/excluded_phrases.txt (skip links, privacy/consent, legal
boilerplate) and match it case- and punctuation-insensitively as a
contiguous run of words. Phrases live in a data file so they can be
extended without touching code.
News sites often reject minimal-bot requests; send a full browser header
set (User-Agent, Accept, Sec-Fetch-*, client hints). Accept-Encoding is
left to requests/urllib3 to avoid undecodable brotli responses.
Jared Bell