Available now
Content Pipeline & News Aggregator Kit
Architecture for an RSS-based content aggregator with AI synthesis.
€39
Architecture for an RSS-based content aggregator with AI synthesis.
Every aggregator starts the same way. You wire a feed parser to a cron job, a template renderer to the parser output, and an email service to the renderer. Three days later you have a working RSS-to-newsletter tool. Three weeks later it is broken in six places — duplicates pile up, two feeds eat your rate limit, one bad article poisons your daily digest, and an accidental SSRF got your IP banned from a feed source.
The reason is not that the original code was wrong. The reason is that an aggregator at scale is not one problem; it is four problems. Naive implementations pretend it is one. This kit covers all four: fetch reliability (handling broken feeds, retries, timeouts, dedup), quality filtering (scoring heuristics to avoid junk input), synthesis (AI summarization with idempotency and schema validation), and SEO (avoiding thin-content penalties that make your work invisible to Google).
You get a 6-chapter book explaining the architecture, plus six TypeScript templates (normalized item schema, fetch worker, extractor adapter, rate limiter, enrichment pipeline, SEO config) designed to work together. Install once, customize for your stack, and ship with confidence that the patterns you are building on have failed and recovered in production.
Why most RSS-to-newsletter tools break at scale. The four architectural axes: fetch reliability, deduplication, quality, and synthesis. What this kit ships and what you bring.
Handling broken feeds, retries with exponential backoff, timeout policies, dedup via URL hash and content fingerprint, quality scoring heuristics, and the canonical schema for an item record. Free preview chapter.
Adapter pattern for HTML extractors with fallback chain (extractor A → B → raw HTML). Failure cases: login walls, cookie banners, client-side rendered content. When to skip vs. flag for review.
Politeness rules and robots.txt compliance. SSRF prevention via URL allowlist and IP-range checks. Token-bucket rate limiter per host with circuit breaker hooks and error budget.
Full URL → fetched content → AI synthesis flow. Async queue, retry semantics, prompt template structure, output validation against JSON Schema, and idempotency key per item to prevent duplicate enrichment.
Canonical strategy for syndicated content, indexation rules to avoid thin-content penalties, internal linking for crawlability, and sitemap structure. The chapter most aggregator builders learn the hard way.
feed-schema.json (normalized item), fetch-pipeline (retries, dedup, scoring), extractor-adapter (fallback chain), crawler-rate-limiter (token bucket), enrichment-pipeline (AI synthesis), seo-config.
Chapter 2 (Clean Fetch Pipeline) walks you through dedup layers, quality scoring, and retry logic. Read in 15 minutes; you will design every future fetcher differently.
Yes. Six TypeScript-shaped templates (one JSON Schema, one Markdown config) drop into any Node.js or Deno project. No vendor lock-in. Placeholders you fill in for your stack.
TypeScript. The async story in Node.js is closer to production aggregators (fan-out HTTP, rate-limited workers, queue-driven enrichment). Translating to Python is mechanical — read the book first, port as you implement.
No, intentionally. The book is generic patterns; the templates use placeholder values. You bring the feed list, the queue (BullMQ, SQS, custom), and the database. The kit is the architecture you wrap around them.
FreshRSS is an RSS reader — fetch and render for humans. This kit is for builders writing aggregators that *transform* feeds into a derived product (newsletter, AI digest, summarized stream). You can absolutely use FreshRSS upstream and build your transform layer here.
Yes. v1.x point releases (v1.1, v1.2) are free to v1.0 buyers — redownload from your purchase page. A hypothetical v2.0 major revision would be a separate SKU.
No. install.sh is fully offline. No network calls, no analytics, no license server. The only network activity is whatever your aggregator does at runtime — that's your code, not the kit.
30 days, no questions. If the pack or book does not land for you, email info@shippedstack.com and get a full refund. Single-seat license — please do not share the files.
Content Pipeline & News Aggregator Kit — €39 one-time, lifetime v1.x updates, 30-day refund.