Data storytelling · News analytics
Jakarta, 730 Hari
An AI-assisted data film about two years of Jakarta in the news.
- 690,828news articles analysed
- 102weeks, Nov 2024 to Oct 2026
- 7scenes in the film
- 8issues tracked
Overview
Why it exists, and what it does
The problem
News about a city arrives as thousands of separate stories. Almost no one can see what dominated, when attention rose and faded, or who was asked to answer for it.
The approach
A pipeline collects and cleans the news, removes duplicates, labels issues, tone and actors, and tests each claim statistically. A static film then tells the result in seven scenes, where every dot stands for about 1,200 articles.
Design principleCoverage is not reality.
More flood stories do not prove more floods, and a missing week is not a quiet week. The film measures what the media covered, shows gaps as gaps, and labels every finding with how confident it is.
Findings
Two years, six findings.
Each finding is labelled with how confident it is, based on sample size and a statistical test.
- High confidence21%
of all Jakarta news in one week of March 2025 was about floods
20× the normal share, and the highest peak of any issue in two years.
- High confidence1 wk
Media attention on floods halves within a week
Five of the six major floods follow this pattern.
- High confidence51%
January 2026 was the darkest month
The highest share of negative-toned articles in 24 months.
- High confidence+46%
Only public transport and crime grew convincingly
Share of coverage in year 2 versus year 1. Crime grew by 23%.
- Medium confidence4.3×
Traffic stories are born from events at Monas and MRT works
Traffic and public-space stories appear together 4.3× more often than expected.
- Low confidence4.0×
Foreign media write 4× more about pollution
The foreign sample is small, so the direction is clear but the size is not.
What this does not claim
- The film measures news coverage, not the city's actual condition.
- Missing is not zero: two weeks in June 2025 are absent from the snapshot and shown as gaps.
- “730 days” is the film's name for about two years; the data itself spans 102 weeks, from November 2024 to October 2026.
- Topic, tone and entity labels are model output until checked against human-labelled examples.
- International outlets are a small share of the sample.
Walkthrough
Don't just read about it. See each screen.
5 real screens from the film. Pick a tab to see what each one shows.
Jakarta as a map of dots, one per roughly 1,200 articles, filling in week by week. Orange weeks lean negative and blue weeks lean positive, with a live ticker of incoming headlines.
Screenshots are from the built-in demo mode, which uses a fictional project.How it works
From raw news to a tested story
- 1
Collect and clean
RSS feeds, sitemaps and responsible crawling, with GDELT as a supplement. Duplicates are removed on three levels, from identical URLs to the same event told by different outlets.
- 2
Label and measure
Every article gets issues, tone and the actors involved. Ambiguous cases go to a language model for review, then statistics test which patterns are real.
- 3
Publish a snapshot
Checked results become a versioned snapshot. The site only reads that file, so it never crawls or calls a model while you watch.
How the labels are made
Three levels of de-duplication
- Exact: SHA-256 hashes and canonical URLs.
- Near-duplicate: MinHash with LSH over five-word shingles and estimated Jaccard similarity.
- Shared event: multilingual-e5-base embeddings plus time, place and source context, to separate copies from independent coverage of one event.
Two-stage labelling
- Stage 1, local: GLiNER2 assigns issues, area relevance, sentiment and entities, with a confidence score and model version stored for every label.
- Stage 2, review: ambiguous or high-impact cases escalate to a language model that answers in a validated, logged format.
- Sentiment is scored per actor, so a resident's complaint and an official's reply in one story can differ.
Escalation triggers: a top issue score below 0.60, a gap under 0.15 between the top two, uncertain location, or a major story covered by 10+ outlets.
Architecture
Seven roles, one pipeline
Only some roles use a language model. The rest are deterministic or classic NLP, which keeps model calls and cost low.
- Scout
- Crawler
- Dedup
- Classifier LLM review for unclear cases
- Analyst
- Verifier
- Publisher
- Discovery
Scout
Finds articles through RSS feeds, sitemaps and GDELT, strips tracking parameters and checks the queue before any fetch.
- Rules + extraction
Crawler
Fetches within robots.txt, per-domain rate limits and paywall rules, then removes menus, ads and recommendations.
- Hash + embeddings
Dedup
Separates duplicate copies from independent coverage of the same event.
- NLP + LLM
Classifier
Runs GLiNER2 locally and escalates only the unclear or high-impact cases to a language model.
- Statistics
Analyst
Issue share, surge detection, outlets per event, media half-life, co-occurrence and bootstrap intervals.
- Quality gates
Verifier
Eight gates from source health to publishing. Failed extractions are flagged, never counted as zero.
- Schema + review
Publisher
Validates the snapshot, opens a pull request for review, then ships it to the site.
Stack
- Discovery
- RSS · sitemaps · GDELT · httpx · Crawl4AI · Trafilatura
- Orchestration
- LangGraph · Prefect · Langfuse
- Storage
- PostgreSQL 16 with pgvector · Elasticsearch 8 · MinIO/S3
- Models
- GLiNER2 · multilingual-e5 embeddings · a structured LLM for escalations
- Delivery
- Pydantic validation · GitHub pull-request review · Next.js on Vercel
Quality & privacy
Built to be audited, and honest about its limits
Quality gates
- Eight quality gates, from source health to publishing.
- Every insight links back to article IDs; runs record stage counts, cost and model versions.
- Missing weeks stay visible as gaps, and unsupported claims are not exported.
- If the pipeline fails, the last published snapshot stays live.
- Evaluation criteria were set up front: a 1,000-article human-labelled reference set. These are targets, not reported results.
Responsible data
- Crawlers follow robots.txt and keep to one request every two seconds per domain.
- Paywalls and access rules are respected, and public excerpts are kept short.
- Raw HTML is kept for 180 days, and headlines stay credited to their publishers.
- An independent personal project, not an official government service.
Want to watch it?
Press play, or step through the seven scenes yourself. Switch to EN for English.
