upstream monitoring system

The Upstream Changelog Nobody Read

Real upstream changes that broke real pipelines. Every entry here cost someone a broken pipeline.

Browse incidents
Scroll
Incident log

Changes that broke production

Each entry is a real upstream change that went unnoticed. Names anonymized. Damage preserved.

INC-001
Change: Column renamed in payments API
Detected after 11 days
payments_api/v2/schema: - amount: DECIMAL(10,2) + total_amount: DECIMAL(10,2) currency: STRING created_at: TIMESTAMP
Buried in CHANGELOG.md, line 847
Casualties Revenue dashboard, 3 dbt models, 1 on-call weekend destroyed
// the column was renamed in a "minor" release
INC-002
Change: Column type changed from INT to STRING
Detected after 3 weeks
upstream.users: - user_id: INT64 + user_id: STRING email: STRING created_at: TIMESTAMP
Buried in Migration #4271, no changelog entry
Casualties All JOINs on user_id silently returned 0 rows for 3 weeks
// BigQuery didn't error — it just returned nothing
INC-003
Change: NOT NULL constraint removed from email
Detected after 6 days
CREATE TABLE customers ( id INT PRIMARY KEY, - email TEXT NOT NULL, + email TEXT, segment TEXT );
Buried in PR #892, merged with no review
Casualties Email campaign sent to 14,000 null addresses. Marketing lost a week.
// "it's just a constraint change, what could go wrong"
INC-004
Change: Status enum expanded from 5 to 17 values
Detected after 2 months
-- downstream CASE statement: CASE status WHEN 1 THEN 'active' WHEN 2 THEN 'churned' WHEN 3 THEN 'suspended' WHEN 4 THEN 'trial' WHEN 5 THEN 'vip' - END -- values 6-17: ??? + ELSE 'unknown' -- 68% of rows
Buried in API docs footer, gray text on gray background
Casualties Every CASE WHEN assumed 5 statuses. 68% of customers labeled "unknown" for 2 months.
// the board was making decisions based on these numbers
INC-005
Change: Timestamps shifted from UTC to America/Los_Angeles
Detected after 19 days
events.created_at: - 2026-02-15T08:00:00Z + 2026-02-15T00:00:00-08:00 -- daily aggregation now -- attributes events to wrong day
Buried in Infrastructure Slack, page 3 of thread
Casualties All daily aggregations off by 8 hours. Finance closed the wrong month.
// nobody subscribed to the #infrastructure channel
INC-006
Change: Column dropped from source table
Detected after 4 days
pricing.products: product_id: INT64 base_price: DECIMAL(10,2) - discount_pct: FLOAT64 currency: STRING
Buried in Sprint retrospective notes, bullet point 14
Casualties Pricing model outputting NULLs. Finance team "reconciling" for a week.
// SELECT discount_pct FROM ... → NULL, no error
INC-007
Change: Column default changed from NULL to 'USD'
Detected after 5 weeks
transactions: amount: DECIMAL(10,2) - currency: STRING DEFAULT NULL + currency: STRING DEFAULT 'USD' -- multi-currency logic: -- WHERE currency IS NOT NULL
Buried in Commit message of an unrelated refactor
Casualties Multi-currency reporting showed everything as USD. CFO presented wrong numbers to investors.
// the WHERE clause worked perfectly — on wrong assumptions
INC-008
Change: Entire table replaced, same view alias
Detected after 48 hours
-- view: public.orders - SELECT * FROM orders_legacy + SELECT * FROM orders_v2 -- orders_legacy.total → DECIMAL -- orders_v2.total → STRING ("$12.50")
Buried in Mentioned in Friday standup. Nobody took notes.
Casualties 48 hours of wrong data in the warehouse. SUM(total) returned 0.
// the view name didn't change, so nobody checked
0
Average days to detection

How long would your team take to notice?

Submit incident

Got a worse one?

Submit an upstream change that broke your pipeline. Anonymize everything. The worst ones get catalogued here.

Why do upstream changes go unnoticed?

Because changelogs are buried in PRs you'd never think to review. Because "minor version" doesn't mean "minor impact". Because the team that changed the schema doesn't know your pipeline exists.

Every incident here is a symptom of the same gap: there's no tooling to connect upstream changes to downstream impact.

0
Incidents catalogued
0
Avg days undetected
0
Preventable
0
Tools that track upstream
Early detection system

Every incident here was preventable.

cairndata crawls your upstream repos, detects schema changes before they hit production, and fixes your dbt models automatically. No more Monday morning surprises.

No spam. One email when we launch. Privacy Policy