For about eleven months, every webhook Stripe sent to Crow, my SaaS for independent auto repair shops, got the same answer: 405 Method Not Allowed. Not from my code. From nginx, which served that URL out of the static frontend build and had no idea what to do with a POST. Every delivery failed until Stripe disabled the endpoint on its own. My backend logged nothing, because my backend never saw a request.
The part that stings is a comment in backend/main.py, sitting right above the line that mounts the payment router: prefix="/api" is REQUIRED. The code knew. The Stripe dashboard didn't, and the dashboard isn't in the repo.
Behind it sat a mismatched signing secret, a column nothing wrote, a tier name the database rejected, and prices that didn't match Stripe. The fixes that mattered weren't more unit tests. They were scripts that compare the repo against the real world.
Why nobody noticed
Two honest reasons. First, the happy path didn't need webhooks. Crow's checkout endpoint writes the user's tier and subscription row synchronously, right after the charge. Webhooks carry everything that happens later: renewals, cancellations, failed cards. If a customer had cancelled in Stripe or a card had declined, Crow would never have found out.
Second, there wasn't much "later" to carry. When I finally pulled the figures from live Stripe instead of my own database, almost all the active subscriptions on the account were mine: dogfooding subscriptions I'd created in live mode. Even my notes were wrong: a customer count in the repo's CLAUDE.md was counting the free-tier rows that registration inserts.
So the exposure was almost entirely my own accounts. That's luck, not design. The GitHub issue I opened on August 29 put it plainly: "If a mechanic subscribes today, the app will not know."
Layer one: a URL missing four characters
The payment router is a FastAPI APIRouter with prefix="/payments", mounted under /api. The real route is /api/payments/webhook. nginx sends /api/ to the backend and everything else to the single-page app.
The live endpoint in the Stripe dashboard was registered as https://crowapp.ca/payments/webhook. No /api. nginx served that path from the static frontend, answered the POST with 405, and kept doing so until Stripe auto-disabled the endpoint.
My first diagnosis was wrong as well. In the PR that fixed subscription linking, I wrote that "no webhook endpoint appears to be registered in the Stripe Dashboard." One was registered. It pointed at the wrong URL, and Stripe had switched it off.
Layer two: the right URL, the wrong secret
Fixing the path didn't make events arrive. It moved the failure.
Production was running live STRIPE_SECRET_KEY and STRIPE_PUBLISHABLE_KEY values alongside a test-mode STRIPE_WEBHOOK_SECRET. That secret had been copied from a test-mode endpoint, and the test endpoint happened to have the correct URL. So once the live endpoint pointed at the right place, deliveries reached the handler and failed stripe.Webhook.construct_event with a 400 Invalid signature. From the outside it looked just as quiet: Stripe saw 400s, and I had an ERROR line in a container log nobody was reading.
The first misconfiguration had been hiding the second. You can't see the second bug until the first one is fixed, and you won't fix the first until something tells you it exists.
Layer three: a perfect delivery would still have done nothing
handle_subscription_updated and handle_subscription_deleted looked users up with WHERE stripe_subscription_id = $1. No code path ever wrote that column. The only place anything mentioned setting it was a comment. Every subscription webhook would have matched zero users and returned success.
Nothing handled the events that create a subscription, either. customer.subscription.created and checkout.session.completed weren't in the dispatch chain.
The fix (PR #69) added handle_subscription_created, which joins on stripe_customer_id because checkout does set that one, plus handle_checkout_completed. Unknown customers and malformed events log and return success rather than raising, because Stripe retries any non-2xx response for up to three days. There's a test for exactly that: test_created_handler_survives_unknown_customer. A backfill script, dry run by default, linked the subscriptions that already existed.
The irony: in May I had shipped idempotent webhook handling, a processed_stripe_events table with an INSERT ... ON CONFLICT DO NOTHING RETURNING gate, to protect against Stripe's retries. It was careful deduplication for an endpoint that had never received a single event. The table got its first row on September 6.
The paying mechanic who got locked out was me
On September 5 I opened the mechanic dashboard as an existing account and got bounced to the plan picker, with no way in except paying again. users.subscription_tier said essential, subscriptions.tier said free, and the computed trial had ended on 2026-01-03.
The checkout code held two mappings for the same concept. The price lookup mapped a mechanic's basic to mechanic_essential. The code that stored the tier mapped it to essential, and essential isn't a value the subscriptions_tier_check constraint allows. So on every mechanic checkout:
UPDATE users SET subscription_tier = 'essential' -- succeeds
DELETE FROM subscriptions WHERE user_id = $1 -- removes the row
INSERT INTO subscriptions (tier = 'essential') -- CHECK violation
except Exception: logger.error(...) -- swallowed
The customer gets charged and ends up with no subscription row at all. The next read goes through get_user_subscription, which lazily inserts a row hardcoded to free, so it's a read that writes. The trial gate then sees free plus a trial computed from account creation. Thirty days after signup, a paying customer hits the paywall. The parts-service entitlement check had also never heard of essential, so paid features were quietly denied along the way.
That essential mapping went in on August 13, 2025, in a commit titled "fixed subscription setup."
Who did it actually hit? Migration 045, which rewrote the bad values, touched two rows in production, and both were my own accounts. The only account ever charged the mechanic plan's live price was my dogfooding subscription. The path was live and it was broken. It just hadn't met a real mechanic yet, and it would have locked out the first one 30 days after they signed up.
The fixes, across PRs #82 and #84:
- One
canonical_tier(user_type, tier)function that both call sites go through, so they can't drift apart again. create_default_subscriptionseeds fromusers.subscription_tierinstead of hardcodingfree. It can only copy what the users table already claims.- Migration 045 rewrites the legacy values, and only for mechanics.
- When the two tables disagree, the paywall now resolves toward access. Wrongly admitting a free user costs some unpaid usage. Wrongly locking out a paying one costs the customer.
- The swallowed exception stays non-fatal, because the charge has already succeeded and failing the request won't undo it. It's no longer quiet, though:
# Before (payment_routes.py, simplified)
except Exception as e:
logger.error(f"Failed to create subscription record: {e}")
# Don't fail the whole request if subscription record creation fails
# After
except Exception as e:
logger.critical(
"PAID BUT NOT PROVISIONED: user %s paid for tier %s and the "
"subscriptions row could not be written: %s. They have been "
"charged and will lose access when the trial window lapses. "
"Fix the row by hand.",
user_id, subscription_tier_to_store, e,
)
Four copies of every price, and only one takes money
Crow states each plan's price in four places: the sales site, the app UI, a backend table, and Stripe. Only Stripe takes money, and a Price object's amount can't be edited, so Stripe is the authority. The other three had all drifted:
- The site, app and backend advertised the mechanic plan at $9.99 a month. Stripe charged $29.99 CAD. Again, the only account ever billed at that price was mine.
- The lifetime purchase created a PaymentIntent for 4999 cents with
currency="usd"hardcoded, bypassing its CAD Price entirely. The site says "Prices in CAD," so any Canadian buyer would have paid roughly 37% more than advertised. - The promo code path priced car-owner Professional at $29.99 a month. The plan is $1.99, so every recorded discount was computed off a base about 15x too high.
- Analytics turned annual prices into monthly ones by multiplying by a flat 0.833, a 16.7% discount no tier actually offers.
- Five configured price IDs returned 404 in live mode, so Car Owner Basic checkout could only ever fail.
The fix in PR #86 collapsed the private price lists into one advertised_cost() helper that reads SUBSCRIPTION_TIERS. test_helper_reads_the_table_rather_than_a_copy_of_it patches that table and requires the helper's answer to change with it, so a reintroduced local copy fails CI.
The repo can't see its own drift
The failures that hid everything else lived outside the codebase: a URL in the Stripe dashboard, a secret in an env file on the server, Price objects in Stripe. Around then the backend suite had 550 passing tests. None of them could have caught the URL, the secret or the prices, because the paths and numbers in the repo were copies of each other, not of the real system. The commit message for the pricing fix says it plainly: "the repo cannot see its own drift."
So the fixes that stop it recurring are two operator scripts. Each compares the repo's assumptions against the live world and exits non-zero on disagreement.
check_stripe_webhook.py works from both ends and doesn't trust either one alone. It asks Stripe for the endpoint URLs it's actually configured to call and POSTs an unsigned payload to each. Only the real handler answers that with a 400 Invalid signature. A 404, a 405, or a 200 full of SPA HTML means deliveries are going nowhere. Then it compares Stripe's recent event volume with what landed in processed_stripe_events:
# check_stripe_webhook.py (simplified)
for endpoint in stripe_get(key, "/webhook_endpoints")["data"]:
if endpoint["status"] != "enabled":
fail("Stripe will not deliver to this endpoint")
code, body = probe(endpoint["url"]) # unsigned POST, bogus signature
reachable = code == 400 and "Invalid signature" in body
if not reachable:
fail(f"HTTP {code}: NOT the handler")
# Same 30-day window on both sides: Stripe's events vs our table.
if sent and not processed:
fail("Stripe has activity and NOTHING reached the database")
The docstring puts it better: "A probe can only prove the door opens; this proves someone came through it." It also deliberately doesn't check the URL against a path constant in the code, because "a constant is just a second copy of the assumption that was already wrong."
check_stripe_prices.py does the same for money. It reads every configured Stripe price ID environment variable, fetches the live Price, and diffs it against the backend table:
# check_stripe_prices.py (simplified)
for name, pid in configured_price_ids.items():
try:
price = stripe_get(key, f"/prices/{pid}")
except urllib.error.HTTPError as e:
if e.code == 404:
fail("MISSING", name) # checkout using this price fails outright
continue
dollars = price["unit_amount"] / 100
if price["currency"].upper() != "CAD":
fail("CURRENCY", name)
tier, field = EXPECTED[name]
if round(float(tiers[tier][field]) - dollars, 2) != 0:
fail("AMOUNT", name)
Both are read-only and run with one docker exec. They just look at the thing the repo can't see.
Same shape, smaller: migration 045 silently failed to stage, because a .gitignore rule meant for database backups matched every file ending in .sql, migrations included. Six older migrations, including the whole fleet schema, were on the production server and nowhere in git.
Verified in production
On September 6 I corrected the endpoint URL and re-enabled it, installed the live signing secret, and recreated the API container. Then I replayed a live customer.subscription.updated event. One second later the logs showed Received Stripe webhook, processed_stripe_events got its first row ever, and a repeat delivery logged Skipping duplicate Stripe webhook, so the May idempotency work finally did its job. The probe fails against the old URL with a 405 and passes against the real one with a 400.
Lessons learned
1. Watch outcomes, not the absence of errors. Nothing errored for eleven months, because the failure was a response my code never received. The signal that would have caught it on day one was "Stripe has events and my table has zero rows."
2. Config that lives outside the repo needs checks that live outside the tests. A URL in the Stripe dashboard, a secret in an env file, and a Price object in Stripe can't be unit tested. Write a script that asks the real system and compares.
3. Don't encode an assumption twice. A path constant in the check would have agreed with the code and still been wrong about the dashboard. Probe the actual thing. The same goes for prices: one table, and something that verifies it against Stripe.
4. Degrade visibly, never silently. I'd written that rule into the spec for a different feature the same week. The swallowed logger.error was the payments version of breaking it. In payment code, "don't fail the request" is often right. "Don't tell anyone" never is.
5. Low volume hides bugs. Every one of these was live and reachable. What kept them harmless was that almost nobody but me was paying yet. That's not a defense. It's a deadline, and the first real customer would have hit it.
Running on Stripe and not sure your webhooks are landing? Let's talk, or see what I built into Crow.