Shipping · advanced

Debugging, crash triage and CI

Reading crash reports like evidence, the LLDB moves that pay daily, and the pipeline design that leads defend in interviews.

13 min read13 min practice0/2 exercises5 recall cards

By the end you will be able to

Triage a crash report from its signal, termination reason and thread state
Use the debugger deliberately — exception breakpoints, po/v, symbolic breakpoints
Design a CI pipeline with fast feedback ordering and defend its gates

Guess firstAnswering before you read makes the explanation stick — even when you get it wrong.

A crash report lands on your desk: EXC_BREAKPOINT (SIGTRAP), crashing thread in your code at Array.subscript.getter. No debugger was attached. What does it tell you?

The crash report as evidence

A crash report answers four questions, in order:

1. What kind of death? — the exception type.

Signature	Meaning	First suspect
`EXC_BREAKPOINT (SIGTRAP)`	Swift/system trap — a check chose to stop	force-unwrap, subscript range, precondition
`EXC_BAD_ACCESS (SIGSEGV/SIGBUS)`	touched memory that isn't yours	object used after free, threading race, C interop
`SIGABRT`	abort — often an uncaught NSException	unrecognised selector, layout exceptions; check the last exception backtrace
Termination: watchdog `0x8badf00d`	"ate bad food" — main thread hung past the limit	synchronous work at launch/foreground (lifecycle lesson)
Jetsam / `EXC_RESOURCE`	memory limit	the growth loop from the Instruments lesson

2. Where? — the crashed thread's stack. Only meaningful once symbolicated. The shipped binary keeps only street addresses; the dSYM is the phone book that turns them back into names — and only the edition printed for that exact build UUID decodes that build's addresses. That is why build settings upload dSYMs to your crash service, and why a report full of raw hex means the right edition is missing.

3. Everyone else? — the other threads. For deadlocks and races the crashed thread is only half the story; the thread holding what everyone awaits is the culprit.

4. How often, and to whom? — the aggregate. One crash is an anecdote; the Organizer/MetricKit view — "94% on one device model", "started at version 3.2.1", "always within 10s of launch" — is the diagnosis. MetricKit (MXCrashDiagnostic, hang diagnostics) delivers this telemetry from the field even without a third-party SDK.

The triage table as a function — signature in, diagnosis and next step out.

struct CrashReport {
    let id: Int
    let exceptionType: String
    let crashedFrame: String
    let terminationCode: String
}

func triage(_ crash: CrashReport) -> String {
    if crash.terminationCode == "0x8badf00d" {
        return "watchdog: main thread blocked too long — audit synchronous launch/foreground work"
    }
    if crash.exceptionType == "EXC_RESOURCE" {
        return "memory limit: run the Allocations generation loop on the suspect screen"
    }
    switch crash.exceptionType {
    case "EXC_BREAKPOINT":
        if crash.crashedFrame.contains("subscript") {
            return "swift trap: index out of range at \(crash.crashedFrame)"
        }
        return "swift trap: check force-unwraps and preconditions at \(crash.crashedFrame)"
    case "EXC_BAD_ACCESS":
        return "memory misuse: lifetime or threading race near \(crash.crashedFrame) — try TSan/ASan"
    case "SIGABRT":
        return "uncaught exception: read the last exception backtrace, not the crashed thread"
    default:
        return "unknown signature: escalate with the full report"
    }
}

let inbox = [
    CrashReport(id: 1, exceptionType: "EXC_BREAKPOINT", crashedFrame: "FeedModel Array.subscript.getter", terminationCode: ""),
    CrashReport(id: 2, exceptionType: "EXC_BAD_ACCESS", crashedFrame: "AudioEngine.render", terminationCode: ""),
    CrashReport(id: 3, exceptionType: "", crashedFrame: "main", terminationCode: "0x8badf00d"),
    CrashReport(id: 4, exceptionType: "SIGABRT", crashedFrame: "UIKitCore objc_exception_throw", terminationCode: ""),
]

for crash in inbox {
    print("#\(crash.id): \(triage(crash))")
}

struct CrashReport {
    let id: Int
    let exceptionType: String
    let crashedFrame: String
    let terminationCode: String
}

func triage(_ crash: CrashReport) -> String {
    if crash.terminationCode == "0x8badf00d" {
        return "watchdog: main thread blocked too long — audit synchronous launch/foreground work"
    }
    if crash.exceptionType == "EXC_RESOURCE" {
        return "memory limit: run the Allocations generation loop on the suspect screen"
    }
    switch crash.exceptionType {
    case "EXC_BREAKPOINT":
        if crash.crashedFrame.contains("subscript") {
            return "swift trap: index out of range at \(crash.crashedFrame)"
        }
        return "swift trap: check force-unwraps and preconditions at \(crash.crashedFrame)"
    case "EXC_BAD_ACCESS":
        return "memory misuse: lifetime or threading race near \(crash.crashedFrame) — try TSan/ASan"
    case "SIGABRT":
        return "uncaught exception: read the last exception backtrace, not the crashed thread"
    default:
        return "unknown signature: escalate with the full report"
    }
}

let inbox = [
    CrashReport(id: 1, exceptionType: "EXC_BREAKPOINT", crashedFrame: "FeedModel Array.subscript.getter", terminationCode: ""),
    CrashReport(id: 2, exceptionType: "EXC_BAD_ACCESS", crashedFrame: "AudioEngine.render", terminationCode: ""),
    CrashReport(id: 3, exceptionType: "", crashedFrame: "main", terminationCode: "0x8badf00d"),
    CrashReport(id: 4, exceptionType: "SIGABRT", crashedFrame: "UIKitCore objc_exception_throw", terminationCode: ""),
]

for crash in inbox {
    print("#\(crash.id): \(triage(crash))")
}

Four reports, four different investigations — and none of them started by re-reading the app's code hoping to spot something. The signature routes the work; that routing is what "senior at debugging" means before any tool opens.

LLDB: the five moves that pay daily

The debugger rewards a small deliberate vocabulary far more than memorising its manual:

The Exception Breakpoint — Breakpoint navigator ▸ + ▸ Exception Breakpoint. Stops at the throw site of any exception instead of the crash aftermath in main. The single highest-value setting in Xcode; turn it on in every project, today.
Swift Error Breakpoint — same menu: stops wherever a Swift throw executes, invaluable when some layer is swallowing errors (try?, the networking lesson's warning) and you cannot find where.
v (frame variable), then po — v prints the variable without running code (fast, reliable); po evaluates an expression with full dynamic description (powerful, can itself run side effects). Habit: v first, po when you need computation.
Symbolic breakpoints — break on a function name you don't own: UIViewAlertForUnsatisfiableConstraints for Auto Layout complaints, or any framework method you suspect. With a condition (item.id == 42) and "log + auto-continue", a breakpoint becomes a tracepoint — printf debugging without rebuilding.
Watchpoints — watchpoint set variable count: stop when memory changes, catching "who is mutating this?" cases that no code-reading finds.

And the two runtime sanitizers that belong in your scheme for a day whenever memory or threading is suspected: Address Sanitizer (use-after-free, buffer overruns — turns EXC_BAD_ACCESS roulette into a named report) and Thread Sanitizer (data races — the pre-Swift-6 escape hatches like @unchecked Sendable are exactly where it still earns its keep).

CI: the lead-round conversation

The interview prompt is "design our pipeline." The senior answer is an ordering principle, then stages:

Fast feedback first: cheap, high-signal gates before expensive ones — fail in minutes, not hours.

PR opened
  1. lint + format          (~1 min)   style debates end here, not in review
  2. build all targets      (~5 min)   compiler errors are the cheapest test suite
  3. unit tests             (~5 min)   the P1 suites: pure logic, fakes, no simulator UI
  4. UI smoke subset        (~10 min)  launch + critical paths only — full suite is nightly
  ── merge allowed ──
  5. nightly: full UI suite, performance tests (XCTOSSignpostMetric baselines)
  6. on main: archive, upload dSYMs, TestFlight internal   (the shipping lesson's loop)

The points that separate a lead answer from a listing:

Signing in CI is the perennial swamp: solved with cloud-managed signing (Xcode Cloud) or a shared-certificate repo (fastlane match) — never by hand-installed certificates on a snowflake build machine that one person understands.
Build numbers come from the counter CI owns — the shipping lesson's monotonic rule, automated.
Flaky tests get quarantined, not deleted and not retried-forever: a flaky test that gates merges trains the team to ignore red, which is the real cost. Quarantine list + owner + deadline.
dSYM upload is a pipeline step, because the crash-triage half of this lesson is dead without symbolication — the two topics are one system.

Check yourself

Users report the app "freezes then closes" on the feed. The Organizer shows terminations with 0x8badf00d clustered on older devices, always within seconds of foregrounding. Your first move?

›

Order the pipeline

Write the code

Implement the fast-feedback rule: order the stages by cost, gate merges before the expensive tail, and report the failure point of a broken PR.

struct Stage {
    let name: String
    let minutes: Int
    let mergeGate: Bool     // must pass before merge?
    let passes: Bool
}

let stages = [
    Stage(name: "full UI suite", minutes: 45, mergeGate: false, passes: true),
    Stage(name: "unit tests", minutes: 5, mergeGate: true, passes: false),
    Stage(name: "lint", minutes: 1, mergeGate: true, passes: true),
    Stage(name: "build", minutes: 5, mergeGate: true, passes: true),
    Stage(name: "UI smoke subset", minutes: 10, mergeGate: true, passes: true),
]

func run(_ stages: [Stage]) -> [String] {
    // 1. Merge gates first, sorted by minutes ascending (ties: alphabetical);
    //    then non-gates, same ordering.
    // 2. Run in order; print "pass <name> (<m>m)" per success.
    // 3. First failing MERGE GATE stops everything:
    //    "FAIL <name> (<m>m) — merge blocked after <total>m"
    return []
}

for line in run(stages) {
    print(line)
}

Solution unlocks after 3 attempts

›

Quarantine the flake

Write the code

Implement the flaky-test policy: consistent tests report normally; flaky ones are quarantined with an owner rather than gating merges or being deleted.

struct TestRecord {
    let name: String
    let last10: [Bool]      // recent pass/fail history
}

func verdict(for test: TestRecord) -> String {
    // all pass                  -> "green"
    // all fail                  -> "red — fix before merge"
    // mixed                     -> "QUARANTINE — flaky (<p>/10 passes), assign an owner"
    return ""
}

let suite = [
    TestRecord(name: "testCheckoutTotal", last10: [true, true, true, true, true, true, true, true, true, true]),
    TestRecord(name: "testLoginFlow", last10: [true, false, true, true, false, true, true, true, false, true]),
    TestRecord(name: "testMigrationV2", last10: [false, false, false, false, false, false, false, false, false, false]),
]

for test in suite {
    print("\(test.name): \(verdict(for: test))")
}

Solution unlocks after 3 attempts

⌘Practise on real evidence

Manufacture the three crashes: force-unwrap a nil, index past an array's end, and fatalError() — run without the debugger (uncheck "Debug executable" in the scheme's Run options, or launch from the home screen), then read each report in Window ▸ Organizer ▸ Crashes (or Devices ▸ view device logs). Match the signatures to this lesson's table.
Turn on the Exception Breakpoint and the Swift Error Breakpoint; throw an error through three layers and watch where each stops you.
Make a tracepoint: symbolic breakpoint on any method, add a "Log Message" action with @count@ items and auto-continue — printf debugging with no rebuild.
Break symbolication on purpose: strip a build's dSYM, view a raw-hex crash, then re-symbolicate with Xcode's CrashSymbolicator.py (or atos for a single frame) — after this you will never treat "upload dSYMs" as an optional pipeline step.
Wire the minimal pipeline: Xcode Cloud (or a GitHub Action running xcodebuild test) with lint → build → unit tests gating a PR — and confirm a deliberately broken test blocks the merge at the right minute.

This one needs the real Xcode — the browser runtime cannot model it.

Design itSaved on this device. Never graded.

Write the one-page "on-call debugging runbook" for your team: given a new crash group in the Organizer, the first four checks in order (signature → symbolication → aggregate pattern → thread state), which two breakpoints every project has on by default, and the rule for when a crash blocks the release train vs ships behind a phased rollout. This document is a genuinely useful artefact — teams that have it triage in minutes.

Checkpoint

You can now:

Route a crash report by signature before opening any code
Use the debugger's five high-leverage moves and the two sanitizers deliberately
Design and defend a fast-feedback pipeline with honest flake policy
Connect dSYMs, MetricKit and CI into one crash-response system

The interview-eight arc is complete — performance, caching, sync, security, Core Data, lifecycle, and triage: the senior loop's substance, all runnable.

How well do you know this now? Rating yourself honestly, then being tested on it, is how you find out where your intuition is wrong.