On-device intelligence: the landscape · intro

On-device AI: the landscape and the judgment

Why inference is moving onto the phone, the five tiers of Apple intelligence, and the architecture judgment — where does this feature's model actually live?

12 min read10 min practice0/2 exercises5 recall cards

By the end you will be able to

Argue the four forces that pull inference on-device — and the workloads that stay server-side
Place any AI feature on the five-tier ladder, walking down only on requirement failures
Quote the on-device model's honest constraints (size, context, device floor) from Apple's own docs
Design for unavailability — the enum every on-device feature must handle

Guess firstAnswering before you read makes the explanation stick — even when you get it wrong.

Product asks for "an AI summary of the user's monthly spending" in your banking app. What is the first architectural question a senior engineer asks?

Why inference is moving onto the phone

Four forces, and every one of them is sharper in fintech than almost anywhere else:

Privacy as a feature, not a policy. Data that never leaves the device needs no consent screen for cloud processing, no data-processing agreement, no residency analysis. "The prompt is generated entirely on device — personal entries stay personal" is Apple's own developer pitch, and for regulated data it converts an AI feature from a compliance project into a product feature.
Latency. No round trip: first tokens in tens of milliseconds, streaming as the model generates. Categorising a transaction as the cell renders is feasible; a 900ms server hop per cell is not.
Cost. The platform model is free at any scale — zero marginal cost per request. A feature that calls a metered cloud API on every keystroke has a unit-economics problem; the same feature on-device has none.
Offline. The model in your pocket works on a plane. Any feature built on it inherits the offline story your sync lesson (D2-04) fought so hard for.

And the honest counterweight, immediately: serious workloads still live server-side. Fraud models score transactions on the backend where they see the whole graph, not one phone's slice. Long-document analysis and deep reasoning outgrow a small model. The skill interviews actually probe is not "put everything on the phone" — it is knowing which tier each feature belongs to, and saying why.

The five-tier ladder

Apple's stack is a ladder. The senior habit: start at the top, walk down only when a genuine requirement fails.

Tier	What	Costs you	Fails when
0 — Built-in frameworks	Vision (OCR), Natural Language, Speech, Translation	nothing — no model shipped, no prompt written	the task needs open-ended generation
1 — Platform model, on-device	Foundation Models framework: the ~3B Apple model on the phone	4,096-token context; eligible devices only	context, capability, or device floor
2 — Private Cloud Compute	Apple's server models behind the same framework (iOS 27+)	network required; entitlement + quotas; 32K context	you need a frontier model or your own stack
3 — Third-party models	Claude/Gemini via the `LanguageModel` protocol (iOS 27+), or plain HTTPS APIs	per-token cost, data leaves the device, DPAs	cost or privacy kills it
4 — Your own model	Core ML / Create ML / MLX — a model you convert, compress, ship	the whole ML engineering discipline	(this is the floor)

Two things make this ladder more than a slide from a keynote. First, tier 0 is criminally underused — receipt scanning is Vision's RecognizeTextRequest, not a prompt. Second, since iOS 27 tiers 1–3 sit behind one Swift API: LanguageModelSession takes a model, and Apple's on-device model, Apple's cloud model, and third-party providers all conform to the same LanguageModel protocol. The routing decision became a value you pass, which is precisely what makes it testable architecture instead of vendor lock-in.

The ladder as a policy — feature requirements in, tier and rationale out.

struct FeatureSpec {
    let name: String
    let needsGeneration: Bool     // open-ended text, or a fixed perception task?
    let sensitiveData: Bool
    let contextTokens: Int
    let mustWorkOffline: Bool
}

func tier(for spec: FeatureSpec) -> String {
    if !spec.needsGeneration {
        return "Tier 0 — a built-in framework already does this"
    }
    if spec.contextTokens <= 3000 && spec.mustWorkOffline {
        return "Tier 1 — on-device platform model (fits the window, works offline)"
    }
    if spec.contextTokens <= 3000 {
        return "Tier 1 — on-device platform model (free, private, fast)"
    }
    if spec.sensitiveData {
        return "Tier 2 — Private Cloud Compute (32K window, data stays in Apple's enclave)"
    }
    return "Tier 3 — frontier model via LanguageModel protocol or API"
}

let features = [
    FeatureSpec(name: "receipt scan", needsGeneration: false, sensitiveData: true, contextTokens: 0, mustWorkOffline: true),
    FeatureSpec(name: "spending summary", needsGeneration: true, sensitiveData: true, contextTokens: 1200, mustWorkOffline: true),
    FeatureSpec(name: "annual portfolio report", needsGeneration: true, sensitiveData: true, contextTokens: 22000, mustWorkOffline: false),
    FeatureSpec(name: "market news digest", needsGeneration: true, sensitiveData: false, contextTokens: 15000, mustWorkOffline: false),
]

for feature in features {
    print("\(feature.name): \(tier(for: feature))")
}

struct FeatureSpec {
    let name: String
    let needsGeneration: Bool     // open-ended text, or a fixed perception task?
    let sensitiveData: Bool
    let contextTokens: Int
    let mustWorkOffline: Bool
}

func tier(for spec: FeatureSpec) -> String {
    if !spec.needsGeneration {
        return "Tier 0 — a built-in framework already does this"
    }
    if spec.contextTokens <= 3000 && spec.mustWorkOffline {
        return "Tier 1 — on-device platform model (fits the window, works offline)"
    }
    if spec.contextTokens <= 3000 {
        return "Tier 1 — on-device platform model (free, private, fast)"
    }
    if spec.sensitiveData {
        return "Tier 2 — Private Cloud Compute (32K window, data stays in Apple's enclave)"
    }
    return "Tier 3 — frontier model via LanguageModel protocol or API"
}

let features = [
    FeatureSpec(name: "receipt scan", needsGeneration: false, sensitiveData: true, contextTokens: 0, mustWorkOffline: true),
    FeatureSpec(name: "spending summary", needsGeneration: true, sensitiveData: true, contextTokens: 1200, mustWorkOffline: true),
    FeatureSpec(name: "annual portfolio report", needsGeneration: true, sensitiveData: true, contextTokens: 22000, mustWorkOffline: false),
    FeatureSpec(name: "market news digest", needsGeneration: true, sensitiveData: false, contextTokens: 15000, mustWorkOffline: false),
]

for feature in features {
    print("\(feature.name): \(tier(for: feature))")
}

Note what the policy did not need: a quality ranking of models. Requirements route features; benchmarks break ties. (And one honest asterisk: a fraud-scoring model isn't on this ladder at all — it lives server-side, where it can see patterns across millions of accounts. Knowing what doesn't belong on the phone is part of the answer.)

The constraints, quoted not vibed

The on-device model is a ~3-billion-parameter language model, quantised aggressively to fit a phone. Apple documents — in the framework's own overview — that it is not suited to basic math, code generation, or complex logical reasoning, and it is not a general-knowledge chatbot. What it is built for: summarise, extract, classify, tag, rewrite, generate short structured content — which happens to be most of what product actually asks for.

Three numbers to say in an interview:

4,096 tokens per session — instructions, prompts, and outputs all draw from one budget. Roughly 3–4 English characters per token; roughly one character per token for Chinese, Japanese and Korean — the same feature holds a quarter of the conversation in Tokyo that it holds in London.
The device floor: Apple Intelligence hardware — iPhone 15 Pro or newer, A17 Pro/M1-class iPads and Macs. A meaningful slice of your installed base cannot run tier 1 at all, for years yet.
Zero — the marginal cost per request, and also the amount of your data Apple trains on from these calls.

Predict, then runA wrong prediction you have committed to is worth more than a right answer you read.

A support-chat feature spends 400 tokens on instructions, then each turn costs ~600 tokens (user message + model reply). The session throws contextSizeExceeded when the 4,096-token budget would be blown. Which turn dies?

let contextSize = 4096
let instructions = 400
let perTurn = 600

var used = instructions
var turn = 0
while true {
    turn += 1
    if used + perTurn > contextSize {
        print("turn \(turn): throws contextSizeExceeded")
        break
    }
    used += perTurn
    print("turn \(turn): ok — \(used)/\(contextSize) used")
}

Design for `unavailable` — always

Tier 1 is a conditional capability, and the framework makes the conditions explicit. Real API, worth reading closely even before the deep-dive lesson:

import FoundationModels

let model = SystemLanguageModel.default

switch model.availability {
case .available:
    // build the AI path
case .unavailable(.deviceNotEligible):
    // hardware floor: feature hidden or served another way — forever, on this device
case .unavailable(.appleIntelligenceNotEnabled):
    // user's choice: invite, never nag
case .unavailable(.modelNotReady):
    // model still downloading — transient; retry later
case .unavailable(let other):
    // future reasons: degrade gracefully
}

Three different unavailability reasons, three different product responses — a permanent hardware floor, a respected user choice, and a transient download. Shipping an AI feature without designing all three paths is the new "works on my Wi-Fi".

Check yourself

The PM's launch plan says: "AI spending summaries for all users at launch — it's on-device, so it's free." What does the senior review flag?

Why this is worth your study hours

Three signals, stated the way you'd defend the investment to a skeptical tech lead. The platform is committing: three consecutive WWDCs of escalating investment — the framework itself (2025), then multimodal input, Private Cloud Compute access, and the third-party LanguageModel protocol (2026) — with the core framework going open source. Adoption is real: within months of iOS 26, dozens of mainstream apps shipped Foundation Models features — journaling summaries, document Q&A, spending categorisation — because AI slots into existing product surfaces, unlike the AR wave that demanded new ones. The skill compounds: because tiers 1–3 share one API, what you learn is not "Apple's model" but session management, guided generation, tool calling and routing judgment — portable across every provider that conforms.

The counterweight, one more time, because honesty is the house style: nobody is hiring "on-device ML engineers" by the thousand. They are hiring senior iOS engineers who can also make these calls — and putting "chose tiers for AI features; shipped the unavailability design" in your interview stories is exactly the pool-narrowing this track is for.

⌘Stand on the ladder yourself

On an Apple Intelligence device (or a supported simulator), build a two-screen app that switches on SystemLanguageModel.default.availability and renders a different explanation per case. Toggle Apple Intelligence off in Settings and watch your .appleIntelligenceNotEnabled path run.
Watch WWDC25 session 360 ("Discover machine learning & AI frameworks on Apple platforms") — the ladder, from Apple's own mouth — and session 286 ("Meet the Foundation Models framework"). Note every point where the presenter names a constraint; those are interview lines.
Send one real prompt: try await LanguageModelSession().respond(to: "Summarise: coffee 4.50, rent 1200, groceries 82.30") in a project's async context, and stream the same prompt with streamResponse(to:) to feel tokens arrive.
Write the one-page tier assignment for an app you know well: five features, a tier each, one sentence of rationale — the artifact you'd bring to a system-design interview.

This one needs the real Xcode — the browser runtime cannot model it.

›

Encode the ladder

Write the code

Implement the tier policy — the reasoning you would narrate in a system-design interview. Requirements route; walk down only on failure.

struct FeatureSpec {
    let name: String
    let needsGeneration: Bool
    let sensitiveData: Bool
    let contextTokens: Int
}

// no generation needed            -> "tier 0"
// fits 3000 tokens                -> "tier 1"
// too big but sensitive           -> "tier 2"
// too big, not sensitive          -> "tier 3"
func tier(for spec: FeatureSpec) -> String {
    return ""
}

let specs = [
    FeatureSpec(name: "card scan", needsGeneration: false, sensitiveData: true, contextTokens: 0),
    FeatureSpec(name: "merchant cleanup", needsGeneration: true, sensitiveData: true, contextTokens: 400),
    FeatureSpec(name: "statement analysis", needsGeneration: true, sensitiveData: true, contextTokens: 18000),
    FeatureSpec(name: "news roundup", needsGeneration: true, sensitiveData: false, contextTokens: 12000),
]

for spec in specs {
    print("\(spec.name): \(tier(for: spec))")
}

Solution unlocks after 3 attempts

›

The budget that decides your UX

Write the code

Model the context ledger: sessions accumulate tokens and die at the ceiling. Your function reports how many full exchanges a design gets before the wall — the number that decides whether "chat" is even the right UI.

// Returns how many complete turns fit before the budget would be exceeded.
// A turn only counts if instructions + (turns * perTurn) stays within contextSize.
func turnsAvailable(contextSize: Int, instructions: Int, perTurn: Int) -> Int {
    return 0
}

print(turnsAvailable(contextSize: 4096, instructions: 400, perTurn: 600))
print(turnsAvailable(contextSize: 4096, instructions: 1500, perTurn: 600))
print(turnsAvailable(contextSize: 32000, instructions: 400, perTurn: 600))
print(turnsAvailable(contextSize: 4096, instructions: 4200, perTurn: 600))

Solution unlocks after 3 attempts

Design itSaved on this device. Never graded.

Take an app you know well — your bank's, ideally. Choose five AI-shaped features (at least one perception task, one generation task on sensitive data, one long-context task). For each: assign a tier, name the requirement that placed it there, and write the one-sentence unavailability plan for below-the-floor devices. Then mark the one feature you would refuse to put on-device at all, and say why in compliance language.

Checkpoint

You can now:

Argue the four on-device forces and name the workloads that stay server-side
Route features down the five-tier ladder by requirement, not by hype
Quote the constraints — ~3B, 4,096 tokens, the device floor — and Apple's own not-for list
Treat unavailability as a designed product state with three distinct causes

Next up: tier 0 in depth — the intelligence frameworks that ship no model at all.

How well do you know this now? Rating yourself honestly, then being tested on it, is how you find out where your intuition is wrong.