iOS Startup Performance in Practice, Part 2: Measuring App Launch Time in Production

How to measure startup on real user devices, define trustworthy production percentiles, and use XCTest and Instruments to explain regressions.

The easiest way to measure app launch time is to start a stopwatch, open the app, and write down the number.

It is also one of the easiest ways to produce a measurement that cannot be reproduced, compared, or trusted.

An iOS launch depends on more than the code in the repository. The result changes with the process state, system caches, device model, thermal state, available memory, network behavior, stored user data, app version, and the destination the user is trying to open. Two measurements can both be technically correct while describing different experiences.

In Part 1, we separated cold launch, warm launch, resume, scene creation, and entry points. We also defined three possible endpoints: First Draw, Meaningful Content, and Time to Interactive.

Now we can turn that model into a measurement system.

This article is the second part of iOS Startup Performance in Practice, a six-part series:

Understanding Launch Types and Entry Points
Measuring App Launch Time Correctly
Finding Bottlenecks from dyld to the First Frame
Designing a Scalable Startup Architecture
From the First Frame to an Interactive UI
Preventing Regressions in Production

The goal of this part is not to find a single perfect number. It is to build a production measurement system that describes what real users experience - and then use controlled tests and profiling to explain why that distribution changes.

A launch number is meaningless without a measurement contract

Suppose one engineer reports a launch time of 1.4 seconds and another reports 2.1 seconds. Before comparing them, we need to know:

Did both tests create a new process?
Were they cold or warm launches?
Was the app opened from the icon or through a link?
Did the timer stop at First Draw or when the screen became interactive?
Did both runs use the same device and OS version?
Was the debugger attached?
Did both users have the same local data and authentication state?
Was the network real, disabled, or stubbed?

Without these answers, the numbers are labels rather than evidence.

Let’s start with production metrics. For every launch metric, define a measurement contract:

Dimension	Example value
Population	Real user launches of app version `8.24.0`
Launch class	New-process launch; resume reported separately
Entry point	Icon, link, notification, and quick action segmented where custom telemetry allows it
Start boundary	System launch start for MetricKit; first application marker for custom metrics
End boundary	First Draw for the system metric; Home Interactive for the product metric
User segments	Signed-out and authenticated users reported separately
Technical segments	Device performance tier, iOS version, app version, and migration state
Aggregation window	Seven complete days after the target rollout reaches a stable cohort
Reported values	Every sample, median, p90, and maximum
Data source	MetricKit or Organizer for system launch; custom telemetry for product milestones
Exclusions	Internal builds, synthetic runs, invalid or incomplete milestone sequences

This table describes measurement on user devices. There are no fixed test phones, stubbed networks, or XCTest iterations in the production contract. Real device and network diversity are not noise to remove; they are part of the experience we are trying to understand.

The first rule of launch measurement is therefore:

Define the population and user-visible endpoint before naming the number.

Instead of writing “launch time is 1.8 seconds,” write:

For production new-process launches of version 8.24.x, Time to First Draw is 1.1 seconds at p50 and 1.8 seconds at p90 across 2.4 million samples. Authenticated users on the lowest device tier have a p90 of 2.6 seconds.

The values are illustrative, but the form is important: it tells us who experienced the delay, which boundary was measured, and whether the percentile is statistically meaningful.

Production requires two complementary clocks

There is a fundamental limitation in iOS startup measurement:

system metrics can observe work before application code begins, but they understand only system-defined launch milestones;
custom application metrics understand product milestones, entry points, and user state, but cannot start before the app has an opportunity to execute code.

This means one production number cannot answer every startup question.

System launch metric

Use Xcode Organizer or MetricKit to understand the system-observed launch distribution. These sources can describe Time to First Draw, optimized launch behavior influenced by prewarming, resume duration, and extended launch measurements supported by the OS version.

This is the authoritative source for questions such as:

How long does launch take before our code can draw the first frame?
Did p90 regress after the release?
Is the regression concentrated on a device or OS segment?
Did launch improve for real users rather than only in a laboratory setup?

Product startup metric

Use lightweight custom telemetry for milestones the operating system cannot infer:

initial route resolved;
home content meaningful;
requested deep-link destination visible;
primary action enabled;
destination interactive.

This is the authoritative source for questions such as:

How long does a user wait after opening a product link?
Are notification launches slower than icon launches?
Does authentication restoration delay the requested destination?
How often does a fast First Draw lead to a slow interactive screen?

Do not present an application timer that begins in didFinishLaunching as full launch time. Name it by its actual boundaries, for example AppCodeToHomeInteractive or RouteReceivedToDestinationInteractive.

Together, the two clocks give us a useful production model:

Metric family	Earliest observable start	Product context	Recommended use
Organizer / MetricKit	System launch	Limited	System launch distribution and device/OS trends
Custom milestone telemetry	First available application marker	Rich	Entry-point-specific Meaningful Content and TTI
XCTest	Test-controlled launch	Deterministic fixture	Reproduce and prevent a known regression
Instruments	Instrumented launch	Diagnostic trace	Explain where time is spent

Start with finding problems

Instruments, XCTest, Xcode Organizer, MetricKit, and custom telemetry do not compete with one another. They answer different questions.

Question	Best starting point
What startup time do real users experience?	Organizer, MetricKit, and production milestone telemetry
How does the distribution change by app version, device, OS, or user state?	Production telemetry segmented by those dimensions
How long until a requested destination becomes useful?	Custom production milestones
Which launch phase is slow?	Instruments App Launch template
Which application service delays startup?	`OSSignposter` intervals
Did this commit introduce a regression?	XCTest on controlled hardware
Is resume slow rather than launch?	Separate resume metric and lifecycle instrumentation

A common mistake is to use the most convenient tool and then reformulate the problem so that it aligns with the result obtained using that tool.

For example:

a timer in didFinishLaunching is called “total startup time” even though it excludes pre-main work;
viewDidAppear is treated as proof that a frame was displayed and interactive;
one Instruments trace is reported as a benchmark;
an XCTest first-frame metric is used to describe a deep-link destination that loads several seconds later;
a production average hides a large group of slow devices.

The correct order is:

Define the production population and user journey.
Define the start and end boundaries.
Collect and segment the real-user distribution.
Identify the cohort or percentile that regressed.
Reproduce that scenario under controlled conditions.
Profile it and verify the fix in production.

Measurement and profiling are different activities

Measurement answers:

How long does this scenario take under defined conditions?

Profiling answers:

Where is the time being spent, and why?

These activities form a loop, but they should not be confused:

Observe a production regression ↓ Reproduce the affected cohort ↓ Profile a representative run ↓ Form one hypothesis and change the implementation ↓ Measure again in the controlled benchmark ↓ Validate in production

Instruments collects thread states, samples, stack traces, allocations, signposts, and launch phases. That visibility has a cost. Apple explicitly demonstrates that CPU time and wall-clock time in an instrumented trace can differ because profiling itself introduces overhead. It makes Instruments a diagnostic tool rather than the final stopwatch.

Use production data to discover and quantify the problem. Use Instruments to explain its root causes. Use XCTest to make the reproduction scenario repeatable and protect the issue from returning. Then, refer back to production data to verify the result.

Reproduce production findings in a controlled environment

As soon as production data reveals which cohort or user journey is performing slowly, create a benchmark that reproduces the situation. This serves as a tool for diagnosing and preventing regressions, rather than as a primary description of the user experience.

Launch tests are sensitive to the state of the whole device. A useful benchmark removes unrelated variance while preserving the characteristic that made the production cohort slow: device class, data volume, migration state, authentication path, or entry point.

Use a physical device

The simulator is useful for development, lifecycle experiments, and fast feedback. It is not a reliable representation of iPhone launch performance because it runs on Mac hardware, shares host resources, and does not reproduce the same storage, memory, thermal, scheduling, and rendering environment.

Use physical devices for performance conclusions.

The minimum useful device set normally includes:

a reference device used for frequent comparisons;
an older or lower-performance device still important to the user base;
optionally, a current device for detecting behavior specific to newer hardware or OS versions.

Do not compare an “after” result from a new phone with a “before” result from an old one.

Use a production-like build

Measure a Release configuration, or a dedicated Performance configuration derived from Release.

The measured build should preserve:

compiler optimization settings;
production dependency graph;
production resource packaging;
representative feature configuration;
symbolication artifacts for later diagnosis.

Avoid drawing conclusions from a Debug build. Debugger attachment, assertions, sanitizers, logging, and the absence of compiler optimizations can change both the duration and the shape of launch.

Keep observability enabled, but make it lightweight. Signposts are designed for this purpose; verbose synchronous logging is not.

Stabilize device conditions

Before a benchmark session:

Reboot the device if you need a clean starting point.
Let the device settle after boot before measuring.
Disconnect the debugger.
Keep the device on a stable power and thermal profile.
Disable Low Power Mode unless it is the scenario under test.
Avoid simultaneous installations, backups, photo indexing, or other heavy background work.
Record the OS version and available storage.
Use the same screen orientation and appearance settings if they affect the first UI.

Do not cool the device only for the “after” measurement. The goal is not to produce the smallest possible number. The goal is to make before and after comparable.

Control the dataset

Startup often depends more on user state than teams initially expect.

Create explicit fixtures such as:

new unauthenticated user;
returning user with a small local database;
returning user with a large realistic database;
user with a pending migration;
user opening a route that requires authentication restoration;
user with an expired cache;
offline user.

The fixture should be deterministic. A performance test that uses an account whose data changes every day is not deterministic even if the test code never changes.

Remove network variance from the local benchmark

The network is part of the user experience, but it is usually a poor dependency for a code-regression benchmark. Latency, DNS, TLS, server load, routing, and radio state can overwhelm a small application-side change.

For local and CI measurements, prefer one of these contracts:

fixed stubbed responses;
a local deterministic server;
a documented offline launch;
cached data with network refresh deferred beyond the measured endpoint.

Then validate real network behavior separately in production.

This creates two complementary truths:

controlled benchmark: did our code become faster or slower?
field telemetry: what did users experience under real conditions?

Cold launch is a scenario, not a button

There is no public API that reliably clears every relevant system cache and requests a “true cold launch.” Force-quitting an app and immediately reopening it creates a new process, but the second launch can still benefit from warmed pages and system services.

Apple recommends warm launches for repeatable development measurements because they are more consistent. XCTest also performs a throwaway iteration before recording measured iterations, reducing first-run cache variance.

That does not mean cold launch is unimportant. It means it needs a different strategy.

Use warm launches for high-signal regression detection:

the scenario is repeatable;
iteration cost is manageable;
small code changes are easier to compare;
the test works well on fixed CI hardware.

Evaluate cold behavior through a combination of:

controlled launches after device reboot and settling;
first launch after installation or update when relevant;
multiple devices and sessions rather than repeated immediate relaunches;
MetricKit and Organizer distributions from the field.

Never label “terminate and immediately relaunch five times” as five cold launches. It is five new-process launches under increasingly warm system conditions.

Reproduce a system launch regression with XCTest

XCTApplicationLaunchMetric provides the most practical automated baseline for application launch.

A minimal UI performance test looks like this:

import XCTest final class AppLaunchPerformanceTests: XCTestCase { func testReturningUserLaunchToFirstFrame() { let app = XCUIApplication() app.launchArguments = [ "-ui-testing", "-launch-scenario", "returning-user" ] app.launchEnvironment = [ "NETWORK_MODE": "stubbed", "DATASET": "returning-user-small" ] let options = XCTMeasureOptions() options.iterationCount = 10 measure( metrics: [XCTApplicationLaunchMetric()], options: options ) { app.launch() } } }

Without an extended launch task, XCTApplicationLaunchMetric() measures to the first frame. XCTest runs the measured block repeatedly and ignores an additional initial iteration used to reduce warm-up variance.

If the requirement is that the first frame is also responsive to input, use:

XCTApplicationLaunchMetric(waitUntilResponsive: true)

This waits until the main thread is responsive after the first frame. It is often a better default for products that can technically draw a frame while still blocking user interaction.

The two metrics answer different questions:

Metric	Endpoint
`XCTApplicationLaunchMetric()`	First frame, plus any explicitly registered extended launch tasks
`XCTApplicationLaunchMetric(waitUntilResponsive: true)`	First frame and responsive main thread

Do not silently switch between them when comparing builds.

Keep scenario setup outside the measured critical path

Launch arguments and environment variables are useful for choosing deterministic fixtures, but the application must process them carefully.

For example:

struct PerformanceScenario { let networkMode: NetworkMode let dataset: Dataset static func fromProcessInfo() -> PerformanceScenario? { let info = ProcessInfo.processInfo guard info.arguments.contains("-ui-testing") else { return nil } return PerformanceScenario( networkMode: .stubbed, dataset: .returningUserSmall ) } }

Do not replace the normal startup architecture with a completely different test-only app. The test should control external variance, not remove the code you intend to measure.

Create a test matrix, not one universal test

A production application usually needs several launch tests:

func testSignedOutLaunch() { /* fixed signed-out fixture */ } func testReturningUserSmallDatasetLaunch() { /* small fixture */ } func testReturningUserLargeDatasetLaunch() { /* large fixture */ } func testPendingMigrationLaunch() { /* migration fixture */ } func testOfflineLaunch() { /* offline fixture */ }

Entry-point-specific journeys can be added separately, but remember that the built-in application launch metric ends at a system launch milestone. If the product destination becomes useful later, add an application-owned measurement for that interval.

Store a baseline carefully

Xcode performance tests can compare results with an accepted baseline and flag regressions. A baseline is only meaningful for a stable environment.

Do not share one absolute baseline across:

different device models;
different OS versions;
Debug and Release builds;
different fixtures;
simulator and physical devices.

For CI, record the device identity, OS, Xcode version, commit, build configuration, fixture, and raw measurements with every run. If any of those dimensions changes, treat the comparison as a new series until you have evidence that the values remain comparable.

Use Instruments to explain the baseline

Once the benchmark shows that launch is slow, open the same scenario with the App Launch template in Instruments.

Profile a Release build on the same class of physical device used for measurement.

The App Launch template separates major phases such as:

system and dynamic-loader work;
runtime and static initialization;
UIKit initialization;
application initialization;
first-frame creation and rendering;
extended launch work, when defined.

In the trace, inspect more than the total duration:

Is the main thread running, blocked, preempted, or waiting?
Which stack owns the longest synchronous interval?
Is work occurring before main because of linked dependencies or static initialization?
Does the first-frame phase build views that are not visible?
Is the main thread waiting for a worker queue?
Is a high-priority launch task blocked by lower-priority work?
Does disk I/O or database migration sit on the critical path?
Are multiple startup services performing the same work?

Pin the app launch phases, main thread, and relevant worker threads. Add Points of Interest so that application-owned phases appear on the same timeline.

Then form one narrow hypothesis. For example:

The root interface waits synchronously for the complete catalog cache, although the first screen needs only the first page.

Change that behavior, return to the controlled XCTest, and measure again. Do not report the duration of the new Instruments trace as the improvement; report the repeated benchmark under the original contract.

Add signposts for application-owned milestones

System metrics can tell us when the first frame appears. They cannot infer that a product screen contains meaningful data or that a requested route is ready.

Use OSSignposter to make application phases visible:

import OS enum StartupSignposts { static let subsystem = Bundle.main.bundleIdentifier ?? "ExampleApp" static let category = "Startup" static let signposter = OSSignposter( subsystem: subsystem, category: category ) }

A small interval wrapper can keep ownership explicit:

import OS final class StartupInterval { private let name: StaticString private let state: OSSignpostIntervalState private var finished = false init(_ name: StaticString) { self.name = name self.state = StartupSignposts.signposter.beginInterval(name) } func finish() { guard !finished else { return } finished = true StartupSignposts.signposter.endInterval(name, state) } }

Use it around a product-owned phase:

@MainActor final class StartupCoordinator { private var interactiveInterval: StartupInterval? func begin() { interactiveInterval = StartupInterval("LaunchToInteractive") buildRootInterface() } func didBecomeInteractive() { interactiveInterval?.finish() interactiveInterval = nil } }

The end marker must represent a real product condition. For a home screen, that may require:

root hierarchy displayed;
authentication state resolved;
primary content rendered or an explicit recoverable error shown;
primary control enabled;
main thread no longer blocked by a startup gate.

Do not end the interval because a convenient callback fired.

Measure a signpost with XCTest

XCTest can record the duration between matching signpost events:

func testLaunchToInteractive() { let app = XCUIApplication() app.launchArguments = [ "-ui-testing", "-launch-scenario", "returning-user" ] let metric = XCTOSSignpostMetric( subsystem: "com.example.app", category: "Startup", name: "LaunchToInteractive" ) let options = XCTMeasureOptions() options.iterationCount = 10 measure(metrics: [metric], options: options) { app.launch() } }

This creates an application-specific benchmark, but it has an important boundary: the signpost cannot begin before application code runs. It measures the application-owned interval, not the full system launch.

Keep both metrics:

system launch to First Draw or responsiveness;
application phase to Meaningful Content or destination interactive.

One does not replace the other.

Avoid turning signposts into another logging system

Good startup signposts are few, stable, and architectural:

RestoreSession
BuildRootInterface
ResolveInitialRoute
HomeMeaningfulContent
LaunchToInteractive

Avoid adding an interval around every function. A trace with hundreds of unstable names becomes harder to compare and expensive to maintain.

Signpost names also become part of the observability contract. Rename them deliberately.

First Draw and Time to Interactive must remain separate

An app can produce a fast first frame and still feel slow.

Consider this sequence:

0.0 s User taps a product link 1.0 s App draws the home container 1.2 s Authentication restoration finishes 2.3 s Initial route is resolved 3.1 s Product data appears 3.5 s Buy button becomes enabled

The system First Draw metric may report approximately one second. The user asked for a product and could not act on it for three and a half seconds.

Both numbers are correct:

Time to First Draw: approximately 1.0 s;
entry-point-specific Time to Interactive: approximately 3.5 s.

The problem begins when the first number is presented as the complete user experience.

Define a product milestone for each important entry point:

Entry point	Suggested endpoint
App icon	Home screen interactive
Product link	Requested product visible and primary action enabled
Notification	Target state rendered and actionable
Scan quick action	Camera preview active and accepting input
Search shortcut	Search field focused and accepting text

This is also why a universal “startup budget” is insufficient. The system launch budget can be shared, but route execution and destination readiness need their own budgets.

Report distributions, not averages

Production launch performance is a distribution across users, devices, operating systems, application states, and entry points. A single average hides the experiences that most need attention.

For production launch data, report at least:

p50: the median experience;
p90: a slow-but-common experience;
p95: the tail experienced by a meaningful minority when the sample is large enough;
sample count: without it, the percentiles cannot be interpreted;
percentage within the SLO: useful for answering how many launches meet the product target.

Always segment the distribution before interpreting it. At minimum, compare:

application versions and builds;
device performance tiers;
iOS versions;
new-process launch and resume;
authenticated and unauthenticated states when they produce different startup paths;
important entry-point groups for custom product metrics;
migration and first-launch-after-update cohorts when relevant.

A global p90 can remain unchanged while the p90 on the oldest supported devices regresses by several seconds. It can also appear to improve simply because adoption shifted toward newer devices. Segmentation is part of the metric, not an optional dashboard filter.

Do not use production maximum as a release KPI. In a sufficiently large population, isolated extreme values are expected and may reflect interruptions the application cannot control. Use high percentiles, SLO compliance, and sample count to quantify the tail; use diagnostics to investigate pathological individual cases.

For a controlled test, preserve every raw iteration and report the median, p90, and maximum. A small local dataset answers a different question: whether the scenario is stable enough to reproduce a change.

Suppose ten controlled runs produce these durations in seconds:

0.91, 0.94, 0.96, 0.98, 1.00, 1.02, 1.05, 1.09, 1.37, 2.10

The median describes the center, while the p90 and maximum reveal instability that an average would blur. Do not delete the slow runs merely because they are inconvenient. Investigate whether they came from the application or from a documented failure of the experiment.

Use percentiles appropriately

Ten local runs can provide quick feedback, but they do not make a production-quality p99. High percentiles require many samples.

Use:

local repetitions for direction and debugging;
scheduled fixed-device CI runs for regression trends;
production telemetry for robust percentiles across the real device population.

Do not compare a p90 from ten local launches with a p90 from millions of production launches as though they were the same experiment.

Collect system production metrics with Organizer and MetricKit

Production data is the primary description of startup quality. Controlled tests answer whether a code change affected a reproducible scenario, but they cannot represent every real user.

Production launch depends on:

device and OS distribution;
memory pressure and system cache state;
application version and installation history;
local data volume;
authentication and migration state;
thermal and power conditions;
prewarming;
real entry-point mix.

Xcode Organizer aggregates launch performance from users who share diagnostics and allows teams to compare distributions across app versions and devices. It is the fastest way to establish a system-level production baseline before building any custom pipeline.

MetricKit provides programmatic access to daily aggregated performance reports. On currently deployed iOS versions, an application can subscribe with MXMetricManager:

import MetricKit final class MetricsReceiver: NSObject, MXMetricManagerSubscriber { func start() { MXMetricManager.shared.add(self) } func stop() { MXMetricManager.shared.remove(self) } func didReceive(_ payloads: [MXMetricPayload]) { for payload in payloads { persistOrUpload(payload.jsonRepresentation()) } } }

MetricKit reports include histogrammed launch and resume information rather than one application-wide scalar. Depending on the OS and SDK generation, launch data includes time-to-first-draw, optimized time-to-first-draw for prewarmed launches, resume time, and extended launch measurements.

Apple is evolving MetricKit toward the newer MetricManager, MetricReport, and typed MetricResult APIs in upcoming SDKs. If you adopt those APIs, isolate the SDK-specific ingestion layer. The measurement model should remain stable even when the transport type changes:

Apple metric payload ↓ Version-specific decoder ↓ Internal launch histogram model ↓ Device / OS / app-version dimensions ↓ p50 / p90 / p95 dashboards and alerts

Do not upload a daily payload and discard its dimensions. At minimum, preserve:

application version and build;
OS version;
device family when available through the reporting surface;
metric type;
histogram buckets and sample counts;
report time range.

Aggregating everything into one average makes regressions on older devices almost invisible.

Organizer and MetricKit still do not know your entire product journey

System metrics understand launch and rendering. They do not automatically know that the product card requested by a notification is ready for purchase.

For entry-point-specific Meaningful Content and Time to Interactive, add privacy-safe custom telemetry:

struct StartupEvent: Encodable { let launchID: UUID let appVersion: String let deviceClass: String let entryPoint: String let lifecyclePath: String let milestone: String let elapsedMilliseconds: Int }

Avoid including raw URLs, notification payloads, user identifiers, search text, or product data unless the analytics design explicitly requires and permits them. The useful dimensions are normally categories, not content.

Do not let measurement code change the launch being measured

Startup observability runs on the critical path, so its design matters.

Avoid:

synchronous file writes for every milestone;
immediate network upload during launch;
constructing a large analytics dependency graph before First Draw;
serializing complete route payloads;
verbose string interpolation in hot paths;
blocking on metric initialization.

Prefer:

monotonic timestamps such as system uptime;
OSSignposter for timeline intervals;
a small in-memory event buffer;
deferred serialization and upload;
bounded metadata with explicit privacy rules;
one launch identifier connecting milestones from the same journey.

A lightweight custom recorder might look like this:

import Foundation actor StartupMetricsBuffer { struct Mark: Sendable { let name: String let uptime: TimeInterval } private var marks: [Mark] = [] func append(_ name: String) { marks.append( Mark( name: name, uptime: ProcessInfo.processInfo.systemUptime ) ) } func drain() -> [Mark] { defer { marks.removeAll(keepingCapacity: true) } return marks } }

Even this code starts only after application code exists. It must never be described as measuring the full interval from the user tap.

Common measurement mistakes

Starting the timer in didFinishLaunching: his excludes process creation, dyld work, runtime initialization, static initializers, and earlier UIKit work. Name it didFinishLaunchingToInteractive if that is what it measures.
Treating force quit as cache reset: It resets the process, not the whole system state. Immediate relaunches are generally warm new-process launches.
Measuring with the debugger attached: The debugger can materially change timing. Use it for investigation, not for the final baseline.
Using the simulator as the performance device: Simulator results can detect dramatic algorithmic regressions, but they should not become iPhone startup targets.
Measuring one run: One run is an anecdote. Record repeated samples and the conditions around them.
Reporting only the average: The mean hides segmentation and tail behavior. Report percentiles and counts.
Comparing different user states: A signed-out empty database and an authenticated account with years of local data are different scenarios.
Making the network part of every regression test: Real network behavior matters, but uncontrolled networking often masks code-level changes. Stub it locally and validate it separately in production.
Stopping at viewDidAppear: The callback does not guarantee that the frame has reached the screen, content is meaningful, or the main thread is available for the next action.
Optimizing the benchmark instead of the product: If test flags skip migrations, routing, or dependency initialization that real users execute, the benchmark can improve while production does not.

A practical production measurement protocol

The following protocol is a useful starting point for a production team. It begins with user devices and introduces laboratory measurements only after the production metric identifies a problem.

Step 1: Choose the production journeys

Select the launch scenarios that represent meaningful user traffic and risk:

icon launch for a returning user;
first launch after update;
large local dataset;
important Universal Link;
important notification destination;
resume to an existing screen.

Keep launch and resume in separate series.

Step 2: Define system and product boundaries

For each journey, write down:

system start and endpoint;
application-owned start and endpoint;
First Draw milestone;
Meaningful Content milestone;
interactive milestone.

Step 3: Define production dimensions

Before creating test fixtures, define the production segments needed to explain the journey: app version, device tier, OS, authentication state, entry-point category, and migration state. Do not collect dimensions that have no analytical purpose.

Step 4: Instrument production

Use Organizer or MetricKit for system launch and resume distributions. Add lightweight custom milestones for Meaningful Content and Time to Interactive. Buffer and upload the data outside the launch critical path.

Step 5: Establish the baseline

Collect a complete, stable rollout window. Report p50, p90, p95, sample count, SLO compliance, and the important segments. Treat resume as a separate metric.

Step 6: Identify the affected cohort

Determine whether the regression belongs to a device tier, OS, app version, entry point, user state, or migration path. This cohort becomes the specification for the controlled experiment.

Step 7: Reproduce it on physical hardware

Choose stable physical devices based on the affected production population. Create deterministic fixtures that reproduce its data volume, authentication state, feature flags, cache state, and network contract.

Step 8: Measure and profile

Use XCTApplicationLaunchMetric for First Draw or responsiveness and signposts for application-owned stages. Use Instruments to connect the slow interval to threads, stacks, dependencies, and launch phases.

Step 9: Change one thing and protect it

Measure after every meaningful change. Once the improvement is reproducible, run the benchmark regularly on fixed hardware and alert on a sustained regression or budget violation rather than one noisy sample.

Step 10: Validate in the field

Compare Organizer, MetricKit, and custom product milestones by app version and the original affected segment. A local win is complete only when the relevant production distribution moves in the expected direction.

What a useful launch report looks like

A production report should first show the user impact and the affected cohort, then connect it to the controlled experiment used to explain the change:

Production metric: New-process launch to First Draw Versions: 8.23.x → 8.24.x Window: Seven complete days at stable rollout Population: 2.4 million launches All production devices: p50 1.08 s → 1.11 s p90 1.76 s → 1.84 s within 2 s SLO 93.2% → 91.8% Affected cohort: Device tier: lowest supported tier User state: authenticated, large local dataset p90 2.41 s → 3.06 s Samples 184,000 Controlled reproduction: Physical reference device for the affected tier Release build, returning-user-large-v3 fixture 10 measured iterations + 1 XCTest throwaway Local before: p50 1.42 s p90 1.81 s max 2.03 s Local after: p50 1.19 s p90 1.43 s max 1.55 s Change: Deferred catalog cache hydration beyond First Draw Production validation: Compare the same affected cohort in version 8.25.x

The exact values above are illustrative. The important property is traceability: the report begins with the real-user regression, identifies the cohort, shows how it was reproduced, and defines how the fix will be validated.

Avoid reports such as:

Startup is now 16% faster.

Faster for which scenario, device, endpoint, and percentile?

The measurement stack

At this point, the pieces form one system:

Real user devices ├── Organizer → aggregated field trends ├── MetricKit → programmatic system histograms └── Custom telemetry → entry-point-specific milestones Affected production cohort ↓ Controlled physical device ├── XCTest → repeatable reproduction and regression protection ├── OSSignposter → application-owned phases └── Instruments → causal diagnosis

Each layer corrects a blind spot in another:

controlled tests remove variance but cannot represent every user;
production data represents users but contains many uncontrolled dimensions;
system metrics have trustworthy launch boundaries but limited product semantics;
custom metrics understand the product but cannot observe work before application code;
Instruments explains causes but adds profiling overhead.

The goal is not to choose one source of truth. It is to know which source is authoritative for each question.

Conclusion

Correct launch measurement begins with real users.

First define the production population, lifecycle path, entry point, start boundary, and meaningful endpoint. Collect enough samples to report p50, p90, p95, SLO compliance, and the device, OS, app-version, and user-state segments that can explain the distribution.

Use Organizer and MetricKit for system-observed production launch and resume metrics. Add privacy-safe custom telemetry for route resolution, Meaningful Content, and Time to Interactive. When a cohort regresses, reproduce that cohort on physical hardware, use XCTest to make the scenario repeatable, and use Instruments to understand the phases, threads, and stacks behind it. Finally, return to the same production segment to validate the fix.

The most important distinction is this:

A benchmark tells you whether the code changed. Production telemetry tells you whether the user experience changed.

In Part 3: Finding Bottlenecks from dyld to the First Frame, we will use the measurement system built here to read an App Launch trace phase by phase. We will examine pre-main work, linked frameworks, static initialization, disk I/O, main-thread blocking, rendering, and the difference between CPU time and waiting time.