How to Test AI-Generated Mobile Apps on iOS and Android
An agent's web change gets a preview link, but its mobile change needs a build. How to run screen-changing agent pull requests on iOS and Android before merge.
One mobile team we spoke with builds twice a day from its development branch. An open pull request isn’t in that build, so building it needs a trigger of its own, and with the build service already running hot, the team didn’t want that firing on every pull request. Only the pull requests with big changes get a build, and the rest first run on a phone in the next scheduled build, after they merge. Add an agent that opens pull requests all day, and that gap widens.
On the web, a preview link shows the change in a minute. On mobile there is nothing to open until the change is built and running on a simulator. Here is how to get there for the agent pull requests that change a screen.
What you’ll learn
- What makes a mobile change harder to check than a web one
- How to get a simulator and emulator build from the pull requests that change screens
- Four ways to check the running app, with what each one asks of you
- When real devices earn their cost, and when they do not
What Makes Mobile Harder to Check Than Web
Testing an AI-generated mobile app means running the agent’s change as a built app on an iOS simulator and an Android emulator and checking what a user would see, not only whether CI passed. The check needs a build of the change, because a phone has no preview URL to open.
Four things make a mobile pull request harder to judge than a web one.
- No DOM: A native screen has no DOM to read. Tools for mobile app automated testing query the accessibility tree or the view hierarchy instead, and a selector pointing into it can break when an agent rearranges the screen.
- A build to run: The change has to be compiled into an app and installed on a simulator or an emulator before anyone can look at it. Building for iOS needs macOS and Xcode, which the mobile CI guide walks through.
- OS versions: Each iOS and Android version is its own case. An agent writing against the newest SDK won’t see what an older one does with the same screen unless something runs it there.
- Store review: Merging does not reach users. App store review sits between the merge and the phone, so a mistake that ships is slower to take back than a web deploy.
An agent writing a screen has never held the phone. Unless something runs the build, it never sees the screen it made. One OpenAI team building with Codex found that as code throughput increased, human QA capacity became its bottleneck. In a mobile team, that capacity is a person installing a build and tapping through it. Agents multiply the pull requests. Nobody multiplies the people doing the tapping.
Build the App for the Pull Requests That Change Screens
The mobile version of a preview URL is a build the pipeline produces for the pull request and a device layer that can run it. Four steps take an agent’s pull request from a diff to something you can watch.
- Build the app: Produce a zipped simulator
.appfor iOS on a macOS runner and an.apkfor Android on a Linux runner. Apple’s run guide puts code signing and the provisioning profile on the physical-device path, so a simulator build keeps that setup out of the per-pull-request job.- iOS:
xcodebuild -scheme <App> -sdk iphonesimulator -destination "generic/platform=iOS Simulator" -configuration Debug -derivedDataPath build CODE_SIGNING_ALLOWED=NO build, then zipbuild/Build/Products/Debug-iphonesimulator/<App>.app(add-workspace <App>.xcworkspaceif you use CocoaPods or React Native). - Android:
./gradlew assembleDebug, which writes the APK toapp/build/outputs/apk/debug/. - Cost: On GitHub-hosted runners, macOS runners are billed at a higher per-minute rate than Linux runners, per GitHub’s Actions billing docs.
- Expo and React Native: A JavaScript-only change can ship as an over-the-air update preview, but a native change still needs a build.
- iOS:
- Run it: Boot an iOS simulator and an Android emulator and pin the OS versions. Use the oldest version you support and the newest.
- Post the evidence: Attach screenshots and a result to the pull request so the reviewer can judge it without rerunning anything.
- A person merges: The evidence goes to the reviewer, and the merge stays the team’s call.
For an agent’s pull requests, borrow the narrower version from the team in the opening: scope the trigger to changes that touch screens, and let the scheduled build cover the rest after merge. In GitHub Actions, that is a paths filter on the pull_request trigger listing your screen folders, for example ios/App/Views/** and android/app/src/main/res/**. Keep that workflow out of required checks, because a workflow skipped by a paths filter leaves its check pending and blocks the merge.
Simulators and emulators cover the per-pull-request run. The same Apple guide warns that simulators “don’t replicate the performance or features of a physical device,” so run real devices before a release. When a team asks whether Pie runs on simulators or real devices, our answer is simulators, with real devices as a premium option.
Four Ways to Check an Agent’s Mobile Change
The check you pick decides who looks at the running app before it merges, and each of these four fits a different team.
| Option | Who checks the change | Works best when | Watch for |
|---|---|---|---|
| Hand-written suite (Appium, XCUITest, Espresso) | Your engineers | The app is stable and a team maintains the tests | Selectors can break when an agent moves a screen |
| agent-device from Callstack | The coding agent, from its prompt | You want the builder to verify as it works | The author is also the checker |
| Pie | Pie’s agents, separate from the coding agent | You want checks the coding agent did not write | Needs a simulator .app or .apk upload |
| Real devices before release | Your QA team or a device farm | Release validation on hardware | Slower and costlier, so not every pull request |
Callstack’s agent-device deserves a close look. Its README calls it an “open source device layer for AI coding agents,” and it reaches physical devices as well as simulators and emulators. Agents read accessibility snapshots instead of reasoning over screenshots alone. One of its example prompts is this post’s job in a sentence: “Implement the onboarding screen, run it on the iOS simulator and Android emulator, and attach screenshots.” It is MIT licensed, and it fits well if you want the coding agent to show its own evidence as it works.
The trade is who checks. When the agent that wrote the change also runs it, the author grades its own work, which is the pattern scenarios vs tests argues needs a second opinion. The agent’s own run is half the answer, because it shows the agent built what it meant to. The other half is a check that asks whether that works for a user.
How Pie Checks an Agent’s Mobile Build
Pie is the separate check in that table, with tests the coding agent didn’t write. Our agentic testing platform covers native mobile apps (iOS and Android) and web apps, and Fi, the smart pet collar company, uses Pie to test its mobile app.
- The build: For a mobile pull request you upload a zipped simulator
.appfor iOS or an.apkfor Android, and Pie can test your pull requests from there. - The tests: Pie explores your app and generates tests. No scripts to write, no locators to break.
- The findings: Pie groups related findings into issues.
- The fixes: With Auto-Fix, Pie proposes fixes as pull requests in the repositories you connect. Your team decides whether any of them merge.
Run those fixes through the same gate as the agent’s own pull requests, as gating AI-generated pull requests lays out. In a software factory testing setup, this is the station that tests the running product on a phone.
Decide What Counts as Verified on a Phone
An agent can open a pull request for a new screen in minutes. Verified on a phone means that change ran as a build on a simulator and an emulator, and a person saw the evidence before approving.
Put the build step in the pipeline before you argue about tools, and start with the pull requests that change screens. Then hand the build to a checker the agent doesn’t control. Pie explores the running app and generates its own tests, so the evidence on the pull request doesn’t come from the author.
The next time an agent opens a pull request that changes a screen, which phone does it run on before anyone approves?
Test the Agent's Build
Send the agent's build. Know how the screen behaves before you approve.
Frequently Asked Questions
Build each pull request that changes a screen, run it on an iOS simulator and an Android emulator, and check what a user would see.
A passing CI run shows the code compiled and the unit tests passed. It says little about whether the screen works.
A native app is a compiled binary that has to be installed on a simulator, an emulator or a device, so there is nothing to open in a browser.
The mobile equivalent of a preview URL is a build of the pull request.
Not for every pull request. Simulators and emulators cover the per-pull-request run, and hardware earns its cost before a release, for the performance and device features a simulator does not reproduce.
Pie offers real devices as a premium option.
No. Apple puts code signing and the provisioning profile on the physical-device path, so an iOS simulator build keeps that setup out of the per-pull-request job.
Builds for physical devices and TestFlight still need both.
Test the oldest version you support and the newest on each platform, then add the versions your users still run.
An agent building against the latest SDK has no way to see how an older version handles the same screen unless something runs it there.
Yes. Open source tools such as Callstack's agent-device let a coding agent run the app on a simulator, an emulator or a device and attach evidence.
The agent that wrote the change is then also checking it, so add a separate check as well.
Yes. Pie can test your pull requests on native iOS and Android apps as well as web apps.
For mobile it needs a build upload, either a zipped simulator .app for iOS or an .apk for Android, and its agents test the running app from there.
Spent a decade in B2B SaaS sales before building Pie. Now obsessed with helping engineering teams ship without the fear of breaking things. LinkedIn →