At some point every mobile team reaches the same conclusion: "We need to test on real phones, and we're not going to buy fifty of them."
So you start looking at real device farms. And pretty quickly they all start to sound the same. Thousands of devices. Fast. Secure. Easy to use. Every landing page promises roughly the same thing.
The differences only show up once you actually start using one. Maybe the phones you need are always busy. Maybe your automation doesn't run the way you expected. Maybe the logs you need to debug aren't there.
So before you sign up for anything, here are the questions I'd ask. They're not about any particular tool. They're about making sure whatever you pick actually fits how your team works.
Choosing a real device farm Lots of phones is a good start. The right phones is what matters.
- Do they have the phones my users actually use?
This is the big one, and it's easy to skip because "thousands of devices" sounds like it must cover you.
But open your analytics first. Look at the top devices and OS versions your users are on. Then check if the device farm has them, and not just the latest flagships. You want budget and mid-range Android phones, phones from the brands your users like, older iPhones you still support, and maybe tablets or foldables if your app runs on them.
A huge catalogue doesn't help much if it's missing the cheap phone half your users carry.
- Are they really real devices?
Some platforms mix real phones with emulators and simulators. That's not necessarily bad, emulators are useful for quick checks, but you should always know which one you're testing on.
Make sure it's clearly labelled. If a bug only happens on real hardware, you don't want to find out later that the "device" you tested on was actually a virtual one.
- How easy is it to get a device when I need one?
Having a phone on the list isn't the same as being able to use it. Popular devices can be busy, especially at peak times.
Ask how availability works. Can you get the device you need without waiting? Can you reserve devices for important test runs? Is there a way to get dedicated devices if your team uses them a lot?
- Does it work with the tests I already have?
You probably already have automation, or plan to. Appium, Espresso, XCUITest, Selenium, Playwright, whatever your team uses.
The best case is that your existing tests run with almost no changes, just pointing them at a different device. If you'd have to rewrite your tests into some special format, that's a big hidden cost. Try running one of your real tests during a trial before you decide.
- Can it plug into my CI/CD pipeline?
Real device testing is most useful when it runs automatically, after merges or before releases, not just when someone remembers.
Check that it works with whatever you use, like GitHub Actions, GitLab CI/CD or Jenkins. And ask how many tests you can run at the same time. Running on ten phones in parallel versus one after another can be the difference between a ten-minute run and a two-hour one.
- What do I get when a test fails?
This one matters more than people expect.
When something breaks on a phone you don't own, you need to see what happened. Look for video recordings, screenshots, device logs and network logs for every session. Without those, a failed test just says "failed" and leaves you guessing.
- Can I test things that aren't public yet?
Most of the time you're testing builds that live on a staging server or even your own laptop, not the live internet.
Ask whether there's a secure way to reach those, often called a local tunnel or local testing. Without it, you can only test what's already public, which isn't much use before a release.
- Can I test real-world stuff like location, language and accessibility?
Real phones are great because they can show real behaviour. So check whether you can do things like change the location for location-based features, switch languages and font sizes, test with VoiceOver and TalkBack, and install your own app builds easily.
These are often exactly the things that break for real users, so you want to be able to check them.
- Is my app and data safe?
You'll be uploading unreleased builds and logging in with test accounts on phones other people also use. So it's fair to ask how they keep things separate.
Ask what happens to a device after your session. Is your app removed and the device wiped? Is the connection encrypted? Can you control who on your team has access? Do they offer private or dedicated devices if you need extra separation?
And on your side, always use test accounts, never real personal ones.
- Does the pricing match how we'll use it?
Pricing models vary a lot. Some charge per user, some per parallel test, some per minute. One isn't automatically better than another, it depends on how your team works.
Think about how many people will use it, how many tests you'll run at once, and how often. Then do the maths for your real usage, not the best-case scenario on the pricing page. And use a free trial or plan if there is one, to test with your own app before paying.
A quick way to compare
When you're trying a few options, it helps to run the same small test on each one. Pick one important flow, like login or checkout. Run it manually on a cheap Android and an iPhone. Run it through your automation from CI. Then break something on purpose and see how easy it is to figure out what went wrong from the video and logs.
That one exercise tells you more than any feature list ever will.
Where AI fits in
Lots of platforms now talk about AI, things like writing tests, fixing broken locators or explaining failures. That's genuinely helpful and worth looking at.
Just remember that AI is only as good as the data it gets. AI on top of real devices, with real logs and videos, is far more useful than AI on top of emulators alone. So the questions above still matter more than whatever AI features are on the box.
Wrapping up
Picking a real device farm isn't really about who has the biggest number on their homepage. It's about whether they have the phones your users own, whether your existing tests run on them, whether you can see what went wrong when something fails, and whether it fits your workflow and budget.
Ask these questions, try it with your own app, and you'll know pretty quickly whether it's a good fit.
If you're looking for somewhere to start, you can try testing on real iPhones and Android phones with TestKase and see how it fits your team.

