Security
How PocketQA keeps each team's data separate, and exactly what leaves the computers that run your tests. Each section lists the tests that check it, which run before every release. For a security review: what we install on a lab computer.
Signing in
You can sign in with GitHub, Google, Apple, Microsoft, a link sent to your email, or your team's own single sign-on (OpenID Connect). Only email addresses that have been verified count. GitHub sends its verified emails, so an email invitation finds you. Google and Apple verify theirs (Apple's "Hide My Email" addresses included). Microsoft's count for personal accounts, or when a work account's token says so (xms_edov), because a work account's admin can set its email to anything. A team's single sign-on counts only for the domain the team proved it owns with a DNS record. An emailed link proves its address by arriving.
Single sign-on uses PKCE and a nonce (for Apple, a nonce and a client secret signed afresh each time), and the ID token's issuer, audience, nonce, expiry and email are all checked.
An emailed link works once, for 15 minutes, and only in the browser that asked for it, so nobody can sign you into their account with one. It signs you in only when you press the button on its page, so a mail scanner opening it does nothing. One address can ask for five links an hour. A domain can belong to one team only, and never a public email provider's. When a team requires single sign-on, an email at its domain can get in only that way.
The service doesn't keep any provider's token. It keeps a signed session cookie, its signature checked in constant time, that lasts 30 days from the last time you opened the dashboard and never more than 90 days from signing in. It keeps a team's single sign-on secrets, and never shows them back.
Your browser remembers which way you signed in last (GitHub, Google, Apple or Microsoft, not who you are), so a sign-in takes you straight back there. Signing out forgets that, and ends every session you have on the service, not just this browser's.
Checked by 10 tests before every release
- one sign-in page, every way in end to end
- never past 90 days from signing in end to end
- signing out forgets the way in end to end
- and the link works once end to end
- signing out ends that person end to end
- it works only in the browser that asked for it end to end
- an email link asked for from another site is refused end to end
- SSO: a public email provider can end to end
- its client secret signed with the team end to end
- required: a company email can only sign in through SSO end to end
One account, one object
Each account's sites, computers, test queue, results and live progress are kept in a store of their own (a Durable Object, acct:<id>). A separate directory holds only who is on which account, pairing codes and counts across the service. Whatever the dashboard, a report or Claude's tools read comes only from the accounts the signed-in person is on, and each row's account (its tenant column) is checked again before anything is sent.
Checked by 1 test before every release
- and so is a change to someone else's account end to end
People
sign in as a GitHub username or an email address, and each person sees only the accounts they're on. The first time a GitHub username signs in, it's tied to GitHub's own number for that account. If someone gives the name up and someone else takes it, the new holder is refused. If the account is renamed, it keeps its teams under its new name.
Checked by 2 tests before every release
- a GitHub name now held by a different GitHub account is refused end to end
- a renamed GitHub account keeps its teams under its new name end to end
Roles on an account
Each person on an account has one role, and the service checks it on every change, whether it comes from the dashboard or an agent.
A viewer sees everything and changes nothing, tests included. A member adds and changes sites and runs tests. An admin can also manage people and their roles, alerts, privacy, team sign-in and labs. Only the owner (one per account) can delete the account, let support in, or hand the account to someone already on it (staying on as an admin), and nobody else can change who the owner is.
Someone with no role set is a member. An agent acts with the role of the person it acts for.
Checked by 8 tests before every release
- a viewer may change nothing service
- every dashboard change is checked against a role service
- an agent's changes ask for a role too service
- roles: a member (the default) can't change alerts or people end to end
- the owner makes them a viewer: they can look, not test end to end
- an admin manages people, but can't demote the owner or delete the account end to end
- only the owner hands the account to someone else service
- ownership: only the owner hands it over, to someone on it, and stays on as an admin end to end
Changes come from the dashboard
Every change (adding a site or a person, pairing a computer, deleting something) must be a JSON request whose Origin is the service itself, so a form on another website can't make one.
Checked by 2 tests before every release
- a change from somewhere other than the dashboard is refused end to end
- a teammate can't let support in; only the owner end to end
Pairing a computer
The dashboard makes a one-time code: eight characters, good for 30 minutes and one use. After ten wrong guesses in a minute from one address, guessing is cut off.
The installer swaps the code for the computer's own key. On a Mac, the key is kept in the login Keychain (an item named Pocket Lab), not in a file, so it isn't in a copy or backup of the work folder. It reaches the Keychain through security's standard input, never on a command line, and a key file left by an older version is moved there on the lab's next start and then deleted. On other computers, or when the Keychain won't take it, the key is ~/pocketqa-work/lab-key, readable by that user alone. Uninstalling deletes it.
Everything the computer sends (that it's there, taking work, results, live progress) is signed with that key (HMAC-SHA256), and the service checks the signature against the key it holds for that computer. Each message carries its time and a random nonce: a message more than five minutes old is refused, and so is the same message sent twice.
Checked by 15 tests before every release
- and the code doesn't work twice end to end
- guessing pairing codes is cut off after ten a minute end to end
- a computer that isn't this one can't take its work end to end
- a message caught on the way and sent again is refused end to end
- its computer is cut off at once end to end
- a code works once, and the installer says why installer
- the lab's key (issue 523): on a Mac, pairing keeps it in the login Keychain, not in a file lab
- never on a command line lab
- an older lab's key file moves into the Keychain on its next start lab
- elsewhere (Linux) a file only this user can read lab
- its key readable by this user alone installer
- uninstalling takes it off its account lab
- refused for five minutes, it counts as removed and says so once lab
- and then nothing leaves it lab
- its setup check says it was removed lab
Lab request sizes
Pairing and signed lab endpoints count bytes while reading, before parsing JSON or checking signatures: 4 KiB for small messages and progress, 24 KiB for presence, and 4 MiB for results, evidence and pictures. They stop and cancel the unread body when the limit is exceeded, even without an accurate Content-Length header. These limits bound application buffering; they do not replace platform rate limits or upload timeouts.
Removing a computer, or deleting the account, cuts it off at once: its key is deleted and its open connection closed. The computer then says so in its log and its setup check, and asks again only once a day, in case that was a mistake, until it's paired again.
Checked by 4 tests before every release
- stops at the first oversized chunk, without trusting Content-Length service
- limits UTF-8 bytes, not JavaScript characters service
- body limits do not bypass authentication service
- signed text is preserved byte-for-byte after UTF-8 decoding service
What leaves the computer
With each run: which device, passed or failed, page timings, each page's errors and the requests that failed, the last 15 lines of each device's console, and each flow step: its words, whether it passed, and, unless the team switches it off (Settings › Privacy › Send each flow step's screen text), a line of what a tap hit and of what was on the screen after.
Before any of it leaves, addresses lose their query strings, tokens and emails are masked, and what a flow typed is never sent (the step keeps its field).
With the computer's status: its model, its model identifier (which model and year: a Mac's "MacBookPro16,2", a PC's firmware model name; never a serial number), chip, memory and system (macOS), its free disk space, whether it's on its charger and its battery's level (a number), the phones' models and versions, their health (battery, charging, temperature, free memory and storage, hours since restart), what changes how a site looks or runs on them (screen size and refresh rate, text size, display zoom, dark mode, animations, language and region, Low Power Mode, how hot they are, battery health), whether they're on Wi-Fi with its signal and speed, and whether a VPN is on (never the Wi-Fi network's name), its setup checks and the end of its own log; with a run, also the names of what the site is built with (Flutter, React…), never more of the page; no network name, and a phone is told apart by a short id keyed by the lab's own key, never its serial number.
The service also notes which country the computer's network is in, as Cloudflare sees the request (never its address).
No screenshots or page contents: a page that looks different from its last good run is sent as how much of the screen changed and where (top, middle or bottom), worked out on the computer from screenshots that stay there.
Tests started on the computer itself (the local MCP) report the same way, to the team's own dashboard. Tables are capped at 1.8 MB a send; runs older than 90 days are dropped on the computer.
Checked by 21 tests before every release
- nothing of anyone else's: no customers, projects or results installer
- what leaves: addresses without their query strings lab
- no tokens or emails lab
- never what a journey typed lab
- the last 15 console lines at most lab
- screen text only while the team sends it lab
- keyed by the lab lab
- the computer sends no network name lab
- the computer sends its model identifier (which model and year), never its serial number or its name lab
- the computer's battery leaves as its level lab
- the Wi-Fi network\'s name never leaves the computer lab
- no internet: the phone's Wi-Fi network name never leaves the computer lab
- the site\'s stack leaves as plain names only lab
- readings that move every look aren\'t a change worth sending lab
- a page that looks different from its last clean run says how much lab
- a flow step live is what the result sends for it, screen text on lab
- a flow step live is what the result sends for it, screen text off lab
- and with screen text off, no tapped words and no screen text lab
- lab/agent.py tells each run the team's screen-text choice lab
- masked by the result's own rule: no query string, no typed text, no email end to end
- and with the team's screen text switched off, no tapped words and no screen text end to end
Step screenshots, only when a team chooses
A team can switch on Send step screenshots for a site (Sites › the site › Settings › Privacy). Then, for each page tested and after each flow step, the lab sends a small screenshot (a JPEG at most 540 pixels across), which the team's agents can see, and which the run's report link shows (its last few steps on each device; anyone holding the link sees them, as they see the rest of the report, and after 30 days only the team does). The service keeps them 30 days. Switching it off deletes the ones kept straight away, and a lab's screenshots sent while it's off aren't kept. It's off for every new team.
Checked by 7 tests before every release
- a site's own step screenshots: on for that site with the account's off end to end
- switched off for the site, its kept ones are gone at once end to end
- step screenshots are off unless the team switches them on end to end
- a lab's screenshots aren't kept while it's off end to end
- switched off, the ones kept are gone at once end to end
- and the run's report link shows them too end to end
- from its report link too end to end
A site's picture, only when a team chooses
A site's card can show a picture of it from the lab: off for every site until an admin turns on Keep a picture of it in that site's settings. Then, after each test, the lab sends one small JPEG (at most 720 pixels across) of the site's first page; the service keeps only the newest and shows it only to the team. Switching it off deletes it straight away, and one sent while it's off isn't kept. Without it, the card shows the service's own first look (below) or the site's icon.
Checked by 3 tests before every release
- a site's picture isn't kept until the team turns it on for that site end to end
- on, the newest is kept and shown to the team (and nobody else) end to end
- off, the lab's is gone at once end to end
First look, from the service's own browser
When a team adds a site, the service opens it once in Cloudflare's browser (Browser Rendering), at a phone's size and a laptop's, so there's a result before any lab is set up. It opens only a public web address the team added: never an IP address, a local or private name, or anything behind the lab. It keeps the latest look of each site (two small screenshots, shown only to the team) and deletes it with the site; ten looks a day an account.
Checked by 3 tests before every release
- First look opens the site at phone and laptop size end to end
- and stays out of the way for an address that isn end to end
- lookable: public web addresses only service
Safari's load hook
Desktop Safari's driver passes no console on, so the lab loads a small web extension (lab/vendor/loadhook) into each Safari test session (Safari 26 or newer). It runs only the lab's own listeners, the same ones the lab reads every page with after it loads: it listens for script errors and unhandled rejections and keeps the page's console.error and console.warn messages (each call still goes through as before), in window.__lab for the lab to read. It runs only in that test session, only on the test's own site and the lab's control page (their addresses are written into a copy made for that test), and is taken off and its copy deleted when the test ends. It changes nothing on the page (no content, requests, cookies or storage), and its only permission is scripting. If Safari asks before running it, nobody answers: the run says load hook: off and tests as it did before.
Checked by 6 tests before every release
- Safari's load hook (#474): the lab's own listeners, the same as page_js.HOOK's lab
- scoped to this test's own addresses: its site and the lab's control page, nothing else lab
- loaded into this test session only, a copy with those addresses written in lab
- and taken off before the session ends, its copy deleted lab
- a Safari that can't load it says so, nothing else changes lab
- by a Manifest V3 extension with no more than scripting lab
Cookie banners
When a cookie banner covers a flow step's target (or, while exploring, most of the page), the lab closes it as a visitor would, the most private way the banner offers: Reject all, then Necessary only, then a close button, and Accept only when nothing else gets past it. It presses that button with a real tap; the script it runs in the page to find the banner only reads (it scrolls the button into view, as it does a step's target). A banner a flow's own step names (its words are on the banner) is left to the flow. One it can't close is never tapped through: the step fails and says a cookie banner covered it. What it keeps is one step in fixed words ("the lab closed a cookie banner: Reject all"), never the banner's text or which platform drew it, so nothing leaves the computer beyond each flow step's words, as above.
Checked by 8 tests before every release
- cookie detection never confirms an unrelated consent dialog lab
- cookie banners (issue 892): one over a flow step's target is closed with Reject all, never Accept lab
- Accept only when it's the one way past lab
- a banner the step names is left to the step lab
- one it can't close fails the step, naming the banner, and nothing is tapped through it lab
- the step it keeps is fixed words: never the banner's text or platform lab
- the script that finds it only reads lab
- exploring closes one over the page first lab
The iPhone's Web Inspector
iPhone Safari's driver passes no console on either, so a lab with an iPhone (one on its cable, one it has tested, or chosen with --iphone) gets appium-remote-debugger, pinned, in its own work folder (no sudo, nothing global, no install scripts run), and reads Safari's Web Inspector over the cable while a test runs (lab/inspector.mjs, over usbmux with no tunnel, on every iOS version). It attaches only to the page this test's own Safari session opened (found by the address the lab is opening, a one-off marker first), never to another tab on the phone, and writes nothing of any other tab, not even its address. It reads the console, each request and its status, and errors with their stacks. It sends the page nothing but the inspector's own enables and, as the test ends, one read of where the page is (its state and its address without the query); it changes no setting on the iPhone and never pauses the page. What it hears stays on the computer (inspector.json beside the run); what leaves is what leaves for Chrome: the pages' errors, the requests that failed, the console's last lines, and for a test that stopped short, what Safari was doing (its page and state, a request it never finished, its last console line). Addresses lose their query strings before anything is written.
Checked by 8 tests before every release
- setup (issue 873): no iPhone on this lab, nothing installed and npm never asked lab
- pinned, into the lab's own folder, no install scripts, nothing global, no sudo lab
- the evidence sends whether it attached and is proven, and what Safari was doing; never its console or requests lab
- its console and requests stay on this computer, in inspector.json lab
- addresses lose their query strings (no token written) lab
- a test that passed doesn't lab
- only this test's own page (Sam, 2026-11-09): another tab, even in front, is never attached to, and nothing of it is written, not its address lab
- a tab seen beside ours is never chosen, even at the expected address lab
Phones between sites
Desktop Chrome gets a fresh profile every session. Only a test device's Chrome on Android is driven (below): ChromeDriver's own documentation says it starts Chrome there with a clear data directory, and the lab also clears cookies, cache and the site's storage before and after every run (web.py --clear). Either way that's the phone's own Chrome, so a personal phone's never is: a personal phone is tested in Chrome Beta (com.chrome.beta) instead, a separate app with its own data, started and cleared the same way. Storage other sites left (a sign-in provider's, say) isn't cleared yet (a gap we're closing).
Not tested automatically: clearing a phone needs a real phone; checked by hand on the labs.
Phones: a test device or someone's own
Each phone has a role the team sets in its dialog (Labs › the lab › the phone › Status), asked until someone answers; until then it's someone's own (personal).
On a personal phone PocketQA changes no settings: it doesn't keep it awake, lengthen its screen timeout or change its charging, and the end of a run doesn't set any of them back. Its browser is never the owner's.
An iPhone tests in Safari's own automation window, which starts from a clean slate, hides the owner's tabs until it ends and destroys what the test left (Apple's WebDriver).
Chrome on Android can't give a test a separate browser, so a personal Android phone isn't tested in Chrome, by a test or through the Grid. A test runs in Chrome Beta instead, when it's installed beside Chrome: its own app with its own data, cleared for the test, so the owner's Chrome is never started, cleared or sent anything. Only Chrome Beta's version is read (as Chrome's is), the phone is tested only while its owner has it awake and unlocked, and nothing opens on it afterwards.
Someone who uses Chrome Beta as their own browser would lose its data, so the dashboard and the result name the app. Without Chrome Beta it isn't tested, and its result says how to install it. Its screenshots stay in the run's folder on the computer and go with it, and on the service a run's step screenshots go once its results do.
On a test device PocketQA may keep it awake on its charger, set a 30-minute screen timeout, stop it charging at 80% when the team chooses, and clear its browser between tests.
PocketQA never restarts a phone, whichever it is: when Safari on an iPhone stays held by a test session that ending the session and restarting the lab's driver didn't free, the result and lab_doctor say what a person does. Each result says which a phone was. Making a test device personal stops these changes; it doesn't undo settings already made.
Run diagnostics include fixed pipeline statuses, exit codes and exception categories for CSV generation, evidence collection and detail collection. They never include exception text, commands or home-folder paths. They stay with the account's test result, under the same account and support-access controls.
Checked by 22 tests before every release
- a phone nobody has said about is personal, and not set service
- a phone nobody has said about is end to end
- marked a Test device from its dialog end to end
- its lab hears each phone end to end
- phone fixes skip a personal phone: not kept awake, no screen timeout, no 80% charging lab
- is never woken, kept awake or driven in its owner's Chrome lab
- is tested in Chrome Beta (issue 1089) lab
- never touches com.android.chrome's data lab
- its lock never dismissed, never kept awake, nothing opened on it after lab
- without Chrome Beta, its result says to install Chrome Beta first lab
- that's asleep or locked isn't woken lab
- and the end of a run never sets a personal phone lab
- the Grid never drives a personal Android phone lab
- a skipped personal phone says so in its evidence lab
- a test device gets them: awake while charging lab
- held on someone's own phone: WebDriver and a fresh safaridriver tried, the phone never restarted lab
- nor a test device with no screen code, where #890 restarted it (#933) lab
- the lab never restarts an iPhone (#933) lab
- a personal Android phone without Chrome Beta isn service
- one tested in Chrome Beta says which app and version service
- step screenshots go once its results have service
- results: a personal iPhone says a clean browser service
The lab computer itself
Its key is readable by its user alone, and stays so when it's paired again.
Setup never keeps a computer awake always (no sleep 0, disablesleep or masked sleep, which keep a shut or unplugged laptop running, where it can't cool), and undoes an older setup's that did: PocketQA itself holds a computer awake only while it's on its charger and only for testing, or while a test runs, and lets go otherwise.
A Mac the team lets test with its lid shut gets a lid watcher, installed once with the Mac's password as root's own copy: it runs the Mac closed only on its charger, and only while the lab keeps saying so; unplugged, or the lab stopped, it sleeps.
ChromeDriver is downloaded over https only from Google's Chrome for Testing storage, using Chrome for Testing's own list. Download redirects and remote list overrides are refused; versions are checked before they form a path. The driver is used only once it says it's the driver for that Chrome.
The password-free lab_fix safari_sessions repair can ask the person's own Safari to quit when it still holds an old automation session and the Mac has been idle for at least five minutes. If idle time cannot be read, it leaves Safari alone. It asks Safari to quit normally and never force-ends it if a prompt or unsaved work keeps it open; the result asks the person to finish.
A Linux lab (Android phones and Chrome) runs as its user's own systemd service, never as root, through the same signed updater; setup uses sudo only to install packages, undo an older always-awake, join plugdev and keep it running.
Checked by 15 tests before every release
- pairing again leaves the key readable by this user alone lab
- setup never keeps a computer awake always lab
- the lid watcher runs a closed Mac only on its charger lab
- installed once, as root lab
- and undoes an older setup that did lab
- the real list is read over https only, from Chrome for Testing lab
- a download that says it's another version is refused, not used lab
- a Linux lab runs as its user's own systemd service, never as root lab
- and setup uses sudo only to install packages lab
- ChromeDriver downloads only use Google's Chrome for Testing storage (#1091) lab
- ChromeDriver downloads refuse redirects to another host (#1091) lab
- ChromeDriver versions from arguments and the list cannot form paths (#1091) lab
- Safari repair refuses to quit when idle time is unknown (#1217) lab
- Safari repair waits for the person when Safari will not quit (#1217) lab
- Safari quit never force-ends the browser when a prompt keeps it open (#1217) lab
What a computer can ask for
A test's devices are only ever web.sh's own names (checked on the service and again on the computer), and a report's address only ever a report's, so nothing a computer sends becomes a link elsewhere or a word in a shell. Site addresses can't carry a username or password.
Checked by 3 tests before every release
- Test now with devices chosen: only real device names are kept end to end
- a report address that isn end to end
- a site address with a password in it is refused end to end
The lab computer's own network
A lab sits on someone's office or home network, and nobody can point it there: not an agent, a script, a teammate or the service itself.
The service refuses a site or one-off address on a private network (192.168.x, 10.x, .local, a bare name…) from every caller until the team approves it (Settings › Privacy).
The computer decides again for itself, and refuses unless the service said it's approved and it's listed on the computer (~/pocketqa-work/allow-hosts); it checks where a public name really leads, refuses a redirect onto its network, opens only http and https with no password and pages that are paths on the site, fails a flow the moment it leaves its site, and pins desktop Chrome to the address it checked (so a name can't change where it leads mid-test).
The same initial-address and redirect checks cover downloaded flow files and build fingerprints, including both approvals for private addresses.
An expect step answers only true or false. Its own localhost (a dev server) is always allowed.
Checked by 11 tests before every release
- point the lab at its own network end to end
- a site on the lab computer end to end
- lab: an address on its own network, the service saying nothing, is refused lab
- enough on its own lab
- only http and https lab
- pages are paths on the site lab
- a journey that leaves its site fails there lab
- desktop Chrome opens the site at the address the lab checked lab
- expect says true or false lab
- private redirects need both team and local approval lab
- a private initial address is refused before fetching lab
The Grid on a lab computer
The WebDriver hub (lab/hub.py) listens on the computer itself and needs its own token, made at first start and readable only by that user. It refuses anything a web page open on that computer could send it (an Origin header, another Host, a body that isn't JSON), and any capability that names a program, its arguments, an extension, a profile, a browser to attach to or an app to download: a test needs none of them, and each would run code on the computer. Appium’s grouped options are flattened and checked before selecting the device; they cannot override the device the hub selected.
Checked by 7 tests before every release
- hub: a token is made, readable by this user only lab
- nothing without it lab
- a request from a web page (Origin) is refused lab
- as is one by another name (DNS rebinding) lab
- and no capability can start a program lab
- nested Appium options cannot bypass the capability checks in either match set lab
- safe grouped Appium options are normalized before choosing and forwarding a device lab
Settings that protect people are checked
The service refuses to work (503) while one is missing or weak: the session-cookie key (32 random characters at least), its own address (MCP_RESOURCE), the host names it answers on (HOSTNAMES, nothing else), operators pinned to GitHub numbers, and GitHub sign-in. CI checks the same list before every deploy (live/src/settings.ts), and checks the live service after.
Checked by 4 tests before every release
- no cookie key service
- an operator by name alone service
- the service answers only on its own host names end to end
- and the operator's name is pinned by its configured number end to end