OpenAPI test generators compared: Schemathesis, Portman, Postman import and apitestgen
"Generate tests from my OpenAPI spec" means four different things depending on the tool. Some run thousands of generated requests and never write a file. Some give you a Postman collection. Some give you requests with no assertions at all. And some write test source code you commit. Picking the wrong kind is the usual reason these tools get tried once and dropped.
Full disclosure: I build apitestgen, one of the four. I've tried to describe the others the way their own docs do, and to say plainly where they are the better choice.
The short version
| Tool | What you get | Best when |
|---|---|---|
| Schemathesis | Property-based testing: it reads the schema and runs many generated requests (valid and invalid) against your API at test time. CLI and pytest plugin; no test files are written. | You want to find crashes and schema violations you didn't think of. The strongest option for fuzzing-style coverage. |
| Portman | A Postman collection with contract tests, variation tests and integration tests injected, run through Newman. Configured with JSON/YAML files. | Your team already lives in Postman and you want CI checks generated from the spec. |
| Postman's OpenAPI import | A collection with requests, folders and examples. No assertions: you add the tests yourself. | You want a clickable collection to explore the API, not a test suite. |
| apitestgen | Readable test source files: pytest (with conftest.py), Jest or a Postman collection with tests; Playwright and Schemathesis on Pro. Per endpoint: happy path, missing auth, missing required params, wrong types, response-schema check, plus create→read→update→delete flows. | You want ordinary test files in your repo that a developer can read, edit
and run with plain pytest or npx jest. |
(Dredd used to be the default answer for "check my API against its description", but its repository was archived in November 2024 and is no longer maintained.)
Schemathesis: when you want the machine to look for bugs
Schemathesis doesn't hand you test code. It treats the schema as a description of every input your API accepts and throws generated data at it, including the edge cases nobody writes by hand: odd unicode, boundary numbers, missing fields, extra fields. It reports server errors, responses that don't match the schema and validation gaps, and it can chain requests into multi-step workflows. It runs from a CLI or as a pytest plugin.
The trade-off: a run is dynamic, so there's no fixed file listing "these are our API tests" for a reviewer to read, and the results depend on what the generator tried this time. For finding bugs that's a feature. For a small, explicit regression suite some teams want something they can read line by line.
Portman: when the team already uses Postman
Portman converts an OpenAPI 3.0/3.1 spec into a Postman collection and injects
tests into it: contract tests (status code, content type, JSON schema, headers),
variation tests and integration tests, with optional fuzzing. You run it with Newman,
so it fits a CI pipeline. Behaviour is driven by config files
(portman-config.json and friends), which is powerful once set up and a
bit of reading before the first run. It's open source (Apache-2.0).
The trade-off: the output is Postman test scripts. If your team writes tests in pytest or Jest, it's a second test stack to maintain.
Postman's import: requests, not tests
Importing a spec into Postman gives you a well-organised collection with every endpoint and example bodies. It does not add assertions derived from the schema, so "it ran" is not "it passed". Good for exploring an API; not a test suite until you write the tests.
apitestgen: when you want test files you own
apitestgen reads the spec and writes ordinary test code: a pytest file with a
conftest.py (auth from environment variables, base URL from
API_BASE_URL), a Jest file, or a Postman collection that already has
tests. Each endpoint gets a happy-path test that validates the response body against
the documented schema, plus missing-auth, missing-parameter and wrong-type checks.
When a spec has a collection and item endpoints, it adds a flow that creates a
resource, passes its id through read, update and delete, and checks it is gone. The
generator is deterministic code, no AI model, so the same spec always gives the same
tests.
Where it is weaker, honestly:
- A fixed set of cases. It writes the checks you'd write by hand, not thousands of generated inputs. For bug-hunting, Schemathesis goes much further (and apitestgen Pro can emit a Schemathesis setup for that reason).
- Flows are one resource deep. Create→read→update→delete of a single resource; no arbitrary multi-resource scenarios.
- Generated code is yours to maintain. When the spec changes, you regenerate. Pro includes a GitHub Action that flags spec changes on pull requests and fails when the committed tests are out of date.
- It's a hosted tool. You paste the spec into the site (it isn't stored; generated code is kept in your history only when you're signed in). Free is 5 generations a day; Pro is $15/month.
Two tries without signing in, then five a day free with GitHub.
Or read the output first: 58 pytest tests generated from the Swagger Petstore spec.
Which one should you pick?
- "Find bugs in my API I haven't thought of" → Schemathesis.
- "We use Postman and want CI checks from the spec" → Portman.
- "I just want to click around the endpoints" → Postman import.
- "I want a readable pytest/Jest suite in the repo, today" → apitestgen.
They also combine well: a generated, readable suite as the regression baseline in every pull request, and a property-based run on a schedule to go looking for the weird stuff.