API Testing in Practice: Contract Tests with pytest + Schemathesis (OpenAPI) + a Tiny CI Setup

API testing isn’t just “hit the endpoint and hope.” In real projects, regressions often sneak in when a response shape changes, a status code shifts, or a validation rule gets loosened/tightened. The fastest way to catch those issues is to test your API against its contract—an OpenAPI spec—and automatically generate lots of meaningful requests.

In this hands-on guide, you’ll build a practical API testing workflow using:

pytest for readable, maintainable tests
schemathesis for OpenAPI-driven property testing (generated requests + contract validation)
Simple, deterministic checks for auth, error payloads, and idempotency
A minimal CI example so tests run on every push

The approach works whether your backend is FastAPI, Laravel, Node, etc., as long as it exposes an OpenAPI schema (or you can export one).

What You’ll Build

We’ll assume your API exposes an OpenAPI document at something like:

http://localhost:8000/openapi.json (common for FastAPI)
or a checked-in file like openapi.yaml in your repo

You’ll write two kinds of tests:

Contract/property tests generated from OpenAPI (great coverage quickly)
Targeted tests for business rules (auth, error formats, idempotency)

Install Dependencies

Create (or use) a virtual environment, then install:

pip install pytest requests schemathesis hypothesis

Recommended project structure:

. ├── openapi.yaml # or openapi.json (optional if you test via URL) ├── tests │ ├── test_contract.py │ ├── test_auth_and_errors.py │ └── conftest.py └── pytest.ini # optional, for config

Step 1: Point Schemathesis at Your OpenAPI Contract

Schemathesis can load your schema from a local file or a URL. For local dev, a URL is convenient because it matches the running API. For CI, a checked-in schema is often more stable.

Here’s a test that loads from a URL and verifies that responses match the schema (status codes, required fields, types, etc.).

# tests/test_contract.py import os import schemathesis BASE_URL = os.getenv("API_BASE_URL", "http://localhost:8000") SCHEMA_URL = os.getenv("OPENAPI_URL", f"{BASE_URL}/openapi.json") schema = schemathesis.from_uri(SCHEMA_URL) @schema.parametrize() def test_api_contract(case): # Generates a request that matches the OpenAPI contract response = case.call() # Validates response against the schema: # - status code is documented # - response body matches the JSON schema (types/required fields) # - headers documented (where applicable) case.validate_response(response)

Run it:

API_BASE_URL=http://localhost:8000 pytest -q

If your API violates its own OpenAPI spec—wrong status code, missing fields, wrong types—this test will fail with a useful diff.

Step 2: Add Authentication (Bearer Token) to Generated Calls

Most real APIs require auth. You can attach headers to Schemathesis requests using hooks.

# tests/test_contract.py import os import schemathesis BASE_URL = os.getenv("API_BASE_URL", "http://localhost:8000") SCHEMA_URL = os.getenv("OPENAPI_URL", f"{BASE_URL}/openapi.json") API_TOKEN = os.getenv("API_TOKEN", "") schema = schemathesis.from_uri(SCHEMA_URL) @schema.hooks.apply def add_auth_header(context, case): if API_TOKEN: case.headers = case.headers or {} case.headers["Authorization"] = f"Bearer {API_TOKEN}" @schema.parametrize() def test_api_contract(case): response = case.call() case.validate_response(response)

This keeps your contract tests running against protected endpoints without hardcoding secrets. In CI, you can provide API_TOKEN via encrypted environment variables.

Step 3: Filter Out “Dangerous” Operations (Optional but Practical)

Generated tests can create lots of data. In early stages, you may want to focus on GET endpoints first, then expand to writes.

# tests/test_contract.py import os import schemathesis BASE_URL = os.getenv("API_BASE_URL", "http://localhost:8000") SCHEMA_URL = os.getenv("OPENAPI_URL", f"{BASE_URL}/openapi.json") schema = schemathesis.from_uri(SCHEMA_URL) # Only test safe methods at first: SAFE_METHODS = {"GET", "HEAD", "OPTIONS"} @schema.parametrize() def test_api_contract(case): if case.method.upper() not in SAFE_METHODS: return response = case.call() case.validate_response(response)

Once you’re confident, remove the filter or narrow it to exclude only the few endpoints that truly shouldn’t run in shared environments.

Step 4: Add Targeted Tests for Error Shape (Your “API UX”)

Contract tests help ensure the documented schema is respected. But many teams also want a consistent error format so frontend/devs can reliably handle failures.

Example rule: errors always look like:

{"error": {"code": "...", "message": "...", "details": ...}}

Write a targeted test with requests to ensure this stays stable.

# tests/test_auth_and_errors.py import os import requests BASE_URL = os.getenv("API_BASE_URL", "http://localhost:8000") def test_unauthorized_requests_return_consistent_error_shape(): # Choose an endpoint that requires auth (adjust to your API) url = f"{BASE_URL}/api/me" resp = requests.get(url, timeout=10) assert resp.status_code in (401, 403) data = resp.json() assert "error" in data assert "code" in data["error"] assert "message" in data["error"]

Why this matters: if your auth middleware changes, or a framework update alters the default error body, you’ll catch it immediately.

Step 5: Test Idempotency for PUT (or “Retry Safety”)

Clients retry requests. If a PUT endpoint is supposed to be idempotent, two identical requests should yield the same result and not corrupt state.

Below is a generic example. You’ll need to adapt endpoint paths and payloads to your API.

# tests/test_auth_and_errors.py import os import requests BASE_URL = os.getenv("API_BASE_URL", "http://localhost:8000") API_TOKEN = os.getenv("API_TOKEN", "") def auth_headers(): if not API_TOKEN: return {} return {"Authorization": f"Bearer {API_TOKEN}"} def test_put_is_idempotent_for_profile_update(): url = f"{BASE_URL}/api/profile" payload = {"display_name": "Test User", "timezone": "UTC"} # First update r1 = requests.put(url, json=payload, headers=auth_headers(), timeout=10) assert r1.status_code in (200, 204) # Second identical update (should not change outcome) r2 = requests.put(url, json=payload, headers=auth_headers(), timeout=10) assert r2.status_code in (200, 204) # If your API returns the resource, compare bodies if r1.status_code == 200 and r2.status_code == 200: assert r1.json() == r2.json()

If your endpoint returns a timestamp like updated_at, consider comparing only stable fields.

Step 6: Make Contract Tests Faster and Less Flaky

Property tests can generate many cases. You can tune how many to run and set timeouts.

Keep timeouts short (e.g., 5–10 seconds) so failures surface quickly
Start with fewer generated examples, then increase in nightly builds
Run contract tests against a seeded test database or ephemeral environment

For example, you can control Hypothesis settings (Schemathesis uses Hypothesis under the hood):

# tests/conftest.py from hypothesis import settings # Fewer examples for regular CI; raise for nightly builds settings.register_profile("ci", max_examples=25, deadline=None) settings.load_profile("ci")

This keeps your pipeline quick while still providing real value.

Step 7: Minimal CI Example (GitHub Actions)

Here’s a small workflow that:

Starts your API (example shows Docker Compose)
Runs tests against http://localhost:8000

# .github/workflows/api-tests.yml name: API Tests on: push: pull_request: jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: "3.11" - name: Install test deps run: | python -m pip install --upgrade pip pip install pytest requests schemathesis hypothesis - name: Start API (example) run: | docker compose up -d --build # optional: wait for healthcheck / readiness sleep 5 - name: Run tests env: API_BASE_URL: http://localhost:8000 # API_TOKEN: ${{ secrets.API_TOKEN }} run: | pytest -q - name: Shutdown if: always() run: docker compose down -v

If you don’t use Docker Compose, replace the “Start API” step with whatever boots your service.

Common Pitfalls (and Quick Fixes)

Your OpenAPI spec is out of date. Contract tests will “fail loudly” until you regenerate/export the spec or align implementation. Treat this as a feature: it forces docs and behavior to match.
Random data causes flaky failures. Start with safe methods, seed your database, and cap max_examples in CI.
Write endpoints pollute shared environments. Run tests on ephemeral environments (per-branch preview) or test-only databases. If that’s not possible, exclude destructive routes.
Error responses aren’t documented. Add 4xx/5xx schemas to OpenAPI. Your future self (and frontend teammates) will thank you.

A Practical “Starter Checklist”

Expose or export OpenAPI (/openapi.json or openapi.yaml)
Add one Schemathesis contract test to validate documented responses
Add a few targeted tests for auth and error format consistency
Test idempotency for PUT and safe retry semantics where applicable
Run in CI with modest generation counts; increase in nightly builds

With this setup, you’ll catch breaking response changes early, keep your API contract honest, and gain confidence to refactor faster—without writing hundreds of brittle “one request per endpoint” tests.

API Testing in Practice: Contract Tests with pytest + Schemathesis (OpenAPI) + a Tiny CI Setup