API Testing in Practice: Contract-First Tests with Schemathesis + Pytest (Hands-On)

Pictalingo

·

API Testing in Practice: Contract-First Tests with Schemathesis + Pytest (Hands-On)

API testing often starts as “hit the endpoint and check the status code.” That’s fine for smoke tests, but it won’t catch the subtle bugs that break real clients: missing fields, wrong types, inconsistent error formats, or endpoints that accept invalid input. A practical step up is contract-first testing: you treat your OpenAPI spec as the source of truth and automatically generate tests that validate your API matches the contract.

In this guide, you’ll build a hands-on contract testing setup using Schemathesis (property-based testing for OpenAPI) and pytest. You’ll also add a couple of targeted custom checks (auth, error shape, and “no 500s”), and wire it so juniors/mids can run it locally or in CI.

  • What you’ll get: automated tests that generate many valid/invalid requests from OpenAPI.
  • Why it’s useful: quickly finds mismatches between implementation and spec, including edge cases you didn’t think to write manually.
  • Assumptions: you have an API that exposes OpenAPI (e.g., /openapi.json) or you have an OpenAPI file in your repo.

1) Install dependencies

Create (or use) a Python virtual environment, then install:

python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install pytest schemathesis requests

You’ll run tests with pytest, while schemathesis generates test cases from your OpenAPI schema.

2) Decide how you’ll load your OpenAPI schema

You have two common options:

  • From a running server: your API serves https://api.example.com/openapi.json
  • From a file: you keep openapi.yaml (or openapi.json) in your repo

In practice, teams often use the file in CI (stable) and the live URL locally (fast to try). We’ll show both patterns.

3) Create a Schemathesis test file

Create tests/test_contract.py:

import os import re import requests import schemathesis from schemathesis.checks import not_a_server_error # Option A: load from a live URL (common for local testing) OPENAPI_URL = os.getenv("OPENAPI_URL", "http://localhost:8000/openapi.json") schema = schemathesis.from_uri(OPENAPI_URL) def is_problem_json(response: requests.Response) -> bool: # Optional helper: detect RFC7807-like responses ct = response.headers.get("Content-Type", "") return "application/problem+json" in ct or "application/json" in ct @schema.parametrize() def test_api_contract(case): # Generate a request based on the OpenAPI operation response = case.call() # 1) Basic: validate response against the OpenAPI schema case.validate_response(response) # 2) Strong baseline: never allow 5xx in contract tests # (If your API intentionally returns 5xx in some cases, exclude those endpoints.) not_a_server_error(response)

That’s already useful: it will generate many calls for each endpoint and validate responses match the schema. But real APIs often require auth and have consistent error formats, so let’s improve it.

4) Add authentication in a maintainable way

If your API uses a bearer token, you can inject it into every request. A common approach is environment variables so the same test works locally and in CI.

Update tests/test_contract.py:

import os import requests import schemathesis from schemathesis.checks import not_a_server_error OPENAPI_URL = os.getenv("OPENAPI_URL", "http://localhost:8000/openapi.json") API_TOKEN = os.getenv("API_TOKEN") # e.g. "eyJhbGciOi..." schema = schemathesis.from_uri(OPENAPI_URL) def auth_headers() -> dict: if not API_TOKEN: return {} return {"Authorization": f"Bearer {API_TOKEN}"} @schema.parametrize() def test_api_contract(case): response = case.call(headers=auth_headers()) case.validate_response(response) not_a_server_error(response)

Tip: If some endpoints are public and others require auth, this still works. Public endpoints ignore the header; protected ones get the token.

5) Add custom checks that match how your API behaves

OpenAPI validation is great, but you’ll often want a few extra “team rules.” Here are three practical ones:

  • Error responses should have a predictable shape (helps frontends and mobile clients)
  • Unauthorized requests should return 401/403 (not 200/500)
  • Rate limiting should be visible via 429 and/or headers (optional)

Let’s implement a simple “error shape” check. Suppose your API uses a consistent JSON error structure like:

{ "error": { "code": "VALIDATION_ERROR", "message": "Email is required", "details": {...} } }

Add this to your test file:

def assert_error_shape(response: requests.Response) -> None: # Only enforce for 4xx/5xx JSON responses if response.status_code < 400: return ct = response.headers.get("Content-Type", "") if "application/json" not in ct and "application/problem+json" not in ct: return try: data = response.json() except ValueError: raise AssertionError("Expected JSON error response, got invalid JSON") # Flexible but consistent: must include either 'error' or 'message' if "error" in data: err = data["error"] if not isinstance(err, dict): raise AssertionError("'error' must be an object") if "message" not in err: raise AssertionError("'error.message' is required") elif "message" in data: if not isinstance(data["message"], str): raise AssertionError("'message' must be a string") else: raise AssertionError("Error response must contain 'error' or 'message'")

Now call it inside the test:

@schema.parametrize() def test_api_contract(case): response = case.call(headers=auth_headers()) case.validate_response(response) not_a_server_error(response) assert_error_shape(response)

This is intentionally not too strict. Start broad, then tighten rules once your team agrees on an error standard.

6) Prevent flaky tests with endpoint selection

Property-based tests can hit endpoints that are naturally “stateful” (payments, emails, destructive deletes). If you run those in CI without guardrails, you might create real side effects.

Two practical strategies:

  • Run against a test environment with seed data and no external integrations.
  • Exclude certain operations by tag, path, or HTTP method.

Example: exclude DELETE endpoints and anything under /admin:

def should_skip(case) -> bool: method = case.operation.method.upper() path = case.operation.path if method == "DELETE": return True if path.startswith("/admin"): return True return False @schema.parametrize() def test_api_contract(case): if should_skip(case): return response = case.call(headers=auth_headers()) case.validate_response(response) not_a_server_error(response) assert_error_shape(response)

If your spec includes tags (recommended), you can skip by tag too (e.g., payments).

7) Make it easy to run locally

Create a tiny Makefile (optional but helpful):

test-contract: OPENAPI_URL=http://localhost:8000/openapi.json \ API_TOKEN=dev-token \ pytest -q

Or document a one-liner for your team:

OPENAPI_URL=http://localhost:8000/openapi.json API_TOKEN=dev-token pytest -q

8) Add one “golden” manual test for critical flows

Contract tests are broad; manual tests are deep. A good pattern is:

  • Use Schemathesis for coverage and schema correctness.
  • Use a small number of manual pytest tests for critical user journeys (login, checkout, data export).

Example manual test (simple login flow):

import os import requests BASE_URL = os.getenv("BASE_URL", "http://localhost:8000") def test_login_returns_token(): payload = {"email": "[email protected]", "password": "password123"} r = requests.post(f"{BASE_URL}/auth/login", json=payload, timeout=10) assert r.status_code == 200 data = r.json() assert "access_token" in data assert isinstance(data["access_token"], str) assert data["access_token"]

This complements contract tests nicely: the contract suite ensures the API stays “shape-correct,” and golden tests ensure the main business flow stays “behavior-correct.”

9) Troubleshooting: common issues juniors run into

  • “Schemathesis can’t load my OpenAPI.” Make sure your server is running and OPENAPI_URL is correct. Open the URL in a browser. If it redirects, use the final URL.

  • “My API requires headers (tenant, locale, etc.).” Add them in case.call(headers=...). Keep defaults in env vars so CI can set them too.

  • “It fails on endpoints that send emails / charge cards.” Exclude those endpoints, or point tests at a sandbox environment with mocked integrations.

  • “It generates requests that my API rejects, but OpenAPI says it’s valid.” That usually means the spec is out of date. Contract tests are doing their job—update either the implementation or the spec.

10) What to commit to your repo

  • tests/test_contract.py (the Schemathesis suite)
  • A sample env file (optional) like .env.example documenting OPENAPI_URL, API_TOKEN
  • CI step to run pytest (even if only on main branches at first)

Once this is in place, you’ll catch a surprising number of regressions automatically: response fields that disappear, error formats that drift, and endpoints that “work in Postman” but violate the schema. For junior/mid devs, it’s also a great way to learn OpenAPI by seeing how a spec drives real executable tests.

If you want a next upgrade after this: add hypothesis configuration (max examples, deadlines), run the suite in parallel, and split “safe GET tests” vs “stateful write tests” into separate CI jobs.

Leave a Reply

Your email address will not be published. Required fields are marked *