Practical API Testing: Catch Breaking Changes with pytest + HTTPX + JSON Schema
APIs rarely “fail loudly.” A small change—renaming a field, returning a different status code, or forgetting a header—can quietly break mobile apps, frontends, and integrations. Good API tests focus on contracts: what the API promises to return for a given request.
In this hands-on guide, you’ll build a compact API test suite in Python using pytest, httpx, and jsonschema. You’ll validate status codes, payload shapes, error responses, and a couple of tricky edge cases (timeouts and idempotency). The examples work against any HTTP API; to keep things concrete, we’ll assume you have an API running locally at http://localhost:8000.
What “good” API tests look like
For junior/mid devs, a practical rule of thumb:
- Test the contract, not the implementation (don’t assert internal IDs or exact timestamps unless required).
- Cover both happy path and failure modes (400/401/403/404/409/422, etc.).
- Validate shape and types (required fields, arrays, enums), not only “is JSON”.
- Keep tests deterministic (avoid flakey reliance on real time, randomness, or shared mutable data).
- Prefer black-box tests: treat the service as a URL, not as a Python module.
Project setup
Create a new folder (or add this to an existing repo) and install dependencies:
python -m venv .venv # macOS/Linux: source .venv/bin/activate # Windows: # .venv\Scripts\activate pip install pytest httpx jsonschemaSuggested structure:
api-tests/ tests/ conftest.py test_health.py test_users_contract.py test_errors.py test_timeouts_and_idempotency.pyCreate a reusable HTTP client (faster and cleaner)
Put this in tests/conftest.py. It centralizes the base URL and makes requests consistent.
import os import pytest import httpx BASE_URL = os.getenv("API_BASE_URL", "http://localhost:8000") @pytest.fixture(scope="session") def client(): # Session-scoped client reuses connections (faster tests). with httpx.Client(base_url=BASE_URL, timeout=5.0) as c: yield cNow you can run against a different environment without changing code:
API_BASE_URL="https://staging.example.com" pytest -qStart with a simple health check
Health endpoints are your “smoke test.” If this fails, don’t bother running the rest.
# tests/test_health.py def test_health(client): r = client.get("/health") assert r.status_code == 200 data = r.json() # Keep this flexible; avoid asserting exact strings unless contract demands it. assert data.get("status") in ("ok", "healthy")If your API doesn’t have /health, use whatever endpoint is guaranteed to be stable (e.g., / or /version).
Validate response contracts with JSON Schema
The fastest way to catch breaking changes is schema validation. You don’t need OpenAPI to do this—just define the expected shapes for key endpoints.
Imagine your API has:
GET /users/{id}→ returns a userPOST /users→ creates a user
Create tests/test_users_contract.py:
from jsonschema import validate USER_SCHEMA = { "type": "object", "required": ["id", "email", "name", "created_at"], "properties": { "id": {"type": "integer"}, "email": {"type": "string", "format": "email"}, "name": {"type": "string"}, "created_at": {"type": "string"}, # ISO string; keep it simple unless you enforce format "is_active": {"type": "boolean"}, }, "additionalProperties": True, # allow new fields without breaking tests } CREATE_USER_RESPONSE_SCHEMA = { "type": "object", "required": ["id"], "properties": {"id": {"type": "integer"}}, "additionalProperties": True, } def test_get_user_contract(client): # Arrange: use a known seed user ID in your test environment. user_id = 1 # Act r = client.get(f"/users/{user_id}") # Assert assert r.status_code == 200 data = r.json() validate(instance=data, schema=USER_SCHEMA) assert data["id"] == user_id def test_create_user_contract(client): payload = {"email": "[email protected]", "name": "Jane Doe"} r = client.post("/users", json=payload) assert r.status_code in (200, 201) data = r.json() validate(instance=data, schema=CREATE_USER_RESPONSE_SCHEMA) # Optional: follow-up read to confirm persistence (black-box style) new_id = data["id"] r2 = client.get(f"/users/{new_id}") assert r2.status_code == 200 validate(instance=r2.json(), schema=USER_SCHEMA)Why additionalProperties: True? It prevents tests from failing when the API adds new fields (a non-breaking change). If you’re doing strict contract enforcement, set it to False, but expect more churn.
Test error responses (this is where most APIs are sloppy)
Clients rely on consistent error formats. If your API returns random shapes, frontends become a mess of special cases.
Let’s enforce a basic error contract: {"error": {"code": "...", "message": "..."}}.
# tests/test_errors.py from jsonschema import validate ERROR_SCHEMA = { "type": "object", "required": ["error"], "properties": { "error": { "type": "object", "required": ["code", "message"], "properties": { "code": {"type": "string"}, "message": {"type": "string"}, "details": {"type": ["object", "array", "string", "null"]}, }, "additionalProperties": True, } }, "additionalProperties": True, } def test_404_has_consistent_shape(client): r = client.get("/this-route-should-not-exist") assert r.status_code == 404 validate(instance=r.json(), schema=ERROR_SCHEMA) def test_validation_error_on_bad_payload(client): # Missing required fields (email/name) -> expect 400/422 depending on framework r = client.post("/users", json={}) assert r.status_code in (400, 422) validate(instance=r.json(), schema=ERROR_SCHEMA)This test suite does something subtle but powerful: it forces your API to be predictable when things go wrong.
Handle timeouts and retries without flakiness
Timeout-related tests can get flaky if they depend on real latency. A safer pattern is to test that your client is configured to time out, and that the API can handle a simulated slow endpoint (if you have one).
If your API includes GET /slow?ms=... for testing, you can do:
# tests/test_timeouts_and_idempotency.py import httpx import pytest def test_timeout_is_enforced(): with httpx.Client(base_url="http://localhost:8000", timeout=0.01) as c: with pytest.raises(httpx.ReadTimeout): c.get("/slow?ms=200")If you don’t have a slow endpoint, skip this test or keep it behind an env flag; don’t create flaky suites that “sometimes fail.”
Test idempotency for safe retries (great for payments, orders, webhooks)
For endpoints that create resources (orders, charges, jobs), production systems often retry requests. Without idempotency, retries can create duplicates. A common pattern is an Idempotency-Key header.
Assume POST /orders accepts an idempotency key and returns the created order:
# tests/test_timeouts_and_idempotency.py (continued) def test_create_order_is_idempotent_with_key(client): headers = {"Idempotency-Key": "test-key-123"} payload = {"item_id": 42, "quantity": 1} r1 = client.post("/orders", json=payload, headers=headers) assert r1.status_code in (200, 201) order1 = r1.json() assert "id" in order1 # Retry the same request with the same key r2 = client.post("/orders", json=payload, headers=headers) assert r2.status_code in (200, 201) order2 = r2.json() # Same resulting resource assert order2["id"] == order1["id"]If your API returns 409 for duplicates instead of returning the same resource, adjust the assertion to match your contract—but pick one behavior and make it consistent.
Make tests maintainable: a few practical tips
- Use seeded data in your test environment (e.g., user ID 1 always exists).
- Keep schemas near tests until they grow; then move them into
tests/schemas/. - Avoid shared state between tests. If you create data, clean it up (or use unique identifiers like emails with a UUID).
- Assert the minimum necessary: status code, required fields, and key invariants.
- Run tests locally the same way as CI:
pytest -qshould be the default.
Run it
pytest -qIf you want a quick “smoke only” run, mark the rest as slow/contract and use markers later. But even without advanced setup, the suite above will catch the most common real-world breakages: accidental field renames, inconsistent error payloads, and duplicate creations on retries.
Next steps (when you’re ready)
- Property-based testing (generate many inputs) with Hypothesis to uncover edge cases.
- Consumer-driven contracts (e.g., Pact) so frontend + backend agree on expectations.
- OpenAPI-driven tests: validate responses against the spec automatically if you have one.
But even if you stop here, you’ve already built something valuable: a practical contract test suite that makes API changes safer and regressions obvious.
Leave a Reply