Skip to content

5.1.2: API requests fail with RecursionError (HTTP 500) when DB_POOL is enabled #1778

Open

Description

@MrKeiKun

Current Behavior

With netboxcommunity/netbox:v4.7.5-5.1.2 the API starts answering many requests with HTTP 500 after a few hundred requests:

{"error": "maximum recursion depth exceeded", "exception": "RecursionError", "netbox_version": "4.7.5-Docker-5.1.2", "python_version": "3.14.4"}

It does not depend on the endpoint. GET /api/status/ fails on one request and works on the next, and so do plain POST, GET and DELETE calls on many kinds of objects. Once it starts, it keeps happening until the container is restarted. The log shows only the 500 status, no traceback.

I ran 432 API tests in parallel against a fresh container: 306 passed and 126 failed, with 186 RecursionError responses. The first error appears after about 300 tests.

It does not happen with DB_POOL=False, nor with v4.7.5-5.1.1. I could not make a small reproduction. 8 and 32 parallel clients sending about 5,000 and 8,000 create, get and delete requests did not trigger it.

Expected Behavior

No RecursionError. The same 432 tests all pass on v4.7.5-5.1.1, and on v4.7.5-5.1.2 with DB_POOL=False.

The release notes of 5.1.2 list "Use sensible thread count and DB pool (#1775)". In the image, launch-netbox.sh now sets GRANIAN_BLOCKING_THREADS=4, and the configuration turns the psycopg pool on by default (DB_POOL=True, with DB_CONN_MAX_AGE forced to 0). I suspect the pool, since turning it off is enough. I did not test the thread setting alone.

Docker Compose Version

Docker Compose version v5.7.0

Docker Version

Client: 29.9.3 (API 1.56, Go go1.26.10, OS/Arch windows/amd64)
Server: 29.9.3 (API 1.56, OS/Arch linux/amd64)
Docker Desktop, 8 CPUs, 12 GB memory

The git Revision

Not applicable. I don't use a clone of this repository. I run the published image netboxcommunity/netbox:v4.7.5-5.1.2 with my own docker-compose.yml (below).

The git Status

Not applicable. I don't use a clone of this repository. I run the published image
netboxcommunity/netbox:v4.7.5-5.1.2 with my own docker-compose.yml (below).

Startup Command

docker compose -f docker/docker-compose.yml up --build --detach --wait --wait-timeout 600 netbox

NetBox Logs

[2026-11-13 14:18:58 +0000] ::ffff:172.24.0.1 - "GET /api/dcim/devices/70/ HTTP/1.1" 500 100.348
[2026-11-13 14:18:58 +0000] ::ffff:172.24.0.1 - "GET /api/status/ HTTP/1.1" 200 219.892
[2026-11-13 14:18:59 +0000] ::ffff:172.24.0.1 - "DELETE /api/dcim/device-roles/63/ HTTP/1.1" 500 103.050
[2026-11-13 14:18:59 +0000] ::ffff:172.24.0.1 - "GET /api/status/ HTTP/1.1" 200 26.908
[2026-11-13 14:18:59 +0000] ::ffff:172.24.0.1 - "GET /api/status/ HTTP/1.1" 500 100.079
[2026-11-13 14:18:59 +0000] ::ffff:172.24.0.1 - "GET /api/status/ HTTP/1.1" 500 140.516
[2026-11-13 14:18:59 +0000] ::ffff:172.24.0.1 - "GET /api/dcim/locations/ HTTP/1.1" 200 628.538
[2026-11-13 14:18:59 +0000] ::ffff:172.24.0.1 - "GET /api/status/ HTTP/1.1" 200 28.565

Content of docker-compose.override.yml

services:
  postgres:
    image: postgres:18-alpine
    environment:
      - POSTGRES_USER=netbox
      - POSTGRES_PASSWORD=netbox
      - POSTGRES_DB=netbox
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -q -t 2 -d $$POSTGRES_DB -U $$POSTGRES_USER"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 20s
  redis:
    image: docker.io/valkey/valkey:9.0-alpine
    healthcheck:
      test: ["CMD-SHELL", "valkey-cli ping | grep -q PONG"]
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 5s
  netbox:
    image: netboxcommunity/netbox:v4.7.5-5.1.2
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    ports:
      - 8001:8080
    environment:
      - CORS_ORIGIN_ALLOW_ALL=True
      - DB_NAME=netbox
      - DB_USER=netbox
      - DB_PASSWORD=netbox
      - DB_HOST=postgres
      - REDIS_HOST=redis
      - REDIS_DATABASE=0
      - REDIS_SSL=false
      - REDIS_CACHE_HOST=redis
      - REDIS_CACHE_DATABASE=1
      - REDIS_CACHE_SSL=false
      - SECRET_KEY=0123456789abcdefghij0123456789abcdefghij0123456789
      - SKIP_STARTUP_SCRIPTS=false
      - SKIP_SUPERUSER=false
      - SUPERUSER_NAME=admin
      - SUPERUSER_EMAIL=admin@example.com
      - SUPERUSER_PASSWORD=admin
      - SUPERUSER_API_TOKEN=<test token>
      - SUPERUSER_API_KEY=<test key>
      - API_TOKEN_PEPPER_1=<test pepper>
      - METRICS_ENABLED=true
      - GRANIAN_BACKPRESSURE=100
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/login/"]
      interval: 15s
      timeout: 5s
      retries: 20
      start_period: 180s

Activity

  1. rizlas commented on Nov 14, 2026

    @rizlas

    We hit the same issue in production after upgrading to v4.7.5-5.1.2: about 1,000 HTTP 500 responses in two bursts of roughly 15 minutes each, a few hours after the container started. I think I found the cause. It is the combination of DB_POOL=True and METRICS_ENABLED=true.

    Here is troubleshoot from AI assistant:

    Root cause

    With METRICS_ENABLED=true the database engine is
    django_prometheus.db.backends.postgresql (django-prometheus 2.4.0). Its
    DatabaseWrapper.get_new_connection() wraps the cursor factory of the
    connection it gets back:

    def get_new_connection(self, *args, **kwargs):
        conn = super().get_new_connection(*args, **kwargs)
        conn.cursor_factory = ExportingCursorWrapper(
            conn.cursor_factory or Cursor(),
            self.alias,
            self.vendor,
        )
        return conn

    Without a pool, every call returns a brand-new psycopg connection, so the
    cursor factory is wrapped once. With OPTIONS["pool"], Django's
    get_new_connection() returns self.pool.getconn(), which is the same
    psycopg connection object reused across requests. The connection keeps its
    cursor_factory when it goes back to the pool. Each checkout then adds one
    more CursorWrapper subclass on top of the previous one, and each level's
    execute() calls super().execute().

    After about 1,000 checkouts of the same pooled connection, a single query goes
    past the default recursion limit of 1000. Our traceback, from the health check
    plugin's SELECT 1, shows exactly that:

    File ".../django_prometheus/db/common.py", line 70, in execute
    return super().execute(*args, **kwargs)
    File ".../django_prometheus/db/common.py", line 70, in execute
    return super().execute(*args, **kwargs)
    File ".../django_prometheus/db/common.py", line 70, in execute
    return super().execute(*args, **kwargs)
    [Previous line repeated 978 more times]
    ...
    RecursionError: maximum recursion depth exceeded

  2. tobiasge commented on Nov 15, 2026

    @tobiasge
    Member

    Looks like this issue django-commons/django-prometheus#445 which is fixed in django-commons/django-prometheus#505.
    So the right way to fix would be to upgrade the django-prometheus dependeny in Netbox itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions