Python process running in Docker hangs after ~2 hours, container is still running but there is no stdout, stderr or new data saved

  docker, docker-compose, python

I’ve got a docker-compose.yml which runs a Python process inside a Docker container as run using docker-compose up.

The python process is relatively simple, the pseudocode would be along the lines of:

while True:
    logging.info('Start')
    fetch_some_data()  # takes 5 mins to an hour to complete depending on the amount of new data
    # Sleep for 1 hour
    for minute in range(60):
        logging.info(f'sleeing {minute}')
        time.sleep(60)
    logging.info('End')

The issue is that the process tends to hang after a period of time (~2 hours observed). This can happen on inside fetch_some_data() or during time.sleep(60).

For example the docker container is still up and according to the logs it is between subtasks inside fetch_some_data() but nothing is happening: the python process is visible in top inside the container but there is no stdout or stderr and not data received.

I have also set ENV PYTHONUNBUFFERED=1 in the Dockerfile for unbuffered logging however even if logging was somehow buffered data should still be received, and it definitely isn’t.

The host is running:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal

$ docker -v
Docker version 20.10.5, build 55c4c88

$ docker-compose -v
docker-compose version 1.28.6, build 5db8d86f

Source: Python Questions

LEAVE A COMMENT