Technology overview

A description of the containers used by the server

The server uses a series of containers for different functionalities. This is a brief overview of these containers and their contribution to the data pipeline/monitoring functions.

Data volume

All containers share access to a data volume mounted on the server's data storage. This allows all containers access to data files and the database to remain persistent outside the container.

Database container

This container is responsible for managing a postreSQL database with PostGIS installed, mounted in the data volume. All containers rely on the database to store and access information relating to all model objects involved in the data pipeline. Furthermore, the database also stores users, permission groups, periodic tasks and even this documentation.

Django webserver

This container runs python 3.10 with Django installed. This container hosts this website and is the main point of access to control other containers and the database. All worker containers use the apps that are part of this Django project.

Celery beat

This container runs python 3.10, hosting a celery beat to control all scheduled asynchronous tasks.

Celery workers general

This container runs pythnon 3.10, hosting a scalable number of celery workers to carry out asynchronous tasks. Number of workers scales depending on the number of jobs in the queue.

Celery worker database

This container hosts a single celery worker and queue. This allows asynchronous tasks where order of execution is important - particularly tasks relating to modifying the database - to be safely carried out. This ensures that racing conditions are avoided.