This system polls AWS AppSync for image files that have been changed, retrieves the image, transforms it to a pyramid tiff, and uploads it to an s3 bucket.
This stack depends on the marble network stack being setup.
At a set interval(daily at 5:30am EDT) CloudWatch will start-up a ECS task to query an AWS AppSync API endpoint for a list of changed files, download the image files, generate pyramid tiffs, and upload those resulting files to the image bucket.
To model and provision AWS resources this repository uses the AWS CDK(typescript). Additional scripts are written in python3. It's useful to follow the setup steps below to run/test these scripts locally in a virtual python environment.
- Setup pyenv - https://github.com/pyenv/pyenv
- Setup aws-cdk - https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html
Use venv - https://docs.python.org/3/library/venv.html
venv is included in the Python standard library and requires no additional installation. Additional details in deployment.
- Review the ECS Dockerfile for python production packages
- Setup virtual env
- python -m venv .env
- source .env/bin/activate
- Local development/testing
- Exit virtual env
- deactivate
Here we aim to take images from various sources and generate pyramid tiffs. To accomplish this we'll launch an ECS Fargate instance running our Dockerfile,and generate the pyramid tiff using the pyvips library. Additional details found here.
The image processor fires off a Fargate task via a Cloudwatch scheduled job. The time that job is ran can be changed by assuming a role with proper privileges (WSE DataAdminRole) and running this AWS CLI command:
aws events put-rule --name "<marble-image-ScheduleRuleId>" --schedule-expression "cron(30 11 * * ? *)"