Containers build CI for distribution

[edoyango]

Following on from yesterday's user group meeting, I got the impression that having a container could be useful for others trying to setup pet on their HPC - especially beginners who may not understand how cuda/pytorch interact with their system. This would also come with instruction on how to execute.

I noticed there was a similar issue #54 about containers for development, but it wasn't clear why it was tagged as not planned and sounds like it served a slightly different purpose anyway.

Would implementing container build CI be useful so others don't have to build their own container?

[tennlee]

I think the main PyEarthTools users are either:

• Using it on an individual workstation (typically a Linux box or laptop)
• Using it at NCI
• Using it at another HPC facility
• Would like to use it on Google Colab specifically, but currently it's not clear how (it's probably not hard, but hasn't been done as the user community need has only been communicated recently)

The people experiencing container-related difficulties are some of those at other HPC facilities. The main issue for me is that each HPC facility is likely to be a bit different, and I don't know how useful an NCI-specific container solution would be. I'm also not a container expert, so maybe it's easier and more portable than I imagine. I think those troubles are coming when system versions of system dependencies (like CUDA library versions) are older. I don't know whether containers help with that.

However, there are some very significant other priorities, such as the test coverage, and verification scorecard generation, to support end-to-end model development, intercomparison, and benchmarking. I think it's okay at the moment to focus on the NCI experience, and leave it to other sysadmins to work out their HPC environment needs.

For development specifically, there is also the issue of providing editable Python installations, which is not so straightforward within a container environment. It's doable, but the developer needs an understanding of their quotas, directory layouts, virtual environment setup and python user base. It's hard to entirely handle those issues for people.

Ultimately, it's a lot of complexity to try to hide all the details. For typical setups, the complexity of the containerised approach isn't really justified in my view. I think it's more efficient to use a modules environment plus a custom PYTHONUSERBASE at NCI; or on an individual workstation use a conda or python-venv virtual environment. Containers may be useful to some people for systems reasons, but for me it's not currently the top priority.

I could be wrong about many aspects of this. However, that's my reasoning on why it hasn't been prioritised as compared to the feature roadmap or other aspects of refactoring / tech debt management which are pressing.