Finding the right mulled biocontainer

A guide to finding mulled biocontaienrs for your desired tool chain
Mulled
Docker
Containers
Bioinformatics
Author

LW Pembleton

Published

March 17, 2024

~Photo by Leeloo The First on pexels~

Finding the right mulled biocontainer

In bioinformatics, containers (such as Docker, Singularity, etc.) are becoming increasingly popular as a means to generate reproducible research workflows (for example, in Nextflow) and to ease implementation on HPCs and cloud infrastructure. Typically, containers include a single tool or package, such as a variant caller or a read mapper, and you will use multiple containers throughout your pipeline for each task. However, there are circumstances where you may want or need multiple tool sets in one container. A great example of this is read alignment with BWA-MEM. Following read mapping, you most likely want to sort the resulting bam output with samtools. Now, while there are well-maintained individual BWA and samtools containers, finding one with both starts to become a bit like the wild west 🤠.

That’s where mulled 🍷 containers come in. These are a way to generate minimal containers with multiple tool sets. You can follow the guide and create your own, but trust me, ‘the droids you’re looking for’ are likely there already 😉 finding them, though, is the tricky bit 🔮

If you look in the hash.tsv file, it is relatively easy to find if the combination you want (under #targets) is there as a mulled container; however, annoyingly, the container ID and TAG are not listed.

To find this info, I have a simple workflow.

The steps

Firstly, clone the multi-package-container repository. We only need it cloned with depth=1 as we are just going to search through files, not use it as a git repo.

git clone --depth=1 https://github.com/BioContainers/multi-package-containers.git

Now, we scan all the mulled combinations and pull out the tool chains and container IDs into a tab-separated file - like your own mini database.

find multi-package-containers/combinations -name "mulled-*" -type f -exec awk -v OFS='\t' '{ print FILENAME, $0 } ' {}  \; | awk '!/#targets/' > mulled-container-info.tsv

After that, you can just use a simple awk command to search the table for the combination of tools you are after. For example, if you want a mulled container with BWA and samtools:

awk '/bwa/ && /samtools/' mulled-container-info.tsv

Which, at the time of writing, returns:

multi-package-containers/combinations/mulled-v2-50007135caa85b10b68ee7aed13ccfdfa626e596:99d99d08adeb47fe605a09be1ade3d043c35237b-0.tsv        bwameth=0.2.0,samtools=1.2
multi-package-containers/combinations/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:03dc1d2818d9de56938078b8b78b82d967c1f820-0.tsv        bwa=0.7.15,samtools=1.3.1
multi-package-containers/combinations/mulled-v2-e5d375990341c5aef3c9aff74f96f66f65375ef6:c5b8c4b7735290369693e2b63cfc1ea0732fde07-0.tsv        samtools=1.13,bwa-mem2=2.2.1    quay.io/bioconda/base-glibc-busybox-bash:latest 0
multi-package-containers/combinations/mulled-v2-7ef549f04aa19ef9cd7d243acfee913d928d9b88:f5ff855ea25c94266e524d08d6668ce6c7824604-0.tsv        bwameth=0.2.0,bwa=0.7.12,samtools=1.2
multi-package-containers/combinations/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:23592e4ad15ca2acfca18facab87a1ce22c49da1-0.tsv        bwa=0.7.17,samtools=1.8
multi-package-containers/combinations/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:1476e745a911a5a2ac22207311b275c51e745ba9-0.tsv        samtools=1.6,bwa=0.7.17
multi-package-containers/combinations/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:adf6dbcd664c38605c6653b8ffcc270d54faceb9-0.tsv        bwa=0.7.17,samtools=1.5

The first column contains the container ID and TAG, and the second column lists the combination of tools. Identify the combination of tools and versions that you want and then head over to the biocontainer repository at quay.io and search using the container ID and select the relevant tag to get the container pull command.

Alternatively, take your ID and TAG and add it to the end of: docker pull quay.io/biocontainers/

For example:

docker pull quay.io/biocontainers/mulled-v2-50007135caa85b10b68ee7aed13ccfdfa626e596:99d99d08adeb47fe605a09be1ade3d043c35237b-0

There you have it, a relatively simple way to find containers with the tool combination you desire. If you are using AWS for your compute, you may be aware that the public ECR gallery now holds all biocontainers, so you can reduce your data transfer fees. However, ‘for now’ mulled containers are not listed. In this situation, you may want to pull the mulled container down from quay.io and push it up to your private ECR to ensure you are keeping your data transfer fees to a minimum. I have a guide of pushing containers to private ECR.

Caveats

It appears not all mulled biocontainers and/or specific versions will show up with this method. I suspect this stems from how the container was made and uploaded. For example, when searching for samtools and bwa the most recent tag from the mulled container mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40 is 23592e4ad15ca2acfca18facab87a1ce22c49da1-0 however tag 219b6c272b25e7e642ae3ff0bf0c5c81a5135ab4-0 is the most recent (at the time of writing).

There is a galaxy project utility (galaxy-tool-util), that has a ‘mulled-search’ command. However, from my current testing, it no longer reports mulled biocontainers from quay.io. I have noticed that you can work around this by specifying singularity as the destination, as the containers share the same names and tags.

Although this will report up-to-date container names and tags, it does not list the individual tool versions, something the previous git repo search method did.

So there appears to be no perfect solution (👋 shoutout if you have suggests), instead there are two paths to choose.

  1. Follow the described git repo search method and maybe settle for an older container, but one you know the tool versions for. ℹ️ Hey, if the latest GATK debacle taught us anything, it’s sometimes older versions of software are better - updates are not always for the best.

  2. Use the mulled-search command to find the latest container version, pull the container down locally, run it in interactive mode and then query 🔍 the versions of the individual tools to see if they suit your needs. For example:

docker pull quay.io/biocontainers/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:219b6c272b25e7e642ae3ff0bf0c5c81a5135ab4-0
docker run -it quay.io/biocontainers/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:219b6c272b25e7e642ae3ff0bf0c5c81a5135ab4-0

samtools --version
# samtools 1.16.1
bwa
# 0.7.17-r1188

Update - 3rd alternative method

Thanks to Moriz E. Beber there is a somewhat simpler method to get the container name if you know the tool versions you require. Presenting 📢 the Multi-Package BioContainers Image Name Generator 🥳 This website essentially generates the hashid (mulled container name) that would exist based on the tools that you define.

If you enter the name of the packages/tools and versions (optional), it will display a hashid and if the container image is known to exist a ✅ will be displayed, if the container does not exist on quay.io it will display a ❌ - its as simple that.

Now technically you don’t need to know the version number of the packages/tools you require, however, the resulting hashid gives you no indication of what version of the tools it contains (unless you go digging via some of the previously discussed methods), which is a risky path to travel, so I would recommend entering the desired version/s.