Unit tests

Unit tests for showyourwork! are located in the tests subdirectory of the showyourwork package. Each unit test subclasses the TemporaryShowyourworkRepository class in tests/helpers/temp_repo.py, which defines an asynchronous method called test_repo to test the workflow on a given repository by running a series of steps. Each step is defined by a method within the class, such as create_remote to create the remote repository on GitHub, create_local to create and initialize the repository locally, customize to add/edit files in the repository, build_local to run the workflow locally, and run_github_action to push the changes to the remote and trigger the showyourwork-action run. There are a few other methods as well – see tests/helpers/temp_repo.py for details.

To create a new unit test, subclass TemporaryShowyourworkRepository and override any of the aforementioned methods. As an example, consider the unit test that checks the ability of showyourwork to dynamically generate text in a TeX file:

from helpers import TemporaryShowyourworkRepository
from showyourwork.config import edit_yaml

# A script that computes the age of the universe
variable_script = r"""
import paths
import numpy as np

# Compute the age of the universe
np.random.seed(42)
age = np.random.normal(14.0, 1.0)

# Write it to disk
with open(paths.output / "age_of_universe.txt", "w") as f:
    f.write(f"{age:.3f}")
"""

class TestLatexVariable(TemporaryShowyourworkRepository):
    """Test a workflow with dynamic quantities imported into the tex file."""

    def customize(self):
        """Create and edit all the necessary files for the workflow."""
        # Create the script
        with open(
            self.cwd / "src" / "scripts" / "age_of_universe.py", "w"
        ) as f:
            print(variable_script, file=f)

        # Import the variable into the tex file
        ms = self.cwd / "src" / "tex" / "ms.tex"
        with open(ms, "r") as f:
            ms_orig = f.read()
        with open(ms, "w") as f:
            ms_new = ms_orig.replace(
                r"\end{document}",
                r"Based on a detailed analysis of Planck observations of the cosmic "
                r"microwave background, we have determined the age of the universe "
                r"to be \variable{output/age_of_universe.txt} Gyr."
                "\n"
                r"\end{document}",
            )
            print(ms_new, file=f)

        # Add a Snakemake rule to run the script
        with open(self.cwd / "Snakefile", "r") as f:
            contents = f.read()
        with open(self.cwd / "Snakefile", "w") as f:
            print(contents, file=f)
            print("\n", file=f)
            print(
                "\n".join(
                    [
                        "rule age_of_universe:",
                        "    output:",
                        "        'src/tex/output/age_of_universe.txt'",
                        "    script:",
                        "        'src/scripts/age_of_universe.py'",
                    ]
                ),
                file=f,
            )

This test subclasses TemporaryShowyourworkRepository and overrides a single method: customize, which makes local changes to the repository before attempting to run the workflow. In the code above, we create a Python script (src/scripts/age_of_universe.py), which outputs the age of the universe (purportedly from Planck cosmic microwave background data) to the file src/tex/output/age_of_universe.txt. We then edit the default ms.tex file (which is generated during the create_local step of the unit test) to include a call to \variable{output/age_of_universe.txt}, which imports the value generated by the script into the manuscript. Finally, we add a Snakemake rule telling the workflow how to generate the output from the Python script.

That’s it! The parent class (TemporaryShowyourworkRepository) takes care of the rest.

Note that by default, unit tests create a temporary repository in the github.com/showyourwork organization with the same name as the unit test (but in snake_case instead of camelCase). If the remote build is successful, this repository gets automatically deleted. If it fails, the repository hangs around so you can inspect the build logs. Future runs of the test will delete existing repositories with the same name and re-create them.

All of this, of course, relies on users having write privileges to github.com/showyourwork. If you do not have such privileges but would still like to create and experiment with unit tests for showyourwork, you can either run the tests in debug mode by setting the DEBUG=true environment variable prior to calling pytest (e.g., DEBUG=true pytest) or setting the class variable local_build_only = True at the top of your class.

Warning

For users with write privileges: the current approach will NOT scale well as the number of unit tests N or the frequency of commits increase, since upon every commit we create and delete N GitHub repositories within the organization. There are no hard limits on this number (since the repos are all public), but as N gets large this could make folks at GitHub unhappy. At that point, we should reconsider this approach.