Integration tests

Integration tests for showyourwork! are located in the tests/integration subdirectory of the showyourwork package. Each test subclasses the TemporaryShowyourworkRepository class in tests/integration/helpers/temp_repo.py, which defines an asynchronous method called test_repo to test the workflow on a given repository by running a series of steps. Each step is defined by a method within the class, such as create_remote to create the remote repository on GitHub, create_local to create and initialize the repository locally, customize to add/edit files in the repository, build_local to run the workflow locally, and run_github_action to push the changes to the remote and trigger the showyourwork-action run. There are a few other methods as well – see tests/integration/helpers/temp_repo.py for details.

To create a new test, subclass TemporaryShowyourworkRepository and override any of the aforementioned methods. As an example, consider the test that checks the ability of showyourwork to dynamically generate text in a TeX file:

from helpers import TemporaryShowyourworkRepository
from showyourwork.config import edit_yaml

# A script that computes the age of the universe
variable_script = r"""
import paths
import numpy as np

# Compute the age of the universe
np.random.seed(42)
age = np.random.normal(14.0, 1.0)

# Write it to disk
with open(paths.output / "age_of_universe.txt", "w") as f:
    f.write(f"{age:.3f}")
"""

class TestLatexVariable(TemporaryShowyourworkRepository):
    """Test a workflow with dynamic quantities imported into the tex file."""

    def customize(self):
        """Create and edit all the necessary files for the workflow."""
        # Create the script
        with open(
            self.cwd / "src" / "scripts" / "age_of_universe.py", "w"
        ) as f:
            print(variable_script, file=f)

        # Import the variable into the tex file
        ms = self.cwd / "src" / "tex" / "ms.tex"
        with open(ms, "r") as f:
            ms_orig = f.read()
        with open(ms, "w") as f:
            ms_new = ms_orig.replace(
                r"\end{document}",
                r"Based on a detailed analysis of Planck observations of the cosmic "
                r"microwave background, we have determined the age of the universe "
                r"to be \variable{output/age_of_universe.txt} Gyr."
                "\n"
                r"\end{document}",
            )
            print(ms_new, file=f)

        # Add a Snakemake rule to run the script
        with open(self.cwd / "Snakefile", "r") as f:
            contents = f.read()
        with open(self.cwd / "Snakefile", "w") as f:
            print(contents, file=f)
            print("\n", file=f)
            print(
                "\n".join(
                    [
                        "rule age_of_universe:",
                        "    output:",
                        "        'src/tex/output/age_of_universe.txt'",
                        "    script:",
                        "        'src/scripts/age_of_universe.py'",
                    ]
                ),
                file=f,
            )

This test subclasses TemporaryShowyourworkRepository and overrides a single method: customize, which makes local changes to the repository before attempting to run the workflow. In the code above, we create a Python script (src/scripts/age_of_universe.py), which outputs the age of the universe (purportedly from Planck cosmic microwave background data) to the file src/tex/output/age_of_universe.txt. We then edit the default ms.tex file (which is generated during the create_local step of the test) to include a call to \variable{output/age_of_universe.txt}, which imports the value generated by the script into the manuscript. Finally, we add a Snakemake rule telling the workflow how to generate the output from the Python script.

That’s it! The parent class (TemporaryShowyourworkRepository) takes care of the rest.

Note that several integration tests are marked with the remote mark (using pytest.mark), which means they require push access to the showyourwork organization in order to test the entire workflow, including the showyourwork-action and the generation of the PDF when running on the remote. Each of these tests creates a temporary repository in the github.com/showyourwork organization with the same name as the test (but in snake_case instead of camelCase); if a repository already exists, the test will force-push new commits to it, mimicing the behavior of a newly created repository.

Remote tests are spawned from the remote_integration_tests.yml workflow in .github/workflows of the showyourwork/showyourwork repository on push events. These do not get run on forks, since they do not have write access to the showyourwork organization. This means pull request tests can only check unit tests and local integration tests. In order to run remote tests on pull requests, maintainers may label them with the safe to test label, in which case the remote_integration_tests.yml worfklow is executed with the pull_request_target trigger. Maintainers should carefully review the proposed changes to check for malicious code before marking PRs as safe to test, since the workflow will have full write privileges to the organization!