Integration tests
Integration tests#
Integration tests for showyourwork! are located in the tests/integration
subdirectory of the showyourwork
package. Each test subclasses the
TemporaryShowyourworkRepository
class in tests/integration/helpers/temp_repo.py,
which defines an asynchronous method called test_repo
to test the workflow
on a given repository by running a series of steps. Each step is defined by a
method within the class, such as create_remote
to create the remote
repository on GitHub, create_local
to create and initialize the repository
locally, customize
to add/edit files in the repository, build_local
to
run the workflow locally, and run_github_action
to push the changes to the
remote and trigger the showyourwork-action
run. There are a few other
methods as well – see tests/integration/helpers/temp_repo.py for details.
To create a new test, subclass TemporaryShowyourworkRepository
and override
any of the aforementioned methods. As an example, consider the test that checks
the ability of showyourwork
to dynamically generate text in a TeX file:
from helpers import TemporaryShowyourworkRepository
from showyourwork.config import edit_yaml
# A script that computes the age of the universe
variable_script = r"""
import paths
import numpy as np
# Compute the age of the universe
np.random.seed(42)
age = np.random.normal(14.0, 1.0)
# Write it to disk
with open(paths.output / "age_of_universe.txt", "w") as f:
f.write(f"{age:.3f}")
"""
class TestLatexVariable(TemporaryShowyourworkRepository):
"""Test a workflow with dynamic quantities imported into the tex file."""
def customize(self):
"""Create and edit all the necessary files for the workflow."""
# Create the script
with open(
self.cwd / "src" / "scripts" / "age_of_universe.py", "w"
) as f:
print(variable_script, file=f)
# Import the variable into the tex file
ms = self.cwd / "src" / "tex" / "ms.tex"
with open(ms, "r") as f:
ms_orig = f.read()
with open(ms, "w") as f:
ms_new = ms_orig.replace(
r"\end{document}",
r"Based on a detailed analysis of Planck observations of the cosmic "
r"microwave background, we have determined the age of the universe "
r"to be \variable{output/age_of_universe.txt} Gyr."
"\n"
r"\end{document}",
)
print(ms_new, file=f)
# Add a Snakemake rule to run the script
with open(self.cwd / "Snakefile", "r") as f:
contents = f.read()
with open(self.cwd / "Snakefile", "w") as f:
print(contents, file=f)
print("\n", file=f)
print(
"\n".join(
[
"rule age_of_universe:",
" output:",
" 'src/tex/output/age_of_universe.txt'",
" script:",
" 'src/scripts/age_of_universe.py'",
]
),
file=f,
)
This test subclasses TemporaryShowyourworkRepository
and overrides a single
method: customize
, which makes local changes to the repository before
attempting to run the workflow. In the code above, we create a Python script
(src/scripts/age_of_universe.py), which outputs the age of the universe
(purportedly from Planck cosmic microwave background data)
to the file src/tex/output/age_of_universe.txt.
We then edit the default ms.tex file (which is generated during the create_local
step of the test) to include a call to \variable{output/age_of_universe.txt}
,
which imports the value generated by the script into the manuscript.
Finally, we add a Snakemake rule telling the workflow how to generate
the output from the Python script.
That’s it! The parent class (TemporaryShowyourworkRepository
) takes care
of the rest.
Note that several integration tests are marked with the remote
mark (using pytest.mark
),
which means they require push access to the showyourwork
organization in order
to test the entire workflow, including the showyourwork-action
and the generation
of the PDF when running on the remote.
Each of these tests creates a temporary repository in the
github.com/showyourwork organization with the same name as the test
(but in snake_case
instead of camelCase
); if a repository already exists,
the test will force-push new commits to it, mimicing the behavior of a newly
created repository.
Remote tests are spawned from the remote_integration_tests.yml
workflow in .github/workflows of the showyourwork/showyourwork
repository
on push
events. These do not get run on forks, since they do not have
write access to the showyourwork
organization. This means pull request
tests can only check unit tests and local integration tests. In order to run
remote tests on pull requests, maintainers may label them with the safe to test
label, in which case the remote_integration_tests.yml worfklow is executed with the
pull_request_target
trigger. Maintainers should carefully review the proposed
changes to check for malicious code before marking PRs as safe to test
, since
the workflow will have full write privileges to the organization!