Using this template#

Welcome to the developer guidelines! This document is split into two parts:

The repository setup. This section is relevant primarily for the repository maintainer and shows how to connect continuous integration services and documents initial set-up of the repository.
The contributor guide. It contains information relevant to all developers who want to make a contribution.

Setting up the repository#

First commit#

If you are reading this, you should have just completed the repository creation with :

cruft create https://github.com/scverse/cookiecutter-scverse

and you should have

cd scib-metrics

into the new project directory. Now that you have created a new repository locally, the first step is to push it to github. To do this, you’d have to create a new repository on github. You can follow the instructions directly on [github quickstart guide][]. Since cruft already populated the local repository of your project with all the necessary files, we suggest to NOT initialize the repository with a README.md file or .gitignore, because you might encounter git conflicts on your first push. If you are familiar with git and knows how to handle git conflicts, you can go ahead with your preferred choice.

Note

If you are looking at this document in the [cookiecutter-scverse-instance][] repository documentation, throughout this document the name of the project is cookiecutter-scverse-instance. Otherwise it should be replaced by your new project name: scib-metrics.

Now that your new project repository has been created on github at https://github.com/adamgayoso/scib-metrics you can push your first commit to github. To do this, simply follow the instructions on your github repository page or a more verbose walkthrough here:

Assuming you are in /your/path/to/scib-metrics. Add all files and commit.

# stage all files of your new repo
git add --all
# commit
git commit -m "first commit"

You’ll notice that the command git commit installed a bunch of packages and triggered their execution: those are pre-commit! To read more about what they are and what they do, you can go to the related section Pre-commit checks in this document.

Note

There is a chance that git commit -m "first commit" fails due to the prettier pre-commit formatting the file .cruft.json. No problem, you have just experienced what pre-commit checks do in action. Just go ahead and re-add the modified file and try to commit again:

 git add -u # update all tracked file
 git commit -m "first commit"

Now that all the files of the newly created project have been committed, go ahead with the remaining steps:

# update the `origin` of your local repo with the remote github link
git remote add origin https://github.com/adamgayoso/scib-metrics.git
# rename the default branch to main
git branch -M main
# push all your files to remote
git push -u origin main

Your project should be now available at https://github.com/adamgayoso/scib-metrics. While the repository at this point can be directly used, there are few remaining steps that needs to be done in order to achieve full functionality.

Coverage tests with Codecov#

Coverage tells what fraction of the code is “covered” by unit tests, thereby encouraging contributors to write tests. To enable coverage checks, head over to codecov and sign in with your GitHub account. You’ll find more information in “getting started” section of the codecov docs.

In the Actions tab of your projects’ github repository, you can see that the workflows are failing due to the Upload coverage step. The error message in the workflow should display something like:

...
    Retrying 5/5 in 2s..
    {'detail': ErrorDetail(string='Could not find a repository, try using repo upload token', code='not_found')}
Error: 404 Client Error: Not Found for url:
...

While codecov docs has a very extensive documentation on how to get started, if you are using the default settings of this template we can assume that you are using codecov in a github action workflow and hence you can make use of the [codecov bot][].

To set it up, simply go to the [codecov app][] page and follow the instructions to activate it for your repository. Once the activation is completed, go back to the Actions tab and re-run the failing workflows.

The workflows should now succeed and you will be able to find the code coverage at this link: https://app.codecov.io/gh/adamgayoso/scib-metrics. You might have to wait couple of minutes and the coverage of this repository should be ~60%.

If your repository is private, you will have to specify an additional token in the repository secrets. In brief, you need to:

Generate a Codecov Token by clicking setup repo in the codecov dashboard.
- If you have already set up codecov in the repository by following the previous steps, you can directly go to the codecov repo webpage.
Go to Settings and copy only the token _______-____-....
Go to Settings of your newly created repository on GitHub.
Go to Security > Secrets > Actions.
Create new repository secret with name CODECOV_TOKEN and paste the token generated by codecov.

Past these additional lines in /.github/workflows.test.yaml under the Upload coverage step:

- name: Upload coverage
  uses: codecov/codecov-action@v3
  with:
      token: ${{ secrets.CODECOV_TOKEN }}

Go back to github Actions page an re-run previously failed jobs.

Documentation on readthedocs#

We recommend using readthedocs.org (RTD) to build and host the documentation for your project. To enable readthedocs, head over to their website and sign in with your GitHub account. On the RTD dashboard choose “Import a Project” and follow the instructions to add your repository.

Make sure to choose the correct name of the default branch. On GitHub, the name of the default branch should be main (it has recently changed from master to main).
We recommend to enable documentation builds for pull requests (PRs). This ensures that a PR doesn’t introduce changes that break the documentation. To do so, got to Admin -> Advanced Settings, check the Build pull requests for this projects option, and click Save. For more information, please refer to the official RTD documentation.
If you find the RTD builds are failing, you can disable the fail_on_warning option in .readthedocs.yaml.

If your project is private, there are ways to enable docs rendering on readthedocs.org but it is more cumbersome and requires a different subscription for read the docs. See a guide here.

Pre-commit checks#

Pre-commit checks are fast programs that check code for errors, inconsistencies and code styles, before the code is committed.

This template uses a number of pre-commit checks. In this section we’ll detail what is used, where they’re defined, and how to modify these checks.

Pre-commit CI#

We recommend setting up pre-commit.ci to enforce consistency checks on every commit and pull-request.

To do so, head over to pre-commit.ci and click “Sign In With GitHub”. Follow the instructions to enable pre-commit.ci for your account or your organization. You may choose to enable the service for an entire organization or on a per-repository basis.

Once authorized, pre-commit.ci should automatically be activated.

Overview of pre-commit hooks used by the template#

The following pre-commit hooks are for code style and format:

black: standard code formatter in Python.
blacken-docs: black on Python code in docs.
prettier: standard code formatter for non-Python files (e.g. YAML).
ruff based checks:
- isort (rule category: I): sort module imports into sections and types.
- pydocstyle (rule category: D): pydocstyle extension of flake8.
- flake8-tidy-imports (rule category: TID): tidy module imports.
- flake8-comprehensions (rule category: C4): write better list/set/dict comprehensions.
- pyupgrade (rule category: UP): upgrade syntax for newer versions of the language.

The following pre-commit hooks are for errors and inconsistencies:

pre-commit-hooks: generic pre-commit hooks for text files.
- detect-private-key: checks for the existence of private keys.
- check-ast: check whether files parse as valid python.
- end-of-file-fixer: check files end in a newline and only a newline.
- mixed-line-ending: checks mixed line ending.
- trailing-whitespace: trims trailing whitespace.
- check-case-conflict: check files that would conflict with case-insensitive file systems.
- forbid-to-commit: Make sure that *.rej files cannot be commited. These files are created by the automated template sync if there’s a merge conflict and need to be addressed manually.
ruff based checks:
- pyflakes (rule category: F): various checks for errors.
- pycodestyle (rule category: E, W): various checks for errors.
- flake8-bugbear (rule category: B): find possible bugs and design issues in program.
- flake8-blind-except (rule category: BLE): checks for blind, catch-all except statements.
- Ruff-specific rules (rule category: RUF):
  - RUF100: remove unneccesary # noqa comments ()

How to add or remove pre-commit checks#

The pre-commit checks check for both correctness and stylistic errors. In some cases it might overshoot and you may have good reasons to ignore certain warnings. This section shows you where these checks are defined, and how to enable/ disable them.

pre-commit#

You can add or remove pre-commit checks by simply deleting relevant lines in the .pre-commit-config.yaml file under the repository root. Some pre-commit checks have additional options that can be specified either in the pyproject.toml (for ruff and black) or tool-specific config files, such as .prettierrc.yml for prettier.

Ruff#

This template configures ruff through the [tool.ruff] entry in the pyproject.toml. For further information ruff configuration, see the docs.

Ruff assigns code to the rules it checks (e.g. E401) and groups them under a rule category (e.g. E). Rule categories are selectively enabled by including them under the select key:

[tool.ruff]
...

select = [
    "F",  # Errors detected by Pyflakes
    "E",  # Error detected by Pycodestyle
    "W",  # Warning detected by Pycodestyle
    "I",  # isort
    ...
]

The ignore entry is used to disable specific rules for the entire project. Add the rule code(s) you want to ignore and don’t forget to add a comment explaining why. You can find a long list of checks that this template disables by default sitting there already.

ignore = [
    ...
    # __magic__ methods are are often self-explanatory, allow missing docstrings
    "D105",
    ...
]

Checks can be ignored per-file (or glob pattern) with [tool.ruff.per-file-ignores].

[tool.ruff.per-file-ignores]
"docs/*" = ["I"]
"tests/*" = ["D"]
"*/__init__.py" = ["F401"]

To ignore a specific rule on a per-case basis, you can add a # noqa: <rule>[, <rule>, …] comment to the offending line. Specify the rule code(s) to ignore, with e.g. # noqa: E731. Check the Ruff guide for reference.

Note

The RUF100 check will remove rule codes that are no longer necessary from noqa comments. If you want to add a code that comes from a tool other than Ruff, add it to Ruff’s external = [...] setting to prevent RUF100 from removing it.

API design#

Scverse ecosystem packages should operate on AnnData and/or MuData data structures and typically use an API as originally introduced by scanpy with the following submodules:

pp for preprocessing
tl for tools (that, compared to pp generate interpretable output, often associated with a corresponding plotting function)
pl for plotting functions

You may add additional submodules as appropriate. While we encourage to follow a scanpy-like API for ecosystem packages, there may also be good reasons to choose a different approach, e.g. using an object-oriented API.

Using VCS-based versioning#

By default, the template uses hard-coded version numbers that are set in pyproject.toml and managed with bump2version. If you prefer to have your project automatically infer version numbers from git tags, it is straightforward to switch to vcs-based versioning using hatch-vcs.

In pyproject.toml add the following changes, and you are good to go!

--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,11 +1,11 @@
 [build-system]
 build-backend = "hatchling.build"
-requires = ["hatchling"]
+requires = ["hatchling", "hatch-vcs"]
 [project]
 name = "scib-metrics"
-version = "0.3.1dev"
+dynamic = ["version"]
@@ -60,6 +60,9 @@
+[tool.hatch.version]
+source = "vcs"
+
 [tool.coverage.run]
 source = ["scib-metrics"]
 omit = [

Don’t forget to update the Making a release section in this document accordingly, after you are done!

Automated template sync#

Automated template sync is enabled by default. This means that every night, a GitHub action runs cruft to check if a new version of the scverse-cookiecutter template got released. If there are any new changes, a pull request proposing these changes is created automatically. This helps keeping the repository up-to-date with the latest coding standards.

It may happen that a template sync results in a merge conflict. If this is the case a *.ref file with the diff is created. You need to manually address these changes and remove the .rej file when you are done. The pull request can only be merged after all *.rej files have been removed.

Tip

The following hints may be useful to work with the template sync:

GitHub automatically disables scheduled actions if there has been not activity to the repository for 60 days. You can re-enable or manually trigger the sync by navigating to Actions -> Sync Template in your GitHub repository.
If you want to ignore certain files from the template update, you can add them to the [tool.cruft] section in the pyproject.toml file in the root of your repository. More details are described in the cruft documentation.
To disable the sync entirely, simply remove the file .github/workflows/sync.yaml.

Making a release#

Updating the version number#

Before making a release, you need to update the version number. Please adhere to Semantic Versioning, in brief

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes,

MINOR version when you add functionality in a backwards compatible manner, and

PATCH version when you make backwards compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format. We use bump2version to automatically update the version number in all places and automatically create a git tag. Run one of the following commands in the root of the repository

bump2version patch
bump2version minor
bump2version major

Once you are done, run

git push --tags

to publish the created tag on GitHub.

Upload on PyPI#

Please follow the Python packaging tutorial.

It is possible to automate this with GitHub actions, see also this feature request in the cookiecutter-scverse template.

Writing documentation#

Please write documentation for your package. This project uses sphinx with the following features:

the myst extension allows to write documentation in markdown/Markedly Structured Text
Numpy-style docstrings (through the napoloen extension).
Jupyter notebooks as tutorials through myst-nb (See Tutorials with myst-nb)
Sphinx autodoc typehints, to automatically reference annotated input and output types

See the scanpy developer docs for more information on how to write documentation.

Tutorials with myst-nb and jupyter notebooks#

The documentation is set-up to render jupyter notebooks stored in the docs/notebooks directory using myst-nb. Currently, only notebooks in .ipynb format are supported that will be included with both their input and output cells. It is your reponsibility to update and re-run the notebook whenever necessary.

If you are interested in automatically running notebooks as part of the continuous integration, please check out this feature request in the cookiecutter-scverse repository.

Hints#

If you refer to objects from other packages, please add an entry to intersphinx_mapping in docs/conf.py. Only if you do so can sphinx automatically create a link to the external documentation.
If building the documentation fails because of a missing link that is outside your control, you can add an entry to the nitpick_ignore list in docs/conf.py

Building the docs locally#

cd docs
make html
open _build/html/index.html

Using this template

Contents

Using this template#

Setting up the repository#

First commit#

Coverage tests with Codecov#

Documentation on readthedocs#

Pre-commit checks#

Pre-commit CI#

Overview of pre-commit hooks used by the template#

How to add or remove pre-commit checks#

pre-commit#

Ruff#

API design#

Using VCS-based versioning#

Automated template sync#

Making a release#

Updating the version number#

Upload on PyPI#

Writing documentation#

Tutorials with myst-nb and jupyter notebooks#

Hints#

Building the docs locally#