You can use nbdev_test_nbs from nbdev to test notebooks. No customization is necessary for docs sites. This is aliased as nbdoc_test for convenience:

!nbdoc_test --help

usage: nbdoc_test [-h] [--fname FNAME] [--flags FLAGS] [--n_workers N_WORKERS]
                  [--verbose VERBOSE] [--timing] [--pause PAUSE]

Test in parallel the notebooks matching `fname`, passing along `flags`

optional arguments:
  -h, --help             show this help message and exit
  --fname FNAME          A notebook name or glob to convert
  --flags FLAGS          Space separated list of flags
  --n_workers N_WORKERS  Number of workers to use
  --verbose VERBOSE      Print errors along the way (default: True)
  --timing               Timing each notebook to see the ones are slow (default:
                         False)
  --pause PAUSE          Pause time (in secs) between notebooks to avoid race
                         conditions (default: 0.5)

To use nbdev_test_nbs, you must also define a settings.ini file at the root of the repo. For documentation based testing, we recommend setting the following variables:

recursive = True
tst_flags = notest

tst_flags = notest allow you to make commments on cells like #notest to allow tests to skip a specific cell. This is useful for skipping long-running tests. You can read more about this here.

recursive = True sets the default behavior of nbdev_test_nbs to True which is probably is what you want for a documentation site with many folders nested arbitrarily deep that may contain notebooks.

Here is this project's settings.ini (note that the recursive flag is set to False as this project is not a documentation site):

!cat ../settings.ini

[DEFAULT]
host = github
lib_name = nbdoc
user = outerbounds
description = Generate beautiful, testable documentation with Jupyter Notebooks
keywords = documentation python jupyter notebook
author = Hamel Husain
author_email = hamel@outerbounds.com
copyright = Outerbounds
branch = master
version = 0.0.6
min_python = 3.6
audience = Developers
language = English
custom_sidebar = False
license = apache2
status = 2
requirements = nbdev fastcore
nbs_path = nbs
doc_path = docs
recursive = False
doc_host = https://outerbounds.github.io
doc_baseurl = /nbdoc/
git_url = https://github.com/outerbounds/nbdoc/tree/master/
lib_path = nbdoc
title = nbdoc
console_scripts = nbdoc_build=nbdoc.convert:nbdoc_build
	nbdoc_test=nbdev.test:nbdev_test_nbs
tst_flags = notest

from nbdev.test import nbdev_test_nbs
nbdev_test_nbs('test_files/example_input.ipynb', n_workers=0)

testing /Users/hamel/github/nbdoc/nbs/test_files/example_input.ipynb
All tests are passing!

nbdev_test_nbs('test_files/', n_workers=0)

testing /Users/hamel/github/nbdoc/nbs/test_files/example_input.ipynb
testing /Users/hamel/github/nbdoc/nbs/test_files/hello_world.ipynb
testing /Users/hamel/github/nbdoc/nbs/test_files/run_flow.ipynb
testing /Users/hamel/github/nbdoc/nbs/test_files/run_flow_showstep.ipynb
testing /Users/hamel/github/nbdoc/nbs/test_files/writefile.ipynb
All tests are passing!

Testing Notebooks