Extend process_notebooks for testing (#1789)

* Extend process_notebooks for testing

* add command

* spelling and lint

* update docs

* Update contributing.md

* add shebang

* Update contributing.md

* lint

notebook/agentchat_RetrieveChat.ipynb

@@ -3036,7 +3036,8 @@
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": "3.10.12"
-  }
+},
+"test_skip": "Requires interactive usage"
  },
  "nbformat": 4,
  "nbformat_minor": 4

notebook/contributing.md

@@ -74,3 +74,37 @@ Learn more about configuring LLMs for agents [here](/docs/llm_configuration).
 :::
 ````
 ``````
+
+## Testing
+
+Notebooks can be tested by running:
+
+```sh
+python website/process_notebooks.py test
+```
+
+This will automatically scan for all notebooks in the notebook/ and website/ dirs.
+
+To test a specific notebook pass its path:
+
+```sh
+python website/process_notebooks.py test notebook/agentchat_logging.ipynb
+```
+
+Options:
+- `--timeout` - timeout for a single notebook
+- `--exit-on-first-fail` - stop executing further notebooks after the first one fails
+
+### Skip tests
+
+If a notebook needs to be skipped then add to the notebook metadata:
+```json
+{
+    "...": "...",
+    "metadata": {
+        "test_skip": "REASON"
+    }
+}
+```
+
+Note: Notebook metadata can be edited by opening the notebook in a text editor (Or "Open With..." -> "Text Editor" in VSCode)