diff --git a/test_generators/README.md b/test_generators/README.md index 94db105c0..8b62fdf14 100644 --- a/test_generators/README.md +++ b/test_generators/README.md @@ -1,23 +1,22 @@ -# Eth2.0 Test Generators +# Eth 2.0 Test Generators This directory contains all the generators for YAML tests, consumed by Eth 2.0 client implementations. -Any issues with the generators and/or generated tests should be filed - in the repository that hosts the generator outputs, here: [ethereum/eth2.0-tests](https://github.com/ethereum/eth2.0-tests/). +Any issues with the generators and/or generated tests should be filed in the repository that hosts the generator outputs, here: [ethereum/eth2.0-tests](https://github.com/ethereum/eth2.0-tests). -Whenever a release is made, the new tests are automatically built and +Whenever a release is made, the new tests are automatically built, and [eth2TestGenBot](https://github.com/eth2TestGenBot) commits the changes to the test repository. ## How to run generators -pre-requisites: +Prerequisites: - Python 3 installed - PIP 3 - GNU make ### Cleaning -This removes the existing virtual environments (`/test_generators//venv`), and generated tests (`/yaml_tests/`). +This removes the existing virtual environments (`/test_generators//venv`) and generated tests (`/yaml_tests/`). ```bash make clean @@ -25,7 +24,7 @@ make clean ### Running all test generators -This runs all the generators. +This runs all of the generators. ```bash make -j 4 gen_yaml_tests @@ -36,8 +35,7 @@ The `-j N` flag makes the generators run in parallel, with `N` being the amount ### Running a single generator -The make file auto-detects generators in the `test_generators/` directory, - and provides a tests-gen target for each generator, see example. +The makefile auto-detects generators in the `test_generators/` directory and provides a tests-gen target for each generator. See example: ```bash make ./yaml_tests/shuffling/ @@ -45,7 +43,7 @@ make ./yaml_tests/shuffling/ ## Developing a generator -Simply open up the generator (not all at once) of choice in your favorite IDE/editor, and run: +Simply open up the generator (not all at once) of choice in your favorite IDE/editor and run: ```bash # From the root of the generator directory: @@ -65,10 +63,10 @@ eth-utils==1.4.1 ../../test_libs/config_helpers ../../test_libs/pyspec ``` -The config helper and pyspec is optional, but preferred. We encourage generators to derive tests from the spec itself, to prevent code duplication and outdated tests. -Applying configurations to the spec is simple, and enables you to create test suites with different contexts. +The config helper and pyspec is optional, but preferred. We encourage generators to derive tests from the spec itself in order to prevent code duplication and outdated tests. +Applying configurations to the spec is simple and enables you to create test suites with different contexts. -Note: make sure to run `make pyspec` from the root of the specs repository, to build the pyspec requirement. +Note: make sure to run `make pyspec` from the root of the specs repository in order to build the pyspec requirement. Install all the necessary requirements (re-run when you add more): ```bash @@ -77,7 +75,7 @@ pip3 install -e .[pyspec] And write your initial test generator, extending the base generator: -Write a `main.py` file, here's an example: +Write a `main.py` file. See example: ```python from gen_base import gen_runner, gen_suite, gen_typing @@ -134,26 +132,26 @@ if __name__ == "__main__": ``` Recommendations: -- you can have more than just 1 suite creator, e.g. ` gen_runner.run_generator("foo", [bar_test_suite, abc_test_suite, example_test_suite])` -- you can concatenate lists of test cases, if you don't want to split it up in suites, however make sure they could be run with one handler. -- you can split your suite creators into different python files/packages, good for code organization. -- use config "minimal" for performance. But also implement a suite with the default config where necessary. -- you may be able to write your test suite creator in a way where it does not make assumptions on constants. +- You can have more than just one suite creator, e.g. ` gen_runner.run_generator("foo", [bar_test_suite, abc_test_suite, example_test_suite])`. +- You can concatenate lists of test cases if you don't want to split it up in suites, however, make sure they can be run with one handler. +- You can split your suite creators into different python files/packages; this is good for code organization. +- Use config "minimal" for performance, but also implement a suite with the default config where necessary. +- You may be able to write your test suite creator in a way where it does not make assumptions on constants. If so, you can generate test suites with different configurations for the same scenario (see example). -- the test-generator accepts `--output` and `--force` (overwrite output) +- The test-generator accepts `--output` and `--force` (overwrite output). ## How to add a new test generator -In order to add a new test generator that builds `New Tests`: +To add a new test generator that builds `New Tests`: -1. Create a new directory `new_tests`, within the `test_generators` directory. +1. Create a new directory `new_tests` within the `test_generators` directory. Note that `new_tests` is also the name of the directory in which the tests will appear in the tests repository later. 2. Your generator is assumed to have a `requirements.txt` file, with any dependencies it may need. Leave it empty if your generator has none. 3. Your generator is assumed to have a `main.py` file in its root. By adding the base generator to your requirements, you can make a generator really easily. See docs below. 4. Your generator is called with `-o some/file/path/for_testing/can/be_anything -c some/other/path/to_configs/`. - The base generator helps you handle this; you only have to define suite headers, + The base generator helps you handle this; you only have to define suite headers and a list of tests for each suite you generate. 5. Finally, add any linting or testing commands to the [circleci config file](https://github.com/ethereum/eth2.0-test-generators/blob/master/.circleci/config.yml) @@ -168,6 +166,6 @@ Do note that generators should be easy to maintain, lean, and based on the spec. If a test generator is not needed anymore, undo the steps described above and make a new release: -1. remove the generator directory -2. remove the generated tests in the `eth2.0-tests` repository by opening a PR there. -3. make a new release +1. Remove the generator directory. +2. Remove the generated tests in the [`eth2.0-tests`](https://github.com/ethereum/eth2.0-tests) repository by opening a PR there. +3. Make a new release.