diff --git a/README.md b/README.md index f7b4f2d..448bec4 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,171 @@ # gossipsub-hardening-tests -This repo contains [testground](https://github.com/testground/testground) test plans designed to evaluate the performance of gossipsub under various attack scenarios. +This repo contains [testground](https://github.com/testground/testground) test plans designed to evaluate the +performance of gossipsub under various attack scenarios. -## Usage +## Installation & Setup -TODO: +We're using python to generate testground composition files, and we shell out to a few +external commands, so there's some environment setup to do. -- [ ] python requirements -- [ ] testground requirements (daemon must be running, etc) -- [ ] how to run using cli -- [ ] how to run using runner notebook -- [ ] how to run the analysis notebook +### Requirements +#### Testground +You'll need to have the [testground](https://github.com/testground/testground) binary built and accessible +on your `$PATH`. As of the time of writing, we're running from `master`, however releases after v0.5.0 should +be compatible. + +After running `testground` for the first time, you should have a `~/testground` directory. You can change this +to another location by setting the `TESTGROUND_HOME` environment variable. + +#### Cloning this repo + +The testground client will look for test plans in `$TESTGROUND_HOME/plans`, so this repo should be cloned or +symlinked into there: + +```shell +cd ~/testground/plans # if this dir doesn't exist, run testground once first to create it +git clone git@github.com:protocol/gossipsub-hardening.git +``` + +#### Python + +We need python 3.7 or later, ideally in a virtual environment. If you have python3 installed, you can create +a virtual environment named `venv` in this repo and it will be ignored by git: + +```shell +python3 -m venv venv +``` + +After creating the virtual environment, you need to "activate" it for each shell session: + +```shell +# bash / zsh: +source ./venv/bin/activate + +# fish: +source ./venv/bin/activate.fish +``` + +You'll also need to install the python packages used by the scripts: + +```shell +pip install -r scripts/requirements.txt +``` + +#### External binaries + +The run scripts rely on a few commands being present on the `PATH`: + +- the `testground` binary +- `go` +- `gnuplot` (optional, you'll just get an error message and no latency svg file) + +## Running Tests + +### Running using the Runner Jupyter notebook + +With the python virtualenv active, run + +```shell +jupyter notebook +``` + +This will start a Jupyter notebook server and open a browser to the Jupyter file navigator. +In the Jupyter UI, navigate to the `scripts` dir and open `Runner.ipynb`. + +This will open the runner notebook, which lets you configure the test parameters using a +configuration UI. + +You'll need to run all the cells to prepare the notebook UI using `Cell menu > Run All`. You can reset +the notebook state using the `Kernel Menu > Restart and Run All` command. + +You can save configuration snapshots to JSON files and load them again using the buttons at the bottom +of the configuration panel. + +The cell at the bottom of the notebook has a "Run Test" button that will convert the configured parameters +to a composition file and start running the test. It will shell out to the `testground` client binary, +so if you get an error about a missing executable, make sure `testground` is on your `PATH` and restart +the Jupyter server. + +At the end of a successful test, there will be a new `output/pubsub-test-$timestamp` directory (relative to +the `scripts` dir) containing the composition file, the full `test-output.tgz` file collected from testground, +and an `analysis` directory. + +The `analysis` directory has relevant files that were extracted from the `test-output.tgz` archive, along with a +new Jupyter notebook, `Analysis.ipynb`. See below for more details about the analysis notebook. + +If the test fails (`testground` returns a non-zero exit code), the runner script will move the `pubsub-test-$timestamp` +dir to `./output/failed`. + +The "Test Execution" section of the config UI will let you override the output path, for example if you want +to give your test a meaningful name. + +### Running using the cli scripts + +Inside the `scripts` directory, the `run.py` script will generate a composition and run it by shelling out to +`testground`. If you just want it to generate the composition, you can skip the test run by passing the `--dry-run` +flag. + +You can get the full usage by running `./run.py --help`. + +To run a test with baseline parameters (as defined in `scripts/templates/baseline/params/_base.toml`), run: + +```shell +./run.py +``` + +By default, this will create a directory called `./output/pubsub-test-$timestamp`, which will have a `composition.toml` +file inside, as well as a `template-params.toml` that contains the params used to generate the composition. + +You can control the output location with the `-o` and `--name` flags, for example: + +```shell +./run.py -o /tmp --name 'foo' +# creates directory at /tmp/pubsub-test-$timestamp-foo +``` + +You can override individual template parameters using the `-D` flag, for example, `./run.py -D T_RUN=5m`. +There's no exhaustive list of template parameters, so check the template at `scripts/templates/baseline/template.toml.j2` +to see what's defined. + +By default, the `run.py` script will extract the test data from the collected test output archive and copy the +analysis notebook to the `analysis` subdirectory of the test output dir. If you want to skip this step, +you can pass the `--skip-analysis` flag. + +## Analyzing Test Outputs + +After running a test, there should be a directory full of test ouputs, with an `analysis` dir containing +an `Analysis.ipynb` Jupyter notebook. If you're not already running the Jupyter server, start it with +`jupyter notebook`, and use the Jupyter UI to navigate to the analysis notebook and open it. + +Running all the cells in the analysis notebook will convert the extracted test data to +[pandas](https://pandas.pydata.org/) `DataFrame`s. This conversion takes a minute or two depending on the +size of the test and your hardware, but the results are cached to disk, so future runs should be pretty fast. + +Once everything is loaded, you'll see some charts and tables, and there will be a new `figures` directory inside the +`analysis` dir containing the charts in a few image formats. There's also a `figures.zip` with the same contents +for easier downloading / storage. + +### Running the analysis notebook from the command line + +If you just want to generate the charts and don't care about interacting with the notebook, you can execute +the analysis notebook using a cli script. + +Change to the `scripts` directory, then run + +```shell +./analyze.py run_notebook ./output/pubsub-test-$timestamp +``` + +This will copy the latest analysis notebook template into the `analysis` directory and execute the notebook, which +will generate the chart images. + +This command is useful if you've made changes to the analysis notebook template and want to re-run it against a +bunch of existing test outputs. In that case, you can pass multiple paths to the `run_notebook` subcommand: + + ```shell + ./analyze.py run_notebook ./output/pubsub-test-* +# will run the latest notebook against everything in `./output + ``` diff --git a/scripts/analyze.py b/scripts/analyze.py index e9126ba..f436eb5 100755 --- a/scripts/analyze.py +++ b/scripts/analyze.py @@ -269,8 +269,7 @@ def run_tracestat(tracer_output_dir): def extract_test_outputs(test_output_zip_path, output_dir=None, convert_to_pandas=False, prep_notebook=True): if output_dir is None or output_dir == '': - name = os.path.splitext(os.path.basename(test_output_zip_path))[0] + '-processed' - output_dir = os.path.join(os.path.dirname(test_output_zip_path), name) + output_dir = os.path.join(os.path.dirname(test_output_zip_path), 'analysis') mkdirp(output_dir) aggregate_output(test_output_zip_path, output_dir) diff --git a/scripts/run.py b/scripts/run.py index ab174ee..61e738a 100755 --- a/scripts/run.py +++ b/scripts/run.py @@ -23,13 +23,17 @@ K8S_RUN_CONFIG = {} def parse_args(): parser = argparse.ArgumentParser() - parser.add_argument('template_dir', nargs=1, - help='path to directory containing composition template and param files') - parser.add_argument('param_files', nargs='*', help='name of one or more parameter files to use when generating composition from template. ' + 'if a param is defined in multiple files, last one wins.') + parser.add_argument('--name', + help='name of composition. will be used to create output directory.') + + parser.add_argument('--template_dir', + default='./templates/baseline', + help='path to directory containing composition template and param files') + parser.add_argument('-o', '--output', help='directory to write composition file and test outputs to', default='./output') @@ -164,12 +168,9 @@ def render_template(template_dir, params): return template.render(**params) -def composition_name(args): - template_name = os.path.basename(args.template_dir[0]) - variant = 'hardened' if args.hardened else 'vanilla' - strs = [template_name, variant] - strs += [os.path.splitext(os.path.basename(f))[0] for f in args.param_files] - return '-'.join(strs) +def composition_name(): + ts = time.strftime("%Y%m%d-%H%M%S") + return 'pubsub-test-{}'.format(ts) def mkdirp(dirpath): @@ -177,9 +178,8 @@ def mkdirp(dirpath): def run_composition(comp_filepath, output_dir, k8s=False): - ts = time.strftime("%Y%m%d-%H%M%S") archive_type = 'tgz' if k8s else 'zip' - outfilename = 'output-{}.{}'.format(ts, archive_type) + outfilename = 'test-output.{}'.format(archive_type) outpath = os.path.join(output_dir, outfilename) print('running testground composition {}'.format(comp_filepath)) @@ -203,7 +203,7 @@ def branch_commit(branch): def run(): args = parse_args() - template_dir = args.template_dir[0] + template_dir = args.template_dir params = load_params(template_dir, args.param_files) @@ -245,7 +245,10 @@ def run(): parse_n_attack_nodes(params) parse_n_container_nodes_total(params) - comp = composition_name(args) + comp = composition_name() + if args.name: + comp += '-' + args.name + workdir = os.path.join(args.output, comp) pathlib.Path(workdir).mkdir(parents=True, exist_ok=True) diff --git a/scripts/templates/baseline/params/_base.toml b/scripts/templates/baseline/params/_base.toml index ca38ccc..35b17c7 100644 --- a/scripts/templates/baseline/params/_base.toml +++ b/scripts/templates/baseline/params/_base.toml @@ -1,12 +1,8 @@ COMPOSITION_NAME = "baseline" # the set of go build tags to apply when building. -# to target the hardening branch of pubsub, use: -# BUILD_SELECTORS = ['hardened_api'] -# Note that if you set the 'hardened_api' build tag, you also -# need to also set GS_VERSION to a commit from the hardening -# branch, or you'll get a compile error. -BUILD_SELECTORS = [] +# set this to an empty array if you want to target a commit before gossipsub v1.1 was merged. +BUILD_SELECTORS = ['hardened_api'] # version of go-libp2p-pubsub to use when building the test plan GS_VERSION = "latest"