Icicle example: using multiple GPU to hash large dataset

Best-Practices

This example builds on single GPU Poseidon example so we recommend to run it first.

Key-Takeaway

Use device_context::DeviceContext variable to select GPU to use. Use C++ threads to compute Icicle primitives on different GPUs in parallel.

Concise Usage Explanation

Include c++ threads

#include <thread>

Define a thread function. Importantly, device context ctx will hold the GPU id.

void threadPoseidon(device_context::DeviceContext ctx, ...) {...}

Initialize device contexts for different GPUs

device_context::DeviceContext ctx0 = device_context::get_default_device_context();
ctx0.device_id=0;
device_context::DeviceContext ctx1 = device_context::get_default_device_context();
ctx1.device_id=1;

Finally, spawn the threads and wait for their completion

std::thread thread0(threadPoseidon, ctx0, ...);
std::thread thread1(threadPoseidon, ctx1, ...);
thread0.join();
thread1.join();

What's in the example

This is a toy example executing the first step of the Filecoin's Pre-Commit 2 phase: compute 2^{30} Poseison hashes for each column of 11 \times 2^{30} matrix.

Define the size of the example: 2^{30} won't fit on a typical machine, so we partition the problem into nof_partitions
Hash two partitions in parallel on two GPUs
Hash two partitions in series on one GPU
Compare execution times