e8ef48ffd8
This commit: + Adds support for a protocol which enables inter-op between concrete, tfhe-rs and potentially other contributors to the fhe ecosystem. + Gets rid of hand-made serialization in the compiler, and client/server libs. + Refactors client/server libs to allow more pre/post processing of circuit inputs/outputs. The protocol is supported by a definition in the shape of a capnp file, which defines different types of objects among which: + ProgramInfo object, which is a precise description of a set of fhe circuit coming from the same compilation (understand function type information), and the associated key set. + *Key objects, which represent secret/public keys used to encrypt/execute fhe circuits. + Value object, which represent values that can be transferred between client and server to support calls to fhe circuits. The hand-rolled serialization that was previously used is completely dropped in favor of capnp in the whole codebase. The client/server libs, are refactored to introduce a modular design for pre-post processing. Reading the ProgramInfo file associated with a compilation, the client and server libs assemble a pipeline of transformers (functions) for pre and post processing of values coming in and out of a circuit. This design properly decouples various aspects of the processing, and allows these capabilities to be safely extended. In practice this commit includes the following: + Defines the specification in a concreteprotocol package + Integrate the compilation of this package as a compiler dependency via cmake + Modify the compiler to use the Encodings objects defined in the protocol + Modify the compiler to emit ProgramInfo files as compilation artifact, and gets rid of the bloated ClientParameters. + Introduces a new Common library containing the functionalities shared between the compiler and the client/server libs. + Introduces a functional pre-post processing pipeline to this common library + Modify the client/server libs to support loading ProgramInfo objects, and calling circuits using Value messages. + Drops support of JIT. + Drops support of C-api. + Drops support of Rust bindings. Co-authored-by: Nikita Frolov <nf@mkmks.org>
Concrete Compiler
The Concrete Compiler is a set of tools that allows the compilation and from an high-level and crypto free representation of an arithmetic circuit of operations on encrypted integers. This compiler is based on the MLIR project it use the framework, the standard dialects exposed by MLIR and define new fhe specific dialects and passes to lower the high-level fhe dialects to standard MLIR dialects.
Getting started
The source of the project is located in the compiler directory.
cd compiler
Prerequisite: Install build tools
If you intend to build the Compiler from source, you should make sure to have build-essentials (or equivalent if not on Ubuntu), CMake, and Ninja installed. Optionally, you can also install Ccache to speedup compilations after small changes.
Prerequisite: Install Rust
The Compiler depends on some Rust libraries, including the optimizer and the concrete-cpu backend. You can install it from rustup.
Note: some crates use Rust nightly currently, and it might be required to install both stable and nightly distributions.
Prerequisite: Building HPX and enable dataflow parallelism (optional)
In order to implement the dataflow parallelism and the distribution of the computation we use the HPX Standard Library. You can else use your own HPX installation by set the HPX_INSTALL_DIR environment variable or you can install HPX on the default path of our build system thanks the following command:
make install-hpx-from-source
This may fail on some systems when dependencies are missing. Some recent packages required are Cmake, HWLOC and BOOST. For full details see HPX Quickstart guide.
Once you have a proper installation of HPX to enable the dataflow parallelism set the DATAFLOW_EXECUTION_ENABLED=ON.
Prerequisite: Fetch git submodules
This project rely on llvm-project and concrete-optimizer as git submodules so you need to initialize and update the git submodules.
git submodule init
git submodule update
Prerequisite: python packages
Install MLIR python requirements in your dev python environment:
# From repo root
pip install -r ./llvm-project/mlir/python/requirements.txt
# From compiler dir
pip install -r ../llvm-project/mlir/python/requirements.txt
You should also have the python development package installed.
Build from source
We use cmake as the main build system but in order to initialize the build system and define straightforward target for the main artifacts of the project. You can initialize and build all the main artifacts thanks the following command:
make all
or in several steps:
Generate the compiler build system, in a build-* directory
make build-initialized
Build the compiler
make concretecompiler
Run the compiler
./build-Release/bin/concretecompiler
Debug build and custom linker
To build a debug version of the project, you can set BUILD_TYPE=Debug in the Makefile. In Debug
the build system will detect if the lld linker is installed on the system and use it. lld is much faster
than the default ld linker. Release builds with lld can also be enabled by modifying the Makefile.
Installation from source
You can install libs, bins, and include files into a specific directory by running:
make INSTALL_PREFIX=/your/directory install
You will then find lib, bin, and include under /your/directory/concretecompiler.
Tests
You can build all the tests with the following command:
make build-tests
and run them with:
make run-tests
Benchmarks
You can build all the benchmarks with the following command:
make build-benchmarks
and run them with:
make run-benchmarks
Build releases
Build tarball
You can create a tarball containing libs, bins, and include files for the tools of the compiler, by following previous steps of installation from source, then creating a tar archive from the installation directory.
Build the Python Package
Currently supported platforms:
- Linux x86_64 for python 3.8, 3.9, 3.10, and 3.11
pybind11 is required to build the python package, you can install it in your current environment with:
$ pip install pybind11
To specify which python executable to target you can specify the Python3_EXECUTABLE environment variable.
Build wheels in your environment
Building the wheels is actually simple.
$ pip wheel --no-deps -w ../wheels .
Depending on the platform you are using (specially Linux), you might need to use auditwheel to specify the platform this wheel is targeting. For example, in our build of the package for Linux x86_64 and GLIBC 2.24, we also run:
$ auditwheel repair ../wheels/*.whl --plat manylinux_2_24_x86_64 -w ../wheels