docs: migrate from Sphinx to Gitbook

2026-02-09 03:55:04 -05:00 · 2022-05-19 15:15:42 +02:00
parent dc501fb0ae
commit 2a42b5f711
45 changed files with 299 additions and 1712 deletions
--- a/docs/tutorial/compilation_artifacts.md
+++ b/docs/tutorial/compilation_artifacts.md
@@ -0,0 +1,247 @@
+# Compilation artifacts
+
+In this tutorial, we are going to go over the artifact system, which is designed to inspect/debug the compilation process easily.
+
+## Automatic export
+
+In case of compilation failures, artifacts are exported automatically to `.artifacts` directory under the working directory. Let's intentionally create a compilation failure and show what kinds of things are exported.
+
+<!--pytest-codeblocks:skip-->
+```python
+def f(x):
+    return np.sin(x)
+```
+
+This function fails to compile because **Concrete Numpy** doesn't support floating point outputs. When you try to compile it (you might want to check [this](../basics/compiling_and_executing.md) to see how you can do that), an exception will be raised and the artifacts will be exported automatically.
+
+### environment.txt
+
+This file contains information about your setup (i.e., your operating system and python version).
+
+```
+Linux-5.12.13-arch1-2-x86_64-with-glibc2.29 #1 SMP PREEMPT Fri, 25 Jun 2021 22:56:51 +0000
+Python 3.8.10
+```
+
+### requirements.txt
+
+This file contains information about python packages and their versions installed on your system.
+
+```
+alabaster==0.7.12
+appdirs==1.4.4
+argon2-cffi==21.1.0
+...
+wheel==0.37.0
+widgetsnbextension==3.5.1
+wrapt==1.12.1
+```
+
+### function.txt
+
+This file contains information about the function you are trying to compile.
+
+```
+def f(x):
+    return np.sin(x)
+```
+
+### parameters.txt
+
+This file contains information about the parameters of the function you are trying to compile.
+
+```
+x :: EncryptedScalar<Integer<unsigned, 3 bits>>
+```
+
+### 1.initial.graph.txt
+
+This file contains textual representation of the initial computation graph right after tracing.
+
+```
+%0 = x              # EncryptedScalar<uint3>
+%1 = sin(%0)        # EncryptedScalar<float64>
+return %1
+```
+
+### 1.initial.graph.png
+
+This file contains the visual representation of the initial computation graph right after tracing.
+
+![](../_static/tutorials/artifacts/auto/1.initial.graph.png)
+
+### 2.final.graph.txt
+
+This file contains textual representation of the final computation graph right before MLIR conversion.
+
+```
+%0 = x              # EncryptedScalar<uint3>
+%1 = sin(%0)        # EncryptedScalar<float64>
+return %1
+```
+
+### 2.final.graph.png
+
+This file contains the visual representation of the final computation graph right before MLIR conversion.
+
+![](../_static/tutorials/artifacts/auto/2.final.graph.png)
+
+### traceback.txt
+
+This file contains information about the error you got.
+
+```
+Traceback (most recent call last):
+  File "/home/default/Documents/Projects/Zama/hdk/concrete/numpy/compile.py", line 141, in run_compilation_function_with_error_management
+    return compilation_function()
+  File "/home/default/Documents/Projects/Zama/hdk/concrete/numpy/compile.py", line 769, in compilation_function
+    return _compile_numpy_function_internal(
+  File "/home/default/Documents/Projects/Zama/hdk/concrete/numpy/compile.py", line 722, in _compile_numpy_function_internal
+    fhe_circuit = _compile_op_graph_to_fhe_circuit_internal(
+  File "/home/default/Documents/Projects/Zama/hdk/concrete/numpy/compile.py", line 626, in _compile_op_graph_to_fhe_circuit_internal
+    prepare_op_graph_for_mlir(op_graph)
+  File "/home/default/Documents/Projects/Zama/hdk/concrete/numpy/compile.py", line 597, in prepare_op_graph_for_mlir
+    raise RuntimeError(
+RuntimeError: function you are trying to compile isn't supported for MLIR lowering
+
+%0 = x              # EncryptedScalar<uint3>
+%1 = sin(%0)        # EncryptedScalar<float64>
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ only integer outputs are supported
+return %1
+```
+
+## Manual export
+
+Manual exports are mostly used for visualization. Nonetheless, they can be very useful for demonstrations. Here is how to do it:
+
+```python
+import concrete.numpy as cnp
+import numpy as np
+import pathlib
+
+artifacts = cnp.DebugArtifacts("/tmp/custom/export/path")
+
+@cnp.compiler({"x": "encrypted"})
+def f(x):
+    return 127 - (50 * (np.sin(x) + 1)).astype(np.int64)
+
+f.compile(range(2 ** 3), artifacts=artifacts)
+
+artifacts.export()
+```
+
+### 1.initial.graph.txt
+
+This file contains textual representation of the initial computation graph right after tracing.
+
+```
+%0 = 127                             # ClearScalar<uint7>
+%1 = 50                              # ClearScalar<uint6>
+%2 = 1                               # ClearScalar<uint1>
+%3 = x                               # EncryptedScalar<uint3>
+%4 = sin(%3)                         # EncryptedScalar<float64>
+%5 = add(%4, %2)                     # EncryptedScalar<float64>
+%6 = mul(%5, %1)                     # EncryptedScalar<float64>
+%7 = astype(%6, dtype=uint32)        # EncryptedScalar<uint32>
+%8 = sub(%0, %7)                     # EncryptedScalar<uint32>
+return %8
+```
+
+### 1.initial.graph.png
+
+This file contains the visual representation of the initial computation graph right after tracing.
+
+![](../_static/tutorials/artifacts/manual/1.initial.graph.png)
+
+### 2.after-float-fuse-0.graph.txt
+
+This file contains textual representation of the intermediate computation graph after fusing.
+
+```
+%0 = 127                 # ClearScalar<uint7>
+%1 = x                   # EncryptedScalar<uint3>
+%2 = subgraph(%1)        # EncryptedScalar<uint32>
+%3 = sub(%0, %2)         # EncryptedScalar<uint32>
+return %3
+
+Subgraphs:
+
+    %2 = subgraph(%1):
+
+        %0 = 50                              # ClearScalar<uint6>
+        %1 = 1                               # ClearScalar<uint1>
+        %2 = float_subgraph_input            # EncryptedScalar<uint3>
+        %3 = sin(%2)                         # EncryptedScalar<float64>
+        %4 = add(%3, %1)                     # EncryptedScalar<float64>
+        %5 = mul(%4, %0)                     # EncryptedScalar<float64>
+        %6 = astype(%5, dtype=uint32)        # EncryptedScalar<uint32>
+        return %6
+```
+
+### 2.after-float-fuse-0.graph.png
+
+This file contains the visual representation of the intermediate computation graph after fusing.
+
+![](../_static/tutorials/artifacts/manual/2.after-float-fuse-0.graph.png)
+
+### 3.final.graph.txt
+
+This file contains textual representation of the final computation graph right before MLIR conversion.
+
+```
+%0 = 127                 # ClearScalar<uint7>
+%1 = x                   # EncryptedScalar<uint3>
+%2 = subgraph(%1)        # EncryptedScalar<uint7>
+%3 = sub(%0, %2)         # EncryptedScalar<uint7>
+return %3
+
+Subgraphs:
+
+    %2 = subgraph(%1):
+
+        %0 = 50                              # ClearScalar<uint6>
+        %1 = 1                               # ClearScalar<uint1>
+        %2 = float_subgraph_input            # EncryptedScalar<uint3>
+        %3 = sin(%2)                         # EncryptedScalar<float64>
+        %4 = add(%3, %1)                     # EncryptedScalar<float64>
+        %5 = mul(%4, %0)                     # EncryptedScalar<float64>
+        %6 = astype(%5, dtype=uint32)        # EncryptedScalar<uint7>
+        return %6
+```
+
+### 3.final.graph.png
+
+This file contains the visual representation of the final computation graph right before MLIR conversion.
+
+![](../_static/tutorials/artifacts/manual/3.final.graph.png)
+
+### bounds.txt
+
+This file contains information about the bounds of the final computation graph of the function you are trying to compile using the input set you provide.
+
+```
+%0 :: [127, 127]
+%1 :: [0, 7]
+%2 :: [2, 95]
+%3 :: [32, 125]
+```
+
+You can learn what bounds are [here](../../dev/explanation/terminology_and_structure.md).
+
+### mlir.txt
+
+This file contains information about the MLIR of the function you are trying to compile using the input set you provide.
+
+```
+module  {
+  func @main(%arg0: !FHE.eint<7>) -> !FHE.eint<7> {
+    %c127_i8 = arith.constant 127 : i8
+    %cst = arith.constant dense<"..."> : tensor<128xi64>
+    %0 = "FHE.apply_lookup_table"(%arg0, %cst) : (!FHE.eint<7>, tensor<128xi64>) -> !FHE.eint<7>
+    %1 = "FHE.sub_int_eint"(%c127_i8, %0) : (i8, !FHE.eint<7>) -> !FHE.eint<7>
+    return %1 : !FHE.eint<7>
+  }
+}
+```
+
+You can learn more about MLIR [here](../../dev/explanation/mlir.md).
--- a/docs/tutorial/indexing.md
+++ b/docs/tutorial/indexing.md
@@ -0,0 +1,95 @@
+# Indexing
+
+## Constant Indexing
+
+Constant indexing refers to the index being static (i.e., known during compilation).
+
+Here are some examples of constant indexing:
+
+### Extracting a single element
+
+```python
+import concrete.numpy as cnp
+import numpy as np
+
+@cnp.compiler({"x": "encrypted"})
+def f(x):
+    return x[1]
+
+inputset = [np.random.randint(0, 2 ** 3, size=(3,), dtype=np.uint8) for _ in range(10)]
+circuit = f.compile(inputset)
+
+test_input = np.array([4, 2, 6], dtype=np.uint8)
+expected_output = 2
+
+assert np.array_equal(circuit.encrypt_run_decrypt(test_input), expected_output)
+```
+
+You can use negative indexing.
+
+```python
+import concrete.numpy as cnp
+import numpy as np
+
+@cnp.compiler({"x": "encrypted"})
+def f(x):
+    return x[-1]
+
+inputset = [np.random.randint(0, 2 ** 3, size=(3,), dtype=np.uint8) for _ in range(10)]
+circuit = f.compile(inputset)
+
+test_input = np.array([4, 2, 6], dtype=np.uint8)
+expected_output = 6
+
+assert np.array_equal(circuit.encrypt_run_decrypt(test_input), expected_output)
+```
+
+You can use multidimensional indexing as well.
+
+```python
+import concrete.numpy as cnp
+import numpy as np
+
+@cnp.compiler({"x": "encrypted"})
+def f(x):
+    return x[-1, 1]
+
+inputset = [np.random.randint(0, 2 ** 3, size=(3, 2), dtype=np.uint8) for _ in range(10)]
+circuit = f.compile(inputset)
+
+test_input = np.array([[4, 2], [1, 5], [7, 6]], dtype=np.uint8)
+expected_output = 6
+
+assert np.array_equal(circuit.encrypt_run_decrypt(test_input), expected_output)
+```
+
+### Extracting a slice
+
+```python
+import concrete.numpy as cnp
+import numpy as np
+
+@cnp.compiler({"x": "encrypted"})
+def f(x):
+    return x[1:4]
+
+inputset = [np.random.randint(0, 2 ** 3, size=(5,), dtype=np.uint8) for _ in range(10)]
+circuit = f.compile(inputset)
+
+test_input = np.array([4, 2, 6, 1, 7], dtype=np.uint8)
+expected_output = np.array([2, 6, 1], dtype=np.uint8)
+
+assert np.array_equal(circuit.encrypt_run_decrypt(test_input), expected_output)
+```
+
+You can use multidimensional slicing as well.
+
+{% hint style='tip' %}
+There are certain limitations of slicing due to MLIR. So if you stumple into `RuntimeError: Compilation failed: Failed to lower to LLVM dialect`, know that we are aware of it, and we are trying to make such cases compilable.
+{% endhint %}
+
+## Dynamic Indexing
+
+Dynamic indexing refers to the index being dynamic (i.e., can change during runtime).
+Such indexing is especially useful for things like decision trees.
+Unfortunately, we don't support dynamic indexing for the time being.
--- a/docs/tutorial/table_lookup.md
+++ b/docs/tutorial/table_lookup.md
@@ -0,0 +1,163 @@
+# Table lookup
+
+In this tutorial, we are going to go over the ways to perform direct table lookups in **Concrete Numpy**. Please read [Compiling and Executing](../basics/compiling\_and\_executing.md) before reading further to see how you can compile the functions below.
+
+## Direct table lookup
+
+**Concrete Numpy** provides a special class to allow direct table lookups. Here is how to use it:
+
+```python
+import concrete.numpy as cnp
+
+table = cnp.LookupTable([2, -1, 3, 0])
+
+def f(x):
+    return table[x]
+```
+
+where
+
+* `x = "encrypted"` scalar
+
+results in
+
+<!--pytest-codeblocks:skip-->
+```python
+circuit.encrypt_run_decrypt(0) == 2
+circuit.encrypt_run_decrypt(1) == -1
+circuit.encrypt_run_decrypt(2) == 3
+circuit.encrypt_run_decrypt(3) == 0
+```
+
+Moreover, direct lookup tables can be used with tensors where the same table lookup is applied to each value in the tensor, so
+
+* `x = "encrypted"` tensor of shape `(2, 3)`
+
+results in
+
+<!--pytest-codeblocks:skip-->
+```python
+input = np.array([[0, 1, 3], [2, 3, 1]], dtype=np.uint8)
+circuit.encrypt_run_decrypt(input) == [[2, 1, 0], [3, 0, 1]]
+```
+
+Direct table lookups behaves like array indexing in python. Which means, if the lookup variable is negative, table is looked up from the back.
+
+```python
+import concrete.numpy as cnp
+
+table = cnp.LookupTable([2, 1, 3, 0])
+
+def f(x):
+    return table[-x]
+```
+
+where
+
+* `x = "encrypted"` scalar
+
+results in
+
+<!--pytest-codeblocks:skip-->
+```python
+circuit.encrypt_run_decrypt(0) == 2
+circuit.encrypt_run_decrypt(1) == 0
+circuit.encrypt_run_decrypt(2) == 3
+circuit.encrypt_run_decrypt(3) == 1
+circuit.encrypt_run_decrypt(4) == 2
+```
+
+Lastly, a `LookupTable` can have any number of elements, let's call it **N**, as long as the lookup variable is in range \[-**N**, **N**). If you go out of bounds of this range, you will get the following error:
+
+```
+IndexError: index 10 is out of bounds for axis 0 with size 6
+```
+
+Note that, number of elements in the lookup table doesn't affect the performance in any way.
+
+## Direct multi table lookup
+
+Sometimes you may want to apply a different lookup table to each value in a tensor. That's where direct multi lookup table becomes handy. Here is how to use it:
+
+<!--pytest-codeblocks:skip-->
+```python
+import concrete.numpy as cnp
+
+squared = cnp.LookupTable([i ** 2 for i in range(4)])
+cubed = cnp.LookupTable([i ** 3 for i in range(4)])
+
+table = cnp.LookupTable([
+    [squared, cubed],
+    [squared, cubed],
+    [squared, cubed],
+])
+
+def f(x):
+    return table[x]
+```
+
+where
+
+* `x = "encrypted"` tensor of shape `(3, 2)`
+
+results in
+
+<!--pytest-codeblocks:skip-->
+```python
+input = np.array([[2, 3], [1, 2], [3, 0]], dtype=np.uint8)
+circuit.encrypt_run_decrypt(input) == [[4, 27], [1, 8], [9, 0]]
+```
+
+Basically, we applied `squared` table to the first column and `cubed` to the second one.
+
+## Fused table lookup
+
+Direct tables are tedious to prepare by hand. When possible, **Concrete Numpy** fuses the floating point operations into table lookups automatically. There are some limitations on fusing operations, which you can learn more about on the next tutorial, [Working With Floating Points](working\_with\_floating\_points.md).
+
+Here is an example function that results in fused table lookup:
+
+<!--pytest-codeblocks:skip-->
+```python
+def f(x):
+    return 127 - (50 * (np.sin(x) + 1)).astype(np.int64) # astype is to go back to integer world
+```
+
+where
+
+* `x = "encrypted"` scalar
+
+results in
+
+<!--pytest-codeblocks:skip-->
+```python
+circuit.encrypt_run_decrypt(0) == 77
+circuit.encrypt_run_decrypt(1) == 35
+circuit.encrypt_run_decrypt(2) == 32
+circuit.encrypt_run_decrypt(3) == 70
+circuit.encrypt_run_decrypt(4) == 115
+circuit.encrypt_run_decrypt(5) == 125
+circuit.encrypt_run_decrypt(6) == 91
+circuit.encrypt_run_decrypt(7) == 45
+```
+
+Initially, the function is converted to this operation graph
+
+![](../\_static/tutorials/table-lookup/1.initial.graph.png)
+
+and after floating point operations are fused, we get the following operation graph
+
+![](../\_static/tutorials/table-lookup/3.final.graph.png)
+
+Internally, it uses the following lookup table
+
+<!--pytest-codeblocks:skip-->
+```python
+table = cnp.LookupTable([50, 92, 95, 57, 12, 2, 36, 82])
+```
+
+which is calculated by:
+
+<!--pytest-codeblocks:skip-->
+```python
+[(50 * (np.sin(x) + 1)).astype(np.int64) for x in range(2 ** 3)]
+```
--- a/docs/tutorial/working_with_floating_points.md
+++ b/docs/tutorial/working_with_floating_points.md
@@ -0,0 +1,36 @@
+# Working with floating points
+
+## An example
+
+```python
+import concrete.numpy as cnp
+import numpy as np
+
+# Function using floating points values converted back to integers at the end
+@cnp.compiler({"x": "encrypted"})
+def f(x):
+    return np.fabs(50 * (2 * np.sin(x) * np.cos(x))).astype(np.int64)
+    # astype is to go back to the integer world
+
+circuit = f.compile(range(64))
+
+print(circuit.encrypt_run_decrypt(3) == f(3))
+print(circuit.encrypt_run_decrypt(0) == f(0))
+print(circuit.encrypt_run_decrypt(1) == f(1))
+print(circuit.encrypt_run_decrypt(10) == f(10))
+print(circuit.encrypt_run_decrypt(60) == f(60))
+
+print("All good!")
+```
+
+You can look to [numpy supported functions](../howto/numpy\_support.md) for information about possible float operations.
+
+## Limitations
+
+Floating point support in **Concrete Numpy** is very limited for the time being. They can't appear on inputs, or they can't be outputs. However, they can be used in intermediate results. Unfortunately, there are limitations on that front as well.
+
+This biggest one is that, because floating point operations are fused into table lookups with a single unsigned integer input and single unsigned integer output, only univariate portion of code can be replaced with table lookups, which means multivariate portions cannot be compiled.
+
+To give a precise example, `100 - np.fabs(50 * (np.sin(x) + np.sin(y)))` cannot be compiled because the floating point part depends on both `x` and `y` (i.e., it cannot be rewritten in the form `100 - table[z]` for a `z` that could be computed easily from `x` and `y`).
+
+To dive into implementation details, you may refer to [Fusing Floating Point Operations](../../developer/float-fusing.md) document.