docs: updating the doc a bit

refs #1050

docs/dev/explanation/COMPILATION.md

@@ -1,3 +1,9 @@
+
+```{warning}
+FIXME(umut): update with the new API
+```
+
+
 # Compilation Pipeline In Depth
 
 ## What is **concretefhe**?
@@ -54,6 +60,10 @@ Once the MLIR is prepared, the rest of the stack, which you can learn more about
 
 Here is the visual representation of the pipeline:
 
+```{warning}
+FIXME(arthur): check the graph, update with what is missing, notably: torch to numpy, quantization. Maybe an independant graph for quantization may be clearer.
+```
+
 ![Frontend Flow](../../_static/compilation-pipeline/frontend_flow.svg)
 
 ## Tracing

docs/dev/explanation/FLOAT-FUSING.md

@@ -1,3 +1,7 @@
+```{warning}
+FIXME(Arthur): explain recent updates on the fusing
+```
+
 # Fusing Floating Point Operations
 
 ## Why is it needed?

docs/dev/explanation/QUANTIZATION.md

@@ -0,0 +1,8 @@
+```{warning}
+FIXME(Jordan/Andrei): do this
+```
+
+# Quantization in Depth
+
+(as a researcher)
+

docs/dev/explanation/TERMINOLOGY_AND_STRUCTURE.md

@@ -1,3 +1,7 @@
+```{warning}
+FIXME(Benoit): to check this is still valid
+```
+
 # Terminology and Structure
 
 ## Terminology

docs/dev/howto/CONTRIBUTING.md

@@ -1,6 +1,10 @@
 
 # Contributing
 
+```{warning}
+FIXME(alex): to see if something here needs some update
+```
+
 ```{important}
 There are two ways to contribute to **concretefhe**:
 - you can open issues to report bugs, typos and suggest ideas
@@ -51,6 +55,10 @@ The last requirement is to make sure you get a hundred percent code coverage. Yo
 make coverage
 ```
 
+```{warning}
+FIXME(arthur): is this still valid? (that `make pytest` will not return an error)
+```
+
 Remark that only calling `make pytest` will give you information about the coverage, at the end of the execution, but the test will not return a failure if the coverage is not a hundred percent, as opposed to a call to `make coverage`.
 
 Note that this will compare the coverage with `origin/main`. If you want to set a custom base branch, you can specify `BB` environment variable like so `BB=$YOUR_BASE_BRANCH make coverage`.
@@ -73,7 +81,7 @@ git commit -m "feat(debugging): add an helper function to draw intermediate repr
 git commit -m "fix(tracing): fix a bug that crashed pytorch tracer"
 ```
 
-To learn more about conventional commits, check [this](https://www.conventionalcommits.org/en/v1.0.0/) page.
+To learn more about conventional commits, check [this](https://www.conventionalcommits.org/en/v1.0.0/) page. Remark that commit messages are checked in the comformance step, and rejected if they don't follow the rules.
 
 ## Before creating pull request
 

docs/dev/howto/DOCKER.md

@@ -1,3 +1,10 @@
+```{warning}
+FIXME(arthur): to update if needed
+```
+```{warning}
+FIXME(arthur): to add a new .md about pypy if needed
+```
+
 # Docker
 
 ## Setting up docker and X forwarding

docs/dev/howto/PROJECT_SETUP.md

@@ -1,3 +1,6 @@
+```{warning}
+FIXME(Arthur): to check what needs to be updated here
+```
 
 # Project Setup
 

docs/dev/howto/RELEASING.md

@@ -1,3 +1,7 @@
+```{warning}
+FIXME(arthur): to update with the new workflow
+```
+
 # Creating A Release On GitHub
 
 ## Release Candidate cycle

docs/dev/index.rst

@@ -19,3 +19,4 @@ Developer Guide
    explanation/TERMINOLOGY_AND_STRUCTURE.md
    explanation/FLOAT-FUSING.md
    explanation/MLIR.md
+   explanation/QUANTIZATION.md

docs/user/README.md

@@ -17,11 +17,16 @@ FHE is also a killer feature regarding data breaches: as anything done on the se
 In the first version of Concrete, there is a single frontend, called homomorphic numpy (or hnp), which is the equivalent of numpy. With our toolchain, a data scientist can convert a numpy program into an FHE program, without any a-priori knowledge on cryptography.
 ```
 
+```{note}
+On top of the numpy frontend, we are adding an alpha-version of a torch compiler, which basically transforms a subset of torch modules into numpy, and then use numpy frontend and the compiler. This is an early version of a more stable torch compiler which will be released later in the year.
+```
+
 ## Organization of the documentation
 
 Basically, we have divided our documentation into several parts:
 - one about basic elements, notably description of the installation, that you are currently reading
 - one dedicated to _users_ of **Concrete**, with tutorials, how-to's and deeper explanations
+- one detailing the API's of the different functions of the frontend, directly done by parsing its source code
 - and finally, one dedicated to _developers_ of **Concrete**, who could be internal or external contributors to the framework
 
 ## A work in progress
@@ -33,8 +38,13 @@ Concrete is a work in progress, and is currently limited to a certain number of
 The main _current_ limits are:
 - **Concrete** is only supporting unsigned integers
 - **Concrete** needs the integer to be less than 7 bits (included)
-- **Concrete** is mostly restricted to scalars (by opposition to tensors). The only exception is the `dot` operator, which can dot a tensor of encrypted values with a tensor of constant values
 
-The first two limits can be taken care of with the use of quantization, as explained a bit further in [this](explanation/QUANTIZATION.md) and [this](howto/REDUCE_NEEDED_PRECISION.md) parts of the documentation.
+These limits can be taken care of with the use of quantization, as explained a bit further in [this](explanation/QUANTIZATION.md) and [this](howto/REDUCE_NEEDED_PRECISION.md) parts of the documentation.
 
-The scalar limitation is mainly an engineering issue, and will be fixed in the next release. Today, one needs to split all the tensors into small scalars, which is inconvenient and will be no more needed very soon.
+```{warning}
+FIXME(Jordan): speak about our quantization framework
+```
+
+```{warning}
+FIXME(Jordan/Andrei): add an .md about the repository of FHE-friendly models, and ideally .ipynb's
+```

docs/user/explanation/FHE_AND_FRAMEWORK_LIMITS.md

@@ -20,12 +20,12 @@ For most FHE scheme but TFHE, the application of a non-linear function is compli
 
 Since this is an early version of the product, not everything is done, to say the least. What we wanted to tackle first was the cryptographic complexities. This is why we concentrated on the cryptographic part, and let some engineering problems for later.
 
-### Limited to scalars
-
-Today, the **Concrete Framework** is mostly limited to scalars. Notably, in our numpy frontend, we can not use [tensors](https://numpy.org/doc/stable/user/theory.broadcasting.html?highlight=vector). As explained in [this section](FUTURE_FEATURES.md), this limit will be removed in the next version.
-
 ### Currently executing locally
 
+```{warning}
+FIXME(Benoit): we'll see later if this is still a valid limit
+```
+
 As of today, the execution of the FHE program is done locally. Notably, in the current version, there is no client (on which we encrypt the private data, or decrypt the returned result) or server (on which the computation is done completely over encrypted data), but a single host. As explained in [this section](FUTURE_FEATURES.md), this limit will be removed in the next version, such that the **Concrete Framework** can be used in production.
 
 ### Currently slow

docs/user/explanation/FUTURE_FEATURES.md

@@ -5,21 +5,12 @@ is currently in a preliminary version, and quite constrained in term of function
 news is that we are going to release new versions regularly, where a lot of functionalities will be added progressively.
 
 In this page, we briefly list what the plans for next versions of the **Concrete Framework** are:
-- **management of tensors**: today, we are mostly limited to scalars, but in the next version, the functions we compile
-will possibly contain tensors, which is one of the basic features of `numpy`
 - **better performance**: further versions will contain improved versions of the **Concrete Library**, with faster
 execution; also, the **Concrete Compiler** will be improved, to have faster local execution (with multi-threading
 for example) and faster production execution (with distribution over a set of machines or use of hardware accelerations)
-- **more user-friendly API's**: we would like to make our API easier for a user. Notably, we would like to allow direct
-compilations of classic Machine Learning framework models (e.g., tensorflow or pytorch)
-- **more complete benchmarks**: we will have an extended benchmark, containing lots of functions that one day one would
-want to compile; then, we will measure the framework progress by tracking the number of successfully compiled functions
-over time. Also, this public benchmark will be a way for other competing frameworks or technologies to compare fairly
-with us, in terms of functionality or performance
-- **easier installation**: we plan to have pip installation of our framework very soon
-- **Machine Learning helpers**: our midterm direction is to provide our users a set of tools to help her turn her use case
-in an homomorphic equivalent. This set of tools will help her reduce the needed variable precision and/or optimize the
-operations required to make the fastest possible compiled model.
+- **more support for torch, and support for other ML frameworks**: we will continue to extend our support for torch models, and have conversions from Keras, tensorflow
+- **more complete benchmarks**: we will have an extended benchmark, containing lots of functions that one day one would want to compile; then, we will measure the framework progress by tracking the number of successfully compiled functions over time. Also, this public benchmark will be a way for other competing frameworks or technologies to compare fairly with us, in terms of functionality or performance
+- **Machine Learning helpers**: our midterm direction is to provide our users a set of tools to help her turn her use case in an homomorphic equivalent. This set of tools will help her reduce the needed variable precision and/or optimize the operations required to make the fastest possible compiled model.
 
 Also, if you are especially looking for some new feature, you can drop a message to <hello@zama.ai>.
 

docs/user/explanation/QUANTIZATION.md

@@ -1,3 +1,7 @@
+```{warning}
+FIXME(Jordan/Andrei): see if this is still appropriate, update etc; make link with USE_QUANTIZATION.md
+```
+
 # Quantization
 
 ```{note}

docs/user/explanation/WHAT_IS_FHE.md

@@ -10,3 +10,7 @@ You can learn more about FHE using the following links:
 - [monthly technical FHE.org meetup](https://www.meetup.com/fhe-org/)
 - [videos and resources](http://fhe.org/)
 
+```{warning}
+FIXME(Alex/Jeremy): should we link to Zama blogposts or not?
+FIXME(Benoit): if yes, I'll do it
+```

docs/user/howto/COMPILING_AND_EXECUTING.md

@@ -1,3 +1,7 @@
+```{warning}
+FIXME(all): I've moved this section (from how-to) to the basic section, because it must be read before the tutorials
+```
+
 # Compiling and Executing
 
 ## Importing necessary components
@@ -20,6 +24,12 @@ def f(x, y):
 
 ## Compiling the function
 
+```{warning}
+FIXME(arthur): move to the new API
+FIXME(all): do you think guys we should just explain the new API or should we explain the two API's? I would say: just the new one, and maybe the old one in the dev section (but not ever sure).
+
+```
+
 To compile the function, you need to provide what are the inputs that it's expecting. In the example function above, `x` and `y` could be scalars or tensors (though, for now, only dot between tensors are supported), they can be encrypted or clear, they can be signed or unsigned, they can have different bit-widths. So, we need to know what they are beforehand. We can do that like so:
 
 <!--python-test:cont-->
@@ -68,8 +78,14 @@ Be careful about the inputs, though.
 If you were to run with values outside the range of the inputset, the result might not be correct.
 ```
 
+```{warning}
+FIXME(benoit): explain the API to encrypt, run_inference, decrypt, keygen etc when they are available
+
+```
+
 ## Further reading
 
 - [Arithmetic Operations Tutorial](../tutorial/ARITHMETIC_OPERATIONS.md)
 - [Working With Floating Points Tutorial](../tutorial/WORKING_WITH_FLOATING_POINTS.md)
 - [Table Lookup Tutorial](../tutorial/TABLE_LOOKUP.md)
+- [Compiling a torch model](../tutorial/COMPILING_TORCH_MODEL.md)

docs/user/howto/DEBUG_SUPPORT_SUBMIT_ISSUES.md

@@ -1,6 +1,6 @@
 # Debugging / Support / Submitting Issues
 
-First, let's not forget that this version of **Concrete** is a beta product, meaning that it is not completely polished, contains several bugs (would they be known or unknown at this time). Also, let's not forget that FHE is a highly hot topic, and notably, that it cannot be considered as a solved problem.
+First, let's not forget that this version of **Concrete** is a first version of the product, meaning that it is not completely finished, contains several bugs (would they be known or unknown at this time), and will improve over time with the feedback from early users. Also, let's not forget that FHE is a highly hot topic, and notably, that it cannot be considered as a solved problem.
 
 Anyway, let's list some ways to debug your problems here. If nothing seems conclusive, you can still report the issue, as explained in a later section of this page.
 
@@ -31,16 +31,22 @@ Once you're sure it is a bug, it would be nice to try to:
 
 ## Asking the community
 
+```{warning}
+FIXME(Alex): add the different links for the community
+```
+
 We have created a Slack channel (TODO: LINK TO BE ADDED), such that you can directly ask the developers and community about your issue.
 
 Hopefully, it is just a misunderstanding or a small mistake on your side, that one can help you fix easily. And, the good point with your feedback is that, once we have heard the problem or misunderstanding, we can make the documentation even clearer (such as, completing the FAQ).
 
 ## Having a look to the compilation artifacts
 
-When things are more complicated, or if you want to have a look by yourself, you may want to have a look to the compilation reports, which are called artifacts. This is as simple as described in [TODO: add the link to the tutorial about having artifacts].
+When things are more complicated, or if you want to have a look by yourself, you may want to have a look to the compilation reports, which are called artifacts. This is as simple as described in [here](../tutorial/COMPILATION_ARTIFACTS.md)
 
 This function will create a directory, containing notably:
-[TODO: Umut to fix / complete the following information]
+```{warning}
+FIXME(Umut): check it is still accurate
+```
 - bounds.txt: a file describing the expected ranges of data in the different steps of the computation
 - cryptographic_parameters.txt: a file describing the different keys
 - ir_nodes.txt: a file describing the different nodes in the intermediate representation (IR)
@@ -50,7 +56,10 @@ This function will create a directory, containing notably:
 Attaching the artifact with your issue or Slack message may help people to have a look at the core of the problem.
 The more precise your bug, the more likely we can reproduce and fix
 
-[TODO: Umut, is it still needed or do we already have some of those information in artifacts?]
+```{warning}
+FIXME(Umut): is it still needed or do we already have some of those information in artifacts?
+```
+
 In order to simplify our work and let us reproduce your bug easily, any information is useful. Notably, in addition to the python script, some information like:
 - the OS version
 - the python version
@@ -62,4 +71,8 @@ may be useful to us. Don't remember, **Concrete** is a project where we are open
 
 ## Submitting an issue
 
+```{warning}
+FIXME(Alex): add the link
+```
+
 In case you have a bug, which is reproducible, that you have reduced to a small piece of code,  we have our issue tracker (TODO: LINK TO BE ADDED). Remember that a well-described short issue is an issue which is more likely to be studied and fixed. The more issues we receive, the better the product will be.

docs/user/howto/INSTALLING.md

@@ -3,7 +3,11 @@
 ## Docker image
 
 ```{note}
-Currently the project is only available as a docker image. To get the image you need to login to ghcr.io with docker.
+The easiest way to install the framework is as a docker image. To get the image you need to login to ghcr.io with docker.
+```
+
+```{warning}
+FIXME(Arthur): to check this is still valid
 ```
 
 ```shell
@@ -37,3 +41,11 @@ Alternatively you can just open a shell in the docker:
 ```shell
 docker run --rm -it ghcr.io/zama-ai/concretefhe:v0.1.0 /bin/bash
 ```
+
+## python package
+
+```{warning}
+FIXME(Arthur): explain how to install from pypi, when it is ready
+```
+
+

docs/user/howto/PRINTING_AND_DRAWING.md

@@ -1,3 +1,7 @@
+```{warning}
+FIXME(all): should we update to the new API or have it for both?
+```
+
 # Printing and Drawing
 
 Sometimes, it can be useful to print or draw fhe circuits, we provide methods to just do that. Please read [Compiling and Executing](../howto/COMPILING_AND_EXECUTING.md) before reading further to see how you can compile your function into an fhe circuit.

docs/user/howto/REDUCE_NEEDED_PRECISION.md

@@ -1,3 +1,7 @@
+```{warning}
+FIXME(Umut): put somewhere an example of an error dump if the user asks for too much precision and explain
+```
+
 # Having a Function Which Requires Less Precision
 
 ## Why can some computation work with less precision?
@@ -23,3 +27,7 @@ If for some reason you have a tolerance on the result's precision, then you can
 This is illustrated in both advanced examples [Quantized Linear Regression](../advanced_examples/QuantizedLinearRegression.ipynb) and [Quantized Logistic Regression](../advanced_examples/QuantizedLogisticRegression.ipynb).
 
 The end result has a granularity/imprecision linked to the data types used and for the Quantized Logistic Regression to the lattice used to evaluate the logistic model.
+
+```{warning}
+FIXME(jordan/andrei): update with your insights / knowledge on ML
+```

docs/user/howto/USE_QUANTIZATION.md

@@ -0,0 +1,9 @@
+```{warning}
+FIXME(Jordan): do this
+```
+
+# Using Quantization in Concrete Framework
+
+(as a user)
+
+explain issues the user will still fail and that we plan to research on it

docs/user/index.rst

@@ -8,22 +8,26 @@ Getting Started
    README.md
    howto/INSTALLING.md
    benchmarks.md
+   howto/COMPILING_AND_EXECUTING.md
 
 .. toctree::
    :maxdepth: 1
    :caption: Tutorial
 
+   tutorial/COMPILING_TORCH_MODEL.md
    tutorial/ARITHMETIC_OPERATIONS.md
    tutorial/TABLE_LOOKUP.md
    tutorial/WORKING_WITH_FLOATING_POINTS.md
+   tutorial/INDEXING.md
+   tutorial/MACHINE_LEARNING_TOOLS.md
    tutorial/COMPILATION_ARTIFACTS.md
 
 .. toctree::
    :maxdepth: 1
    :caption: How to
 
-   howto/COMPILING_AND_EXECUTING.md
    howto/PRINTING_AND_DRAWING.md
+   howto/USE_QUANTIZATION.md
    howto/REDUCE_NEEDED_PRECISION.md
    howto/DEBUG_SUPPORT_SUBMIT_ISSUES.md
    howto/FAQ.md
@@ -34,6 +38,7 @@ Getting Started
 
    advanced_examples/QuantizedLinearRegression.ipynb
    advanced_examples/QuantizedLogisticRegression.ipynb
+   advanced_examples/QuantizedGeneralizedLinearModel.ipynb
 
 .. toctree::
    :maxdepth: 1

docs/user/tutorial/ARITHMETIC_OPERATIONS.md

@@ -1,3 +1,10 @@
+```{warning}
+FIXME(Umut): update a bit, with the new API
+FIXME(Umut/Arthur): update a bit to explain things with the tensors and new operations. At the same time, I think we can exhaustively give examples of every supported functions, since we start to have a lot, so maybe, we would just explain a bit?
+FIXME(all): actually, I am not even sure we should keep this .md, it can't be exhaustive enough, and looks pretty trivial. What do you think
+
+```
+
 # Arithmetic Operations
 
 In this tutorial, we are going to go over all arithmetic operations available in **Concrete**. Please read [Compiling and Executing](../howto/COMPILING_AND_EXECUTING.md) before reading further to see how you can compile the functions below.

docs/user/tutorial/COMPILATION_ARTIFACTS.md

@@ -1,3 +1,7 @@
+```{warning}
+FIXME(Umut): check it is still valid. I guess yes, but some files may have gone or renamed.
+```
+
 # Compilation Artifacts
 
 In this tutorial, we are going to go over the artifact system, which is designed to inspect/debug the compilation process easily. 

docs/user/tutorial/COMPILING_TORCH_MODEL.md

@@ -0,0 +1,4 @@
+```{warning}
+FIXME(jordan): do this section, maybe from one .ipynb that you would do
+```
+# Compiling a Torch Model

docs/user/tutorial/INDEXING.md

@@ -0,0 +1,9 @@
+```{warning}
+FIXME(Umut): to be done
+```
+
+# Indexing
+
+# Slicing
+
+

docs/user/tutorial/MACHINE_LEARNING_TOOLS.md

@@ -0,0 +1,14 @@
+```{warning}
+FIXME(Andrei/Jordan): to be done
+```
+
+# Machine Learning Tools
+
+Give examples with matrix multiplications, reshape, relu, sigmoid, flatten, clip, transpose, dot etc
+
+Do we need to speak about batch norm?
+
+Do we need to speak about sum?
+
+Do we need to speak about pools?
+

docs/user/tutorial/TABLE_LOOKUP.md

@@ -1,3 +1,8 @@
+```{warning}
+FIXME(Umut): update a bit, with the new API
+FIXME(Umut): add the multiTLU case
+```
+
 # Table Lookup
 
 In this tutorial, we are going to go over the ways to perform table lookups in **Concrete**. Please read [Compiling and Executing](../howto/COMPILING_AND_EXECUTING.md) before reading further to see how you can compile the functions below.

docs/user/tutorial/WORKING_WITH_FLOATING_POINTS.md

@@ -1,3 +1,7 @@
+```{warning}
+FIXME(Arthur): update a bit, with the new API
+```
+
 # Working With Floating Points
 
 ## An example
@@ -110,6 +114,10 @@ List of supported binary functions if one of the two operators is a constant sca
 - true_divide
 <!--- gen_supported_ufuncs.py: inject supported operations [END] -->
 
+```{warning}
+FIXME(Benoit): see what kind of other supported operations we could list here
+```
+
 ## Limitations
 
 Floating point support in **Concrete** is very limited for the time being. They can't appear on inputs, or they can't be outputs. However, they can be used in intermediate results. Unfortunately, there are limitations on that front as well.