github/concrete
1
1
Fork 0
mirror of https://github.com/zama-ai/concrete.git synced 2026-01-10 21:38:00 -05:00
Code Issues Packages Projects Releases Wiki Activity

docs(frontend): add Inventory Matching System tutorial

Browse Source
This commit is contained in:
aquint-zama
2024-07-11 11:42:26 +02:00
committed by Benoit Chevallier-Mames
parent 5a2eea029a
commit 12d79cc0de
4 changed files with 490 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/tutorials/see-all-tutorials.md
View File

@@ -16,6 +16,7 @@
* [XOR distance](../../frontends/concrete-python/examples/xor_distance/README.md)
* [SHA1 with Modules](../../frontends/concrete-python/examples/sha1/README.md)
* [Levenshtein distance with Modules](../../frontends/concrete-python/examples/levenshtein_distance/README.md)
* [Inventory Matching System](../../frontends/concrete-python/examples/prime-match/README.md)
#### Blog tutorials

58
frontends/concrete-python/examples/prime-match/README.md Normal file
View File

@@ -0,0 +1,58 @@
# Inventory Matching System
## Introduction
This tutorial implements two variants of the Inventory Matching System described in this [paper](https://eprint.iacr.org/2023/400.pdf):
`prime-match.py` implements the classical protocol, while `prime-match-semi-honest.py` follows the semi-honest protocol.
The principle is as follows: a bank has a list of orders, and a client has another list of orders. Orders are either `Sell` or `Buy`, followed by an asset. We want to apply the matching between the orders, without the parties to know what are each other orders.
A simple example is
```
Bank Orders:
Sell 10 of C
Buy 47 of A
Sell 31 of D
Client Orders:
Sell 50 of A
Sell 24 of B
Buy 18 of D
```
The corresponding resolution is
```
Bank Orders:
Sell 0 of C
Buy 47 of A
Sell 18 of D
Client Orders:
Sell 47 of A
Sell 24 of B
Buy 18 of D
```
## Executing the classic protocol
We can run our `prime-match.py` to perform the computations: `FHE Simulation` is done in the clear to build expected results, while `FHE` is the real FHE computation. Our execution here was done on an `hpc7a` machine on AWS, with Concrete FIXME.
```
$ python prime-match.py
FIXME: run that on hpc7a machine
```
## Executing the semi honest protocol
We have executed the semi-honest protocol, still on an `hpc7a` machine on AWS, with Concrete FIXME.
```
$ python prime-match-semi-honest.py
FIXME: run that on hpc7a machine
```

123
frontends/concrete-python/examples/prime-match/prime-match-semi-honest.py Normal file
View File

@@ -0,0 +1,123 @@
import random
import time
import numpy as np
from concrete import fhe
# ruff: noqa:S311
# Users can change these settings
NUMBER_OF_SYMBOLS = 10
MINIMUM_ORDER_QUANTITY = 1
MAXIMUM_ORDER_QUANTITY = 50
assert 0 < MINIMUM_ORDER_QUANTITY < MAXIMUM_ORDER_QUANTITY
def prime_match(
bank_order_quantities,
client_order_quantities,
):
with fhe.tag("calculating-matching-order-quantity"):
return np.minimum(bank_order_quantities, client_order_quantities)
inputset = [
(
np.random.randint(
MINIMUM_ORDER_QUANTITY, MAXIMUM_ORDER_QUANTITY, size=(NUMBER_OF_SYMBOLS * 2,)
),
np.random.randint(
MINIMUM_ORDER_QUANTITY, MAXIMUM_ORDER_QUANTITY, size=(NUMBER_OF_SYMBOLS * 2,)
),
)
for _ in range(1000)
]
configuration = fhe.Configuration(
enable_unsafe_features=True,
use_insecure_key_cache=True,
insecure_key_cache_location=".keys",
fhe_simulation=True,
min_max_strategy_preference=fhe.MinMaxStrategy.ONE_TLU_PROMOTED,
show_progress=True,
progress_tag=True,
)
# Only the quantities are encrypted, on both client and bank sides
compiler = fhe.Compiler(
prime_match,
{
"bank_order_quantities": "encrypted",
"client_order_quantities": "encrypted",
},
)
circuit = compiler.compile(inputset, configuration)
print()
start = time.time()
circuit.keys.generate()
end = time.time()
print(f"Key generation took: {end - start:.3f} seconds")
print()
# Generate random orders
sample_bank_order_quantities = [0] * NUMBER_OF_SYMBOLS * 2
sample_client_order_quantities = [0] * NUMBER_OF_SYMBOLS * 2
for i in range(NUMBER_OF_SYMBOLS):
# Randomly choose between a Sell Client / Buy Bank or a Buy Client / Sell Bank
# but avoir Sell Client / Buy Client
idx = i + random.randint(0, 1) * NUMBER_OF_SYMBOLS
sample_bank_order_quantities[idx] = np.random.randint(
MINIMUM_ORDER_QUANTITY, MAXIMUM_ORDER_QUANTITY
)
sample_client_order_quantities[idx] = np.random.randint(
MINIMUM_ORDER_QUANTITY, MAXIMUM_ORDER_QUANTITY
)
sample_args = (sample_bank_order_quantities, sample_client_order_quantities)
# Perform the matching with FHE simulation
print("FHE Simulation:")
simulated_matches = circuit.simulate(*sample_args)
print()
print("\tResult Orders:")
for c1_order, c2_order, result in zip(
sample_bank_order_quantities, sample_client_order_quantities, simulated_matches
):
print(f"\t\t{c1_order}\t{c2_order}\t->\t{result}")
print()
# Perform the matching in FHE
print("FHE:")
start = time.time()
executed_matches = circuit.encrypt_run_decrypt(*sample_args)
end = time.time()
print()
print("\tResult Orders:")
for c1_order, c2_order, result in zip(
sample_bank_order_quantities, sample_client_order_quantities, executed_matches
):
print(f"\t\t{c1_order}\t{c2_order}\t->\t{result}")
print()
# Check
assert all(simulated_matches == executed_matches), "Error in FHE computation"
# Some information about the complexity of the computations
NUMBER_OF_TRANSACTIONS = NUMBER_OF_SYMBOLS * 2
print(f"Complexity was: {circuit.complexity:.3f}")
print()
print(f"Quantities in [{MINIMUM_ORDER_QUANTITY}, {MAXIMUM_ORDER_QUANTITY}]")
print(f"Nb of transactions: {NUMBER_OF_TRANSACTIONS}")
print(f"Nb of Symbols: {NUMBER_OF_SYMBOLS}")
print(
f"Execution took: {end - start:.3f} seconds, ie "
f"{(end - start) / NUMBER_OF_TRANSACTIONS:.3f} seconds per transaction"
)

308
frontends/concrete-python/examples/prime-match/prime-match.py Normal file
View File

@@ -0,0 +1,308 @@
import random
import time
from enum import IntEnum, auto
from typing import Set
import numpy as np
from concrete import fhe
# ruff: noqa:S311
# Users can change these settings
NUMBER_OF_BANK_ORDERS = 10
NUMBER_OF_CLIENT_ORDERS = 5
MINIMUM_ORDER_QUANTITY = 5
MAXIMUM_ORDER_QUANTITY = 60
assert 0 < MINIMUM_ORDER_QUANTITY < MAXIMUM_ORDER_QUANTITY
class OrderType(IntEnum):
Buy = 0
Sell = auto()
class OrderSymbol(IntEnum):
A = 0
B = auto()
C = auto()
D = auto()
E = auto()
F = auto()
G = auto()
H = auto()
# E741 Ambiguous variable name: `I`
# ruff: noqa:E741
I = auto()
J = auto()
# One can add more order types
# K = auto()
# L = auto()
# M = auto()
# N = auto()
# O = auto()
# P = auto()
# Q = auto()
# R = auto()
# S = auto()
# T = auto()
# U = auto()
# V = auto()
# W = auto()
# X = auto()
# Y = auto()
# Z = auto()
class Order:
def __init__(self, order_type: OrderType, order_symbol: OrderSymbol, order_quantity: int):
self.order_type = order_type
self.order_symbol = order_symbol
self.order_quantity = order_quantity
def __str__(self):
return f"{self.order_type.name:>4} {self.order_quantity:3} of {self.order_symbol.name}"
def __repr__(self):
return f"{repr(self.order_type)} {self.order_quantity} of {repr(self.order_symbol)}"
@staticmethod
def random(skiplist: Set[OrderSymbol]) -> "Order":
order_type = random.choice(list(OrderType))
order_symbol = random.choice(list(set(OrderSymbol) - skiplist))
order_quantity = random.randint(MINIMUM_ORDER_QUANTITY, MAXIMUM_ORDER_QUANTITY)
return Order(order_type, order_symbol, order_quantity)
def prime_match(
bank_order_types,
bank_order_symbols,
bank_order_quantities,
client_order_types,
client_order_symbols,
client_order_quantities,
):
with fhe.tag("comparing-order-types"):
types_differ = bank_order_types.reshape(-1, 1) != client_order_types
with fhe.tag("comparing-order-symbols"):
symbols_match = bank_order_symbols.reshape(-1, 1) == client_order_symbols
with fhe.tag("checking-if-order-can-be-filled"):
can_fill = symbols_match & types_differ
with fhe.tag("calculating-matching-order-quantity"):
matching_quantity = np.minimum(
bank_order_quantities.reshape(-1, 1), client_order_quantities
)
with fhe.tag("calculating-filled-order-quantity"):
filled_quantity = fhe.multivariate(lambda x, y: x * y)(can_fill, matching_quantity)
bank_result = (
bank_order_types,
bank_order_symbols,
np.sum(filled_quantity, axis=1),
)
client_result = (
client_order_types,
client_order_symbols,
np.sum(filled_quantity, axis=0),
)
return *bank_result, *client_result
inputset = [
(
np.random.randint(0, len(OrderType), size=(NUMBER_OF_BANK_ORDERS,)),
np.array(
[int(symbol) for symbol in random.sample(list(OrderSymbol), NUMBER_OF_BANK_ORDERS)]
),
np.random.randint(
MINIMUM_ORDER_QUANTITY,
MAXIMUM_ORDER_QUANTITY + 1,
size=(NUMBER_OF_BANK_ORDERS,),
),
np.random.randint(0, len(OrderType), size=(NUMBER_OF_CLIENT_ORDERS,)),
np.array(
[int(symbol) for symbol in random.sample(list(OrderSymbol), NUMBER_OF_CLIENT_ORDERS)]
),
np.random.randint(
MINIMUM_ORDER_QUANTITY,
MAXIMUM_ORDER_QUANTITY + 1,
size=(NUMBER_OF_CLIENT_ORDERS,),
),
)
for _ in range(1000)
]
# comparison_strategy_preference will be used for == and !=
# bitwise_strategy_preference will be used for &
# multivariate_strategy_preference will be used for the multivariate function
# min_max_strategy_preference will be used for np.minimum
#
# Changing these strategies and compiling with show_mlir=True can be a didactic
# way to understand the impact of the different strategies, on the produced
# MLIR and on speed. Remark that a given strategy may be optimal for given
# input bitwidth and not optimal for other bitwidths (so changing the number of
# order symbols will change the optimal choice). Look into
# http://docs.zama.ai/concrete for more precision about the strategies.
configuration = fhe.Configuration(
enable_unsafe_features=True,
use_insecure_key_cache=True,
insecure_key_cache_location=".keys",
fhe_simulation=True,
comparison_strategy_preference=fhe.ComparisonStrategy.ONE_TLU_PROMOTED,
bitwise_strategy_preference=fhe.BitwiseStrategy.ONE_TLU_PROMOTED,
multivariate_strategy_preference=fhe.MultivariateStrategy.PROMOTED,
min_max_strategy_preference=fhe.MinMaxStrategy.ONE_TLU_PROMOTED,
show_progress=True,
progress_tag=True,
)
# All the data are encrypted: the types, the order, the quantities, on both client and bank sides
compiler = fhe.Compiler(
prime_match,
{
"bank_order_types": "encrypted",
"bank_order_symbols": "encrypted",
"bank_order_quantities": "encrypted",
"client_order_types": "encrypted",
"client_order_symbols": "encrypted",
"client_order_quantities": "encrypted",
},
)
circuit = compiler.compile(inputset, configuration)
print()
start = time.time()
circuit.keys.generate()
end = time.time()
print(f"Key generation took: {end - start:.3f} seconds")
print()
# Generate random orders
sample_bank_orders = []
blacklist: Set[OrderSymbol] = set()
for _ in range(NUMBER_OF_BANK_ORDERS):
order = Order.random(blacklist)
blacklist.add(order.order_symbol)
sample_bank_orders.append(order)
sample_client_orders = []
blacklist: Set[OrderSymbol] = set()
for _ in range(NUMBER_OF_CLIENT_ORDERS):
order = Order.random(blacklist)
blacklist.add(order.order_symbol)
sample_client_orders.append(order)
# Show the sample
print("Sample Input:")
print()
print("\tBank Orders:")
for order in sample_bank_orders:
print(f"\t\t{order}")
print()
print("\tClient Orders:")
for order in sample_client_orders:
print(f"\t\t{order}")
print()
print()
sample_bank_order_types = [int(order.order_type) for order in sample_bank_orders]
sample_bank_order_symbols = [int(order.order_symbol) for order in sample_bank_orders]
sample_bank_order_quantities = [order.order_quantity for order in sample_bank_orders]
sample_client_order_types = [int(order.order_type) for order in sample_client_orders]
sample_client_order_symbols = [int(order.order_symbol) for order in sample_client_orders]
sample_client_order_quantities = [order.order_quantity for order in sample_client_orders]
sample_args = [
sample_bank_order_types,
sample_bank_order_symbols,
sample_bank_order_quantities,
sample_client_order_types,
sample_client_order_symbols,
sample_client_order_quantities,
]
def construct_result(results):
raw_bank_orders = zip(results[0], results[1], results[2])
raw_client_orders = zip(results[3], results[4], results[5])
bank_orders = [
Order(OrderType(raw_order_type), OrderSymbol(raw_order_symbol), order_quantity)
for raw_order_type, raw_order_symbol, order_quantity in raw_bank_orders
]
client_orders = [
Order(OrderType(raw_order_type), OrderSymbol(raw_order_symbol), order_quantity)
for raw_order_type, raw_order_symbol, order_quantity in raw_client_orders
]
return bank_orders, client_orders
# Perform the matching with FHE simulation
print("FHE Simulation:")
simulated_matches = circuit.simulate(*sample_args)
simulated_bank_result, simulated_client_result = construct_result(simulated_matches)
print()
print("\tBank Orders:")
for order in simulated_bank_result:
print(f"\t\t{order}")
print()
print("\tClient Orders:")
for order in simulated_client_result:
print(f"\t\t{order}")
print()
print()
# Perform the matching in FHE
print("FHE:")
start = time.time()
executed_matches = circuit.encrypt_run_decrypt(*sample_args)
end = time.time()
executed_bank_result, executed_client_result = construct_result(executed_matches)
print()
print("\tBank Orders:")
for order in executed_bank_result:
print(f"\t\t{order}")
print()
print("\tClient Orders:")
for order in executed_client_result:
print(f"\t\t{order}")
print()
# Check
assert all(
[all(simulated_matches[i] == executed_matches[i]) for i in range(6)]
), "Error in FHE computation"
# Some information about the complexity of the computations
NUMBER_OF_TRANSACTIONS = NUMBER_OF_BANK_ORDERS * NUMBER_OF_CLIENT_ORDERS
print(f"Complexity was: {circuit.complexity:.3f}")
print()
print(f"Nb of transactions: {NUMBER_OF_TRANSACTIONS}")
print(f"Nb of Symbols: {len(OrderSymbol)}")
print(
f"Execution took: {end - start:.3f} seconds, ie "
f"{(end - start) / NUMBER_OF_TRANSACTIONS:.3f} seconds per transaction"
)