From cd855469543648f887f0cfb98fa505971d1ef25f Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 10 Apr 2019 15:09:53 +0100 Subject: [PATCH 01/12] Add option type and null --- specs/simple-serialize.md | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index b78eff93e..1f2c01949 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -30,6 +30,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * `"uintN"`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `"bool"`: `True` or `False` +* '"null"': `Null` ### Composite types @@ -39,6 +40,8 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * angle bracket notation `[type, N]`, e.g. `["uint64", N]` * **list**: ordered variable-length homogenous collection of values * angle bracket notation `[type]`, e.g. `["uint64"]` +* **option**: option type containing one of the given subtypes + * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` We recursively define "variable-size" types to be lists and all types that contains a variable-size type. All other types are said to be "fixed-size". @@ -70,7 +73,13 @@ assert value in (True, False) return b"\x01" if value is True else b"\x00" ``` -### Vectors, containers, lists +### `"null"` + +```python +return b"" +``` + +### Vectors, containers, lists, options If `value` is fixed-size: @@ -87,6 +96,17 @@ serialized_length = len(serialized_bytes).to_bytes(BYTES_PER_LENGTH_PREFIX, "lit return serialized_length + serialized_bytes ``` +If `value` is an option type: + +Define value as an object that has properties `value.value` with the contained value, and `value.type_index` which indexes the type. + +```python +serialized_bytes = serialize(value.value) +serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_PREFIX, "little") +return serialized_length + serialized_bytes +``` + + ## Deserialization Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Efficient algorithms for computing this object can be found in [the implementations](#implementations). @@ -98,6 +118,7 @@ We first define helper functions: * `pack`: Given ordered objects of the same basic type, serialize them, pack them into `BYTES_PER_CHUNK`-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. * `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, if necessary append zero chunks so that the number of chunks is a power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`"uint256"` little-endian serialization) return `hash(root + length)`. +* `mix_in_type`: Given a Merkle root `root` and a type_index `type_index` (`"uint256"` little-endian serialization) return `hash(root + type_index)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: @@ -105,6 +126,7 @@ We now define Merkleization `hash_tree_root(value)` of an object `value` recursi * `mix_in_length(merkleize(pack(value)), len(value))` if `value` is a list of basic objects * `merkleize([hash_tree_root(element) for element in value])` if `value` is a vector of composite objects or a container * `mix_in_length(merkleize([hash_tree_root(element) for element in value]), len(value))` if `value` is a list of composite objects +* `mix_in_type(merkleize(value.value), value.type_index)` if `value` is of option type ## Self-signed containers From 283ba8f7611e3112f1466eac63c0fc2966f94c79 Mon Sep 17 00:00:00 2001 From: jannikluhn Date: Thu, 11 Apr 2019 16:00:53 +0100 Subject: [PATCH 02/12] Update specs/simple-serialize.md Co-Authored-By: dankrad --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 1f2c01949..5b302e2f5 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -103,7 +103,7 @@ Define value as an object that has properties `value.value` with the contained v ```python serialized_bytes = serialize(value.value) serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_PREFIX, "little") -return serialized_length + serialized_bytes +return serialized_type_index + serialized_bytes ``` From 2017ce96149306721bfd8d18553f531594b937b5 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Thu, 11 Apr 2019 16:05:16 +0100 Subject: [PATCH 03/12] Rename "option" -> "union"; "null" only in unions --- specs/simple-serialize.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 5b302e2f5..8a8c78446 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -30,7 +30,9 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * `"uintN"`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `"bool"`: `True` or `False` -* '"null"': `Null` +* `"null"`: `None` + +The type `"null"` is only legal as one of several type in a `union` type. ### Composite types @@ -40,7 +42,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * angle bracket notation `[type, N]`, e.g. `["uint64", N]` * **list**: ordered variable-length homogenous collection of values * angle bracket notation `[type]`, e.g. `["uint64"]` -* **option**: option type containing one of the given subtypes +* **union**: union type containing one of the given subtypes * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` We recursively define "variable-size" types to be lists and all types that contains a variable-size type. All other types are said to be "fixed-size". @@ -79,7 +81,7 @@ return b"\x01" if value is True else b"\x00" return b"" ``` -### Vectors, containers, lists, options +### Vectors, containers, lists, unions If `value` is fixed-size: @@ -96,7 +98,7 @@ serialized_length = len(serialized_bytes).to_bytes(BYTES_PER_LENGTH_PREFIX, "lit return serialized_length + serialized_bytes ``` -If `value` is an option type: +If `value` is an union type: Define value as an object that has properties `value.value` with the contained value, and `value.type_index` which indexes the type. @@ -126,7 +128,7 @@ We now define Merkleization `hash_tree_root(value)` of an object `value` recursi * `mix_in_length(merkleize(pack(value)), len(value))` if `value` is a list of basic objects * `merkleize([hash_tree_root(element) for element in value])` if `value` is a vector of composite objects or a container * `mix_in_length(merkleize([hash_tree_root(element) for element in value]), len(value))` if `value` is a list of composite objects -* `mix_in_type(merkleize(value.value), value.type_index)` if `value` is of option type +* `mix_in_type(merkleize(value.value), value.type_index)` if `value` is of union type ## Self-signed containers From 63bdf95e79e2dcced7e2f5ba3945700972a5ef9d Mon Sep 17 00:00:00 2001 From: dankrad Date: Sat, 20 Apr 2019 11:46:31 +0100 Subject: [PATCH 04/12] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 8a8c78446..c9f845873 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -45,7 +45,7 @@ The type `"null"` is only legal as one of several type in a `union` type. * **union**: union type containing one of the given subtypes * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` -We recursively define "variable-size" types to be lists and all types that contains a variable-size type. All other types are said to be "fixed-size". +We recursively define "variable-size" types to be lists and unions and all types that contain a variable-size type. All other types are said to be "fixed-size". ### Aliases From a481a4e96cf29d13dd19fef33172d0cefa390e04 Mon Sep 17 00:00:00 2001 From: Justin Date: Sat, 20 Apr 2019 20:57:50 +1000 Subject: [PATCH 05/12] Update simple-serialize.md --- specs/simple-serialize.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index c9f845873..2d9b715b3 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -32,7 +32,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * `"bool"`: `True` or `False` * `"null"`: `None` -The type `"null"` is only legal as one of several type in a `union` type. +The `"null"` type is only legal as a union sub-type. ### Composite types @@ -108,7 +108,6 @@ serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_PREFIX, "litt return serialized_type_index + serialized_bytes ``` - ## Deserialization Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Efficient algorithms for computing this object can be found in [the implementations](#implementations). From 101449e71a3492ff97c649c706866a0da2ec6962 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Sat, 27 Apr 2019 21:00:50 +0100 Subject: [PATCH 06/12] Define null as alias of {} --- specs/simple-serialize.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index abef5c669..6858561e8 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -8,12 +8,13 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Typing](#typing) - [Basic types](#basic-types) - [Composite types](#composite-types) + - [Illegal empty composites](#illegal-empty-composites) - [Aliases](#aliases) - [Default values](#default-values) - [Serialization](#serialization) - [`"uintN"`](#uintn) - [`"bool"`](#bool) - - [Containers, vectors, lists](#containers-vectors-lists) + - [Vectors, containers, lists, unions](#vectors-containers-lists-unions) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Self-signed containers](#self-signed-containers) @@ -32,9 +33,6 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * `"uintN"`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `"bool"`: `True` or `False` -* `"null"`: `None` - -The `"null"` type is only legal as a union sub-type. ### Composite types @@ -49,6 +47,10 @@ The `"null"` type is only legal as a union sub-type. We recursively define "variable-size" types to be lists and unions and all types that contain a variable-size type. All other types are said to be "fixed-size". +#### Illegal empty composites + +The empty container `{}` (except as the `"null"` type inside a union, see below) and the empty fixed length list `[type, 0]` are **not** legal types. + ### Aliases For convenience we alias: @@ -56,6 +58,9 @@ For convenience we alias: * `"byte"` to `"uint8"` (this is a basic type) * `"bytes"` to `["byte"]` (this is *not* a basic type) * `"bytesN"` to `["byte", N]` (this is *not* a basic type) +* `"null"`: `{}`, i.e. the empty container + +The `"null"` type is only legal as a union sub-type. ### Default values From 4a483309a544dcd24095e5bb9d5a05b832e97b3b Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 1 May 2019 12:39:07 +0200 Subject: [PATCH 07/12] Update specs/simple-serialize.md Co-Authored-By: dankrad --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 6858561e8..79ed0ef7b 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -113,7 +113,7 @@ fixed_parts = [part if part != None else variable_offsets[i] for i, part in enum return b"".join(fixed_parts + variable_parts) ``` -If `value` is an union type: +If `value` is a union type: Define value as an object that has properties `value.value` with the contained value, and `value.type_index` which indexes the type. From d0447022cb4982e544abdea22e616e2cc7c44444 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 1 May 2019 12:39:24 +0200 Subject: [PATCH 08/12] Update specs/simple-serialize.md Co-Authored-By: dankrad --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 79ed0ef7b..0e0468eb7 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -119,7 +119,7 @@ Define value as an object that has properties `value.value` with the contained v ```python serialized_bytes = serialize(value.value) -serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_PREFIX, "little") +serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_OFFSET, "little") return serialized_type_index + serialized_bytes ``` From cc22432bb91626d48ba62e786da72891f60424de Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 1 May 2019 12:39:44 +0200 Subject: [PATCH 09/12] Update specs/simple-serialize.md Co-Authored-By: dankrad --- specs/simple-serialize.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 0e0468eb7..f6b5cb232 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -45,6 +45,8 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * **union**: union type containing one of the given subtypes * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` +#### Variable-size and fixed-size + We recursively define "variable-size" types to be lists and unions and all types that contain a variable-size type. All other types are said to be "fixed-size". #### Illegal empty composites From a33ee00239f0ecd1f8bb557372844250431d0f68 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 1 May 2019 13:52:37 +0100 Subject: [PATCH 10/12] Update simple-serialize.md --- specs/simple-serialize.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index f6b5cb232..b7a1375e4 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -8,9 +8,10 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Typing](#typing) - [Basic types](#basic-types) - [Composite types](#composite-types) - - [Illegal empty composites](#illegal-empty-composites) + - [Variable-size and fixed-size](#variable-size-and-fixed-size) - [Aliases](#aliases) - [Default values](#default-values) + - [Illegal types](#illegal-types) - [Serialization](#serialization) - [`"uintN"`](#uintn) - [`"bool"`](#bool) @@ -45,14 +46,10 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * **union**: union type containing one of the given subtypes * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` -#### Variable-size and fixed-size +### Variable-size and fixed-size We recursively define "variable-size" types to be lists and unions and all types that contain a variable-size type. All other types are said to be "fixed-size". -#### Illegal empty composites - -The empty container `{}` (except as the `"null"` type inside a union, see below) and the empty fixed length list `[type, 0]` are **not** legal types. - ### Aliases For convenience we alias: @@ -62,19 +59,20 @@ For convenience we alias: * `"bytesN"` to `["byte", N]` (this is *not* a basic type) * `"null"`: `{}`, i.e. the empty container -The `"null"` type is only legal as a union sub-type. - ### Default values The default value of a type upon initialization is recursively defined using `0` for `"uintN"`, `False` for `"bool"`, and `[]` for lists. +### Illegal types + +Empty vector types (i.e. `[subtype, 0]` for some `subtype`) are not legal. The `"null"` type is only legal as a union subtype. + ## Serialization We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `"bytes"`. > *Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signing_root`, `is_variable_size`, etc.) objects implicitly carry their type. - ### `"uintN"` ```python From 563df146b9073da8f91b6ad200cd19273ea8f2ac Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 1 May 2019 13:55:02 +0100 Subject: [PATCH 11/12] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index b7a1375e4..21a87a6f9 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -44,7 +44,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * **list**: ordered variable-length homogeneous collection of values * angle bracket notation `[type]`, e.g. `["uint64"]` * **union**: union type containing one of the given subtypes - * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` + * round bracket notation `(type_1, type_2, ...)`, e.g. `("uint64", "null")` ### Variable-size and fixed-size From b1930d22394911c9549f479adb92f62ec41b0958 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 1 May 2019 15:12:49 +0100 Subject: [PATCH 12/12] Union default values --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 21a87a6f9..6adc9c4b8 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -61,11 +61,11 @@ For convenience we alias: ### Default values -The default value of a type upon initialization is recursively defined using `0` for `"uintN"`, `False` for `"bool"`, and `[]` for lists. +The default value of a type upon initialization is recursively defined using `0` for `"uintN"`, `False` for `"bool"`, and `[]` for lists. Unions default to the first type in the union (with type index zero), which is `"null"` if present in the union. ### Illegal types -Empty vector types (i.e. `[subtype, 0]` for some `subtype`) are not legal. The `"null"` type is only legal as a union subtype. +Empty vector types (i.e. `[subtype, 0]` for some `subtype`) are not legal. The `"null"` type is only legal as the first type in a union subtype (i.e., with type index zero). ## Serialization