merge fix conflicts

Former-commit-id: 446daba283ac08db03039e05bc80bb3bd412c809 [formerly b6f7451abcaf9085ede193768b88d1a7d2030c80]
Former-commit-id: dbd9975edab2a234670cb4b2f233aa693533e39a
This commit is contained in:
Raul Jordan
2018-03-05 22:21:50 -06:00
2 changed files with 202 additions and 191 deletions

146
README.md
View File

@@ -1,5 +1,7 @@
Prysmatic Labs Sharding Implementation
==========================
# Prysmatic Labs Sharding Implementation
![Travis Build](https://travis-ci.org/prysmaticlabs/geth-sharding.svg?branch=master)
This is the main repository for the sharding implementation of the go-ethereum client by [Prysmatic Labs](https://prysmaticlabs.com). For the original, go-ethereum project, refer to the following [link](https://github.com/ethereum/go-ethereum).
Before you begin, check out our [Sharding Reference Implementation Doc](https://github.com/prysmaticlabs/geth-sharding/blob/master/sharding/README.md). This doc serves as the single source of truth for our team, our milestones, and details on the different components of our architecture.
@@ -8,32 +10,30 @@ Interested in contributing? Check out our [Contribution Guidelines](#contributio
[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/prysmaticlabs/geth-sharding?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
Table of Contents
=================
# Table of Contents
- [Installation](#installation)
- [Sharding Instructions](#sharding)
- [Running a Local Geth Node](#running-a-local-geth-node)
- [Transaction Generator](#transaction-generator)
- [Becoming a Validator](#becoming-a-validator)
- [Becoming a Proposer](#becoming-a-proposer)
- [Testing](#testing)
- [Contribution Guidelines](#contribution-guidelines)
- [License](#license)
- [Installation](#installation)
- [Sharding Instructions](#sharding)
- [Running a Local Geth Node](#running-a-local-geth-node)
- [Transaction Generator](#transaction-generator)
- [Becoming a Validator](#becoming-a-validator)
- [Becoming a Collation Proposer](#becoming-a-collation-proposer)
- [Testing](#testing)
- [Contribution Guidelines](#contribution-guidelines)
- [License](#license)
Installation
============
# Installation
Create a folder in your `$GOPATH` and navigate to it
```
mkdir -p $GOPATH/src/github.com/ethereum && cd $GOPATH/src/github.com/ethereum
$ mkdir -p $GOPATH/src/github.com/ethereum && cd $GOPATH/src/github.com/ethereum
```
Clone our repository as `go-ethereum`
```
git clone https://github.com/prysmaticlabs/geth-sharding ./go-ethereum
$ git clone https://github.com/prysmaticlabs/geth-sharding ./go-ethereum
```
For prerequisites and detailed build instructions please read the
@@ -45,22 +45,20 @@ You can install them using your favourite package manager.
Once the dependencies are installed, run
```
make geth
$ make geth
```
or, to build the full suite of utilities:
```
make all
$ make all
```
Sharding Instructions
==============
# Sharding Instructions
To get started with running the project, follow the instructions to initialize your own private Ethereum blockchain and geth node, as they will be required to run before you can become a validator or a proposer.
To get started with running the project, follow the instructions to initialize your own private Ethereum blockchain and geth node, as they will be required to run before you can become a validator or a collation proposer.
Running a Local Geth Node
------------------------------
## Running a Local Geth Node
To start a local Geth node, you can create your own `genesis.json` file similar to:
@@ -73,78 +71,72 @@ To start a local Geth node, you can create your own `genesis.json` file similar
"eip158Block": 0
},
"difficulty": "200",
"gasLimit": "210000000000"
"gasLimit": "210000000000",
"alloc": {
"826f3F66dB0416ea82033aE917A611bfBF4D98b6": { "balance": "300000" },
}
}
```
The `alloc` portion specifies account addresses with prefunded ETH when the Ethereum blockchain is created. You can modify this section of the genesis to include your own test address and prefund it with 100ETH.
Then, you can build `geth` and init a new instance of a local, Ethereum blockchain as follows:
```
$ make geth
$ ./build/bin/geth init ./sharding/genesis.json -datadir /path/to/your/datadir
$ ./build/bin/geth --nodiscover console --datadir /path/to/your/datadir --networkid 12345
```
$ make geth
$ ./build/bin/geth init /path/to/genesis.json -datadir /path/to/your/datadir
$ ./build/bin/geth --nodiscover console --datadir /path/to/your/datadir --networkid 12345
It is **important** to note that the `--networkid` flag must match the `chainId` property in the genesis file.
Then, the geth console can start up and you can start a miner as follows:
```
> personal.newAccount()
> miner.setEtherbase(eth.accounts[0])
> miner.start()
```
> personal.newAccount()
> miner.setEtherbase(eth.accounts[0])
> miner.start(1)
Then, once you are satisfied with mining for a few seconds, stop the miner with
Now, save the passphrase you used in the geth node into a text file called password.txt. Then, once you have this private geth node running on your local network, we will need to generate test, pending transactions that can then be processed into shards by validators and collation proposers. For this, we have created an in-house transaction generator CLI tool.
```
> miner.stop()
```
Now, save the passphrase you used in the geth node into a text file called password.txt. Then, once you have this private geth node running on your local network, we will need to generate fake, pending transactions that can then be processed into shards by validators and proposers. For this, we have created an in-house transaction generator CLI tool.
Transaction Generator
---------------------
## Transaction Generator
Work in Progress. To track our current draft of the tx generator cli spec, visit this [link](https://docs.google.com/document/d/1YohsW4R9dIRo0u5RqfNOYjCkYKVCmzjgoBDBYDdu5m0/edit?usp=drive_web&ouid=105756662967435769870).
Once we have fake transactions broadcast to our local node, we can start a validator and proposer client in separate terminal windows to begin the sharding process.
Once we have test transactions broadcast to our local node, we can start a validator and collation proposer client in separate terminal windows to begin the sharding process.
## Becoming a Validator
Work in progress.
Becoming a Validator
-----------------------
To deposit ETH and join as a validator in the Validator Manager Contract, run the following command:
```
geth sharding-validator --deposit --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
```
This will extract 100ETH from your account balance and insert you into the VMC's validator set. Then, the program will listen for incoming block headers and notify you when you have been selected as an eligible proposer for a certain shard in a given period. Once you are selected, the validator will request collations from a "collation proposals pool" that is created by a proposer node. We will need to run a proposal node concurrently in a separate terminal window as follows:
This will extract 100ETH from your account balance and insert you into the VMC's validator set. Then, the program will listen for incoming block headers and notify you when you have been selected as an eligible validator for a certain shard in a given period. Once you are selected, the validator will request collations from a "collation proposals pool" that is created by a collation proposer node. We will need to run a collation proposal node concurrently in a separate terminal window as follows:
Becoming a Proposer
-----------------------
The proposer node can be started with the following command:
## Becoming a Collation Proposer
```
geth sharding-proposer --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
geth sharding-collator --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
```
Proposers are tasked with state execution, so they will process and validate pending transactions in the Geth node and create collations with headers that are then broadcast to a proposals pool along with an ETH deposit.
Collation proposers are tasked with state execution, so they will process and validate pending transactions in the Geth node and create collations with headers that are then broadcast to a proposals pool along with an ETH deposit.
Validators then subscribe to changes in the proposals pool and fetch the collation headers that offer the highest ETH deposit. Once a validator signs this collation, the proposer needs to provide the full collation body and the validator can then append the collation header to the Validator Manager Contract.
Validators then subscribe to changes in the proposals pool and fetch the collation headers that offer the highest ETH deposit. Once a validator signs this collation, the collation proposer needs to provide the full collation body and the validator can then append the collation header to the Validator Manager Contract.
Once this is done, the full, end-to-end sharding example is complete and another iteration can occur.
# Making Changes
Making Changes
==============
## Rebuilding the Validator Manager Contract Bindings
Rebuilding the Validator Manager Contract Bindings
---------------------------------------------------------
The Validator Manager Contract is built in Solidity and deployed to the geth node upon launch of the client if it does not exist in the network at a specified address. If there are any changes to the VMC's code, the Golang bindigs must be rebuilt with the following command.
```
cd ./sharding && go generate
```
go generate github.com/ethereum/go-ethereum/sharding
# OR
cd sharding && go generate
Testing
=======
# Testing
To run the unit tests of our system do:
@@ -154,8 +146,7 @@ go test github.com/ethereum/go-ethereum/sharding
We will require more complex testing scenarios (fuzz tests) to measure the full integrity of the system as it evolves.
Contribution Guidelines
===============
# Contribution Guidelines
Excited by our work and want to get involved in building out our sharding releases? Our [Sharding Reference Implementation Doc](https://github.com/prysmaticlabs/geth-sharding/blob/master/sharding/README.md) has all you need to know in order to begin helping us make this happen. We created this document as a single source of reference for all things related to sharding Ethereum, and we need as much help as we can get!
@@ -163,22 +154,23 @@ You can explore our [Current Projects](https://github.com/prysmaticlabs/geth-sha
**Contribution Steps**
- Create a folder in your `$GOPATH` and navigate to it `mkdir -p $GOPATH/src/github.com/ethereum && cd $GOPATH/src/github.com/ethereum`
- Clone our repository as `go-ethereum`, `git clone https://github.com/prysmaticlabs/geth-sharding ./go-ethereum`
- Fork the `go-ethereum` repository on Github: https://github.com/ethereum/go-ethereum
- Add a remote to your fork
`git remote add YOURNAME https://github.com/YOURNAME/go-ethereum
- Create a folder in your `$GOPATH` and navigate to it `mkdir -p $GOPATH/src/github.com/ethereum && cd $GOPATH/src/github.com/ethereum`
- Clone our repository as `go-ethereum`, `git clone https://github.com/prysmaticlabs/geth-sharding ./go-ethereum`
- Fork the `go-ethereum` repository on Github: <https://github.com/ethereum/go-ethereum>
- Add a remote to your fork
\`git remote add YOURNAME <https://github.com/YOURNAME/go-ethereum>
Now you should have a remote pointing to the `origin` repo (geth-sharding) and to your forked, go-ethereum repo on Github. To commit changes and start a Pull Request, our workflow is as follows:
- Create a new branch with a clear feature name such as `git checkout -b collations-pool`
- Issue changes with clear commit messages
- Push to your remote `git push YOURNAME collations-pool`
- Go to the [geth-sharding](https://github.com/prysmaticlabs/geth-sharding) repository on Github and start a PR comparing `geth-sharding:master` with `go-ethereum:collations-pool` (your fork on your profile).
- Add a clear PR title along with a description of what this PR encompasses, when it can be closed, and what you are currently working on. Github markdown checklists work great for this.
- Create a new branch with a clear feature name such as `git checkout -b collations-pool`
- Issue changes with clear commit messages
- Run the linter and CI tester as follows `go run build/ci.go test && go run build/ci.go lint`
- Push to your remote `git push YOURNAME collations-pool`
- Go to the [geth-sharding](https://github.com/prysmaticlabs/geth-sharding) repository on Github and start a PR comparing `geth-sharding:master` with `go-ethereum:collations-pool` (your fork on your profile).
- Add a clear PR title along with a description of what this PR encompasses, when it can be closed, and what you are currently working on. Github markdown checklists work great for this.
# License
License
=====
The go-ethereum library (i.e. all code outside of the `cmd` directory) is licensed under the
[GNU Lesser General Public License v3.0](https://www.gnu.org/licenses/lgpl-3.0.en.html), also
included in our repository in the `COPYING.LESSER` file.

View File

@@ -4,59 +4,59 @@ This document serves as a main reference for Prysmatic Labs' sharding implementa
# Table of Contents
- [Sharding Introduction](#sharding-introduction)
- [Basic Sharding Idea and Design](#basic-sharding-idea-and-design)
- [Roadmap Phases](#roadmap-phases)
- [The Ruby Release: Local Network](#the-ruby-release-local-network)
- [The Sapphire Release: Ropsten Testnet](#the-sapphire-release-ropsten-testnet)
- [The Diamond Release: Ethereum Mainnet](#the-diamond-release-ethereum-mainnet)
- [Go-Ethereum Sharding Alpha Implementation](#go-ethereum-sharding-alpha-implementation)
- [System Architecture](#system-architecture)
- [System Start and User Entrypoint](#system-start-and-user-entrypoint)
- [The Validator Manager Contract](#the-validator-manager-contract)
- [Necessary Functionality](#necessary-functionality)
- [Depositing ETH and Becoming a Validator](#depositing-eth-and-becoming-a-validator)
- [Determining an Eligible Proposer for a Period on a Shard](#determining-an-eligible-proposer-for-a-period-on-a-shard)
- [Withdrawing From the Validator Set](#withdrawing-from-the-validator-set)
- [Processing and Verifying a Collation Header](#processing-and-verifying-a-collation-header)
- [Validator Sampling](#validator-sampling)
- [Collation Header Approval](#collation-header-approval)
- [Event Logs](#event-logs)
- [The Validator Client](#the-validator-client)
- [Local Shard Storage](#local-shard-storage)
- [The Proposer Client](#the-proposer-client)
- [Collation Headers and State Execution](#collation-headers-and-state-execution)
- [Peer Discovery and Shard Wire Protocol](#peer-discovery-and-shard-wire-protocol)
- [Protocol Modifications](#protocol-modifications)
- [Protocol Primitives: Collations, Blocks, Transactions, Accounts](#protocol-primitives-collations-blocks-transactions-accounts)
- [The EVM: What You Need to Know](#the-evm-what-you-need-to-know)
- [Sharding In-Practice](#sharding-in-practice)
- [Fork Choice Rule](#fork-choice-rule)
- [Use-Case Stories: Proposers](#use-case-stories-proposers)
- [Use-Case Stories: Validators](#use-case-stories-validators)
- [Current Status](#current-status)
- [Security Considerations](#security-considerations)
- [Not Included in Ruby Release](#not-included-in-ruby-release)
- [Bribing, Coordinated Attack Models](#bribing-coordinated-attack-models)
- [Enforced Windback](#enforced-windback)
- [Explicit Finality for Stateless Clients](#explicit-finality-for-stateless-clients)
- [The Data Availability Problem](#the-data-availability-problem)
- [Introduction and Background](#introduction-and-background)
- [On Uniquely Attributable Faults](#on-uniquely-attributable-faults)
- [Erasure Codes](#erasure-codes)
- [Beyond Phase 1](#beyond-phase-1)
- [Cross-Shard Communication](#cross-shard-communication)
- [Receipts Method](#receipts-method)
- [Merge Blocks](#merge-blocks)
- [Synchronous State Execution](#synchronous-state-execution)
- [Transparent Sharding](#transparent-sharding)
- [Tightly-Coupled Sharding (Fork-Free Sharding)](#tightly-coupled-sharding-fork-free-sharding)
- [Active Questions and Research](#active-questions-and-research)
- [Separation of Proposals and Consensus](#separation-of-proposals-and-consensus)
- [Selecting Eligible Validators Off-Chain](#selecting-eligible-validators-off-chain)
- [Community Updates and Contributions](#community-updates-and-contributions)
- [Acknowledgements](#acknowledgements)
- [References](#references)
- [Sharding Introduction](#sharding-introduction)
- [Basic Sharding Idea and Design](#basic-sharding-idea-and-design)
- [Roadmap Phases](#roadmap-phases)
- [The Ruby Release: Local Network](#the-ruby-release-local-network)
- [The Sapphire Release: Ropsten Testnet](#the-sapphire-release-ropsten-testnet)
- [The Diamond Release: Ethereum Mainnet](#the-diamond-release-ethereum-mainnet)
- [Go-Ethereum Sharding Alpha Implementation](#go-ethereum-sharding-alpha-implementation)
- [System Architecture](#system-architecture)
- [System Start and User Entrypoint](#system-start-and-user-entrypoint)
- [The Validator Manager Contract](#the-validator-manager-contract)
- [Necessary Functionality](#necessary-functionality)
- [Depositing ETH and Becoming a Validator](#depositing-eth-and-becoming-a-validator)
- [Determining an Eligible Proposer for a Period on a Shard](#determining-an-eligible-proposer-for-a-period-on-a-shard)
- [Withdrawing From the Validator Set](#withdrawing-from-the-validator-set)
- [Processing and Verifying a Collation Header](#processing-and-verifying-a-collation-header)
- [Validator Sampling](#validator-sampling)
- [Collation Header Approval](#collation-header-approval)
- [Event Logs](#event-logs)
- [The Validator Client](#the-validator-client)
- [Local Shard Storage](#local-shard-storage)
- [The Proposer Client](#the-proposer-client)
- [Collation Headers and State Execution](#collation-headers-and-state-execution)
- [Peer Discovery and Shard Wire Protocol](#peer-discovery-and-shard-wire-protocol)
- [Protocol Modifications](#protocol-modifications)
- [Protocol Primitives: Collations, Blocks, Transactions, Accounts](#protocol-primitives-collations-blocks-transactions-accounts)
- [The EVM: What You Need to Know](#the-evm-what-you-need-to-know)
- [Sharding In-Practice](#sharding-in-practice)
- [Fork Choice Rule](#fork-choice-rule)
- [Use-Case Stories: Proposers](#use-case-stories-proposers)
- [Use-Case Stories: Validators](#use-case-stories-validators)
- [Current Status](#current-status)
- [Security Considerations](#security-considerations)
- [Not Included in Ruby Release](#not-included-in-ruby-release)
- [Bribing, Coordinated Attack Models](#bribing-coordinated-attack-models)
- [Enforced Windback](#enforced-windback)
- [Explicit Finality for Stateless Clients](#explicit-finality-for-stateless-clients)
- [The Data Availability Problem](#the-data-availability-problem)
- [Introduction and Background](#introduction-and-background)
- [On Uniquely Attributable Faults](#on-uniquely-attributable-faults)
- [Erasure Codes](#erasure-codes)
- [Beyond Phase 1](#beyond-phase-1)
- [Cross-Shard Communication](#cross-shard-communication)
- [Receipts Method](#receipts-method)
- [Merge Blocks](#merge-blocks)
- [Synchronous State Execution](#synchronous-state-execution)
- [Transparent Sharding](#transparent-sharding)
- [Tightly-Coupled Sharding (Fork-Free Sharding)](#tightly-coupled-sharding-fork-free-sharding)
- [Active Questions and Research](#active-questions-and-research)
- [Separation of Proposals and Consensus](#separation-of-proposals-and-consensus)
- [Selecting Eligible Validators Off-Chain](#selecting-eligible-validators-off-chain)
- [Community Updates and Contributions](#community-updates-and-contributions)
- [Acknowledgements](#acknowledgements)
- [References](#references)
# Sharding Introduction
@@ -72,9 +72,9 @@ A sharded blockchain system is made possible by having nodes store “signed met
These collations are holistic descriptions of the state and transactions on a certain shard. A collation header contains the following information:
- Information about what shard the collation corresponds to (lets say shard 10)
- Information about the current state of the shard before all transactions are applied
- Information about what the state of the shard will be after all transactions are applied
- Information about what shard the collation corresponds to (lets say shard 10)
- Information about the current state of the shard before all transactions are applied
- Information about what the state of the shard will be after all transactions are applied
For detailed information on protocol primitives including collations, see: [Protocol Primitives](#protocol-primitives). We will have two types of nodes that do the heavy lifting of our sharding logic: **proposers and validators**. The basic role of proposers is to fetch pending transactions from the txpool, execute any state logic or computation, wrap them into collations, and submit them along with an ETH deposit to a **proposals pool**.
@@ -90,8 +90,8 @@ When deciding and signing a proposed, valid collation, collators have the respon
In this new protocol, a block is valid when
- Transactions in all collations are valid
- The state of collations after the transactions is the same as what the collation headers specified
- Transactions in all collations are valid
- The state of collations after the transactions is the same as what the collation headers specified
Given that we are splitting up the global state of the Ethereum blockchain into shards, new types of attacks arise because fewer hash power is required to completely dominate a shard. This is why the **source of randomness** that assigns validators and the fixed period period of time each validator has on a particular shard is critical to ensuring the integrity of the system.
@@ -109,10 +109,10 @@ The idea is that validators will be assigned to propose collations for only a ce
Prysmatic Labs implementation will follow parts of the roadmap outlined by Vitalik in his [Sharding FAQ](https://github.com/ethereum/wiki/wiki/Sharding-FAQ) to roll out a working version of quadratic sharding, with a few modifications on our releases.
1. **Phase 1:** Basic VMC shard system with no cross-shard communication along with a proposer + validator node architecture
2. **Phase 2:** Receipt-based, cross-shard communication
3. **Phase 3:** Require collation headers to be added in as uncles instead of as transactions
4. **Phase 4:** Tightly-coupled sharding with data availability proofs and robust security
1. **Phase 1:** Basic VMC shard system with no cross-shard communication along with a proposer + validator node architecture
2. **Phase 2:** Receipt-based, cross-shard communication
3. **Phase 3:** Require collation headers to be added in as uncles instead of as transactions
4. **Phase 4:** Tightly-coupled sharding with data availability proofs and robust security
To concretize these phases, we will be releasing our implementation of sharding for the geth client as follows:
@@ -120,11 +120,11 @@ To concretize these phases, we will be releasing our implementation of sharding
Our current work is focused on creating a localized version of phase 1, quadratic sharding that would include the following:
- A minimal, **validator client** system that will deploy a **Validator Manager Contract** to a locally running geth node
- Ability to deposit ETH into the validator manager contract through the command line and to be selected as a validator by the local **VMC** in addition to the ability to withdraw the ETH staked
- A **proposer node client** and Cryptoeconomic incentive system for proposer nodes to listen for pending txs, create collations, and submit them along with a deposit to validator nodes in the network
- A simple command line util to **simulate pending transactions** of different types posted to the local geth nodes txpool for the local collation proposer to begin proposing collation headers
- Ability to inspect the shard states and visualize the working system locally through the command line
- A minimal, **validator client** system that will deploy a **Validator Manager Contract** to a locally running geth node
- Ability to deposit ETH into the validator manager contract through the command line and to be selected as a validator by the local **VMC** in addition to the ability to withdraw the ETH staked
- A **proposer node client** and Cryptoeconomic incentive system for proposer nodes to listen for pending txs, create collations, and submit them along with a deposit to validator nodes in the network
- A simple command line util to **simulate pending transactions** of different types posted to the local geth nodes txpool for the local collation proposer to begin proposing collation headers
- Ability to inspect the shard states and visualize the working system locally through the command line
We will forego many of the security considerations that will be critical for testnet and mainnet release for the purposes of demonstration and local network execution as part of the Ruby Release (See: [Security Considerations Not Included in Ruby](#not-included-in-ruby-release)).
@@ -156,27 +156,27 @@ Here is a full reference spec explaining how our initial system will function:
Our implementation revolves around 5 core components:
- A **locally-running geth node** that spins up an instance of the Ethereum blockchain
- A **validator manager contract (VMC)** that is deployed onto this blockchain instance
- A **validator client** that connects to the running geth node through JSON-RPC, provides bindings to the VMC, and listens for incoming collation proposals
- A **proposer client** that is tasked with state execution, processing pending txs from the Geth node, and creates collations that are then broadcast to validators via a wire protocol
- A **user** that will interact with the sharding client to become a validator and automatically process transactions into shards through the sharding clients VMC bindings.
- A **locally-running geth node** that spins up an instance of the Ethereum blockchain
- A **validator manager contract (VMC)** that is deployed onto this blockchain instance
- A **validator client** that connects to the running geth node through JSON-RPC, provides bindings to the VMC, and listens for incoming collation proposals
- A **proposer client** that is tasked with state execution, processing pending txs from the Geth node, and creates collations that are then broadcast to validators via a wire protocol
- A **user** that will interact with the sharding client to become a validator and automatically process transactions into shards through the sharding clients VMC bindings.
Our initial implementation will function through simple command line arguments that will allow a user running the local geth node to deposit ETH into the VMC and join as a validator that is automatically assigned to a certain period. We will also launch a proposer client that will create collations from the geth nodes tx pool and submit them to validators for them to add their headers to the VMC.
A basic, end-to-end example of the system is as follows:
1. _**A User Starts a Validator Client and Deposits 100ETH into the VMC:**_ the sharding client connects to a locally running geth node and asks the user to confirm a deposit from his/her personal account.
1. _**A User Starts a Validator Client and Deposits 100ETH into the VMC:**_ the sharding client connects to a locally running geth node and asks the user to confirm a deposit from his/her personal account.
2. _**Validator Client Connects & Listens to Incoming Headers from the Geth Node and Assigns User as Validator on a Shard per Period:**_ the validator is selected for the current period and must accept collations from proposer nodes that offer the best prices.
2. _**Validator Client Connects & Listens to Incoming Headers from the Geth Node and Assigns User as Validator on a Shard per Period:**_ the validator is selected for the current period and must accept collations from proposer nodes that offer the best prices.
3. _**Concurrently, the Proposer Client Processes and Executes Pending txs from the Geth Node:**_ the proposer client will create valid collations and submit them to validators on an open bidding system.
3. _**Concurrently, the Proposer Client Processes and Executes Pending txs from the Geth Node:**_ the proposer client will create valid collations and submit them to validators on an open bidding system.
4. _**Validators Will Select Collation Proposals that Offer Highest Payout:**_ Validators listen to collation headers on a certain shard with high deposit sizes and sign them.
4. _**Validators Will Select Collation Proposals that Offer Highest Payout:**_ Validators listen to collation headers on a certain shard with high deposit sizes and sign them.
5. _**The Validator Adds Collation Headers Through the VMC:**_ the validator client calls the add_header function in the VMC and to do on-chain verification of the countersigned and accepted collation header immediately.
5. _**The Validator Adds Collation Headers Through the VMC:**_ the validator client calls the add_header function in the VMC and to do on-chain verification of the countersigned and accepted collation header immediately.
6. _**The User Will be Selected as Validator Again on the VMC in a Different Period or Can Withdraw His/Her Stake from the Validator Pool:**_ the user can keep staking and adding incoming collation headers and restart the process, or withdraw his/her stake and be removed from the VMC validator set.
6. _**The User Will be Selected as Validator Again on the VMC in a Different Period or Can Withdraw His/Her Stake from the Validator Pool:**_ the user can keep staking and adding incoming collation headers and restart the process, or withdraw his/her stake and be removed from the VMC validator set.
Now, well explore our architecture and implementation in detail as part of the go-ethereum repository.
@@ -184,33 +184,41 @@ Now, well explore our architecture and implementation in detail as part of th
Our Ruby Release will require users to start a local geth node running a localized, private blockchain to deploy the **VMC** into. Then, users can spin up a validator client as a command line entrypoint into geth while the node is running as follows:
<<<<<<< HEAD
```
$ geth sharding-validator --deposit --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
```
=======
$ geth sharding-validator --deposit 100eth --password ~/Desktop/password.txt
>>>>>>> 5bc1276af89d6f43060be9c1324e7080aaf79d53
If it is the first time the client runs, it will deploy a new **VMC** into the local chain and establish a JSON-RPC connection to interact with the node directly. The `--deposit` flag tells the sharding client to automatically unlock the users keystore and begin depositing ETH into the VMC to become a validator.
If the initial deposit is successful, the sharding client will launch a **local, transaction simulation generator**, which will queue transactions into the txpool for the geth node to process that can then be added into collations on a shard.
Concurrently, a user will need to launch a **proposer client** that will start processing transactions into collations that can then be “sold” to validators by including a cryptographic proof of an eth deposit in their unsign collation headers. This proof is a guarantee of a state change in the validators account balance for accepting to sign the incoming collation header. The proposer client can also be initialized as follows in a separate process:
Concurrently, a user will need to launch a **proposer client** that will start processing transactions into collations that can then be “sold” to validators by including a cryptographic proof of an eth deposit in their unsign collation headers. This proof is a guarantee of a state change in the validators account balance for accepting to sign the incoming collation header. The proposer client (also known as a collator) can also be initialized as follows in a separate process:
<<<<<<< HEAD
```
geth sharding-collator --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
```
=======
geth sharding-collator --password ~/Desktop/password.txt
>>>>>>> 5bc1276af89d6f43060be9c1324e7080aaf79d53
Back to the validators, the validator client will begin its main loop, which involves the following steps:
1. _**Subscribe to Incoming Block Headers:**_ the client will begin by issuing a subscription over JSON-RPC for block headers from the running geth node.
1. _**Subscribe to Incoming Block Headers:**_ the client will begin by issuing a subscription over JSON-RPC for block headers from the running geth node.
2. _**Check Shards for Eligible Proposer:**_ On incoming headers, the client will interact with the VMC to check if the current validator is an eligible validator for an upcoming periods (only a few minutes notice)
2. _**Check Shards for Eligible Proposer:**_ On incoming headers, the client will interact with the VMC to check if the current validator is an eligible validator for an upcoming periods (only a few minutes notice)
3. _**If Validator is Selected, Fetch Proposals from Proposal Nodes and Add Collation Headers to VMC:**_ Once a validator is selected, he/she only has a small timeframe to add collation headers through the VMC, so he/she looks for proposals from proposer nodes and accepts those that offer the highest payouts. The validator then countersigns the collation header, receives the full collation with its body after signing, and adds it to the VMC through PoS consensus.
3. _**If Validator is Selected, Fetch Proposals from Proposal Nodes and Add Collation Headers to VMC:**_ Once a validator is selected, he/she only has a small timeframe to add collation headers through the VMC, so he/she looks for proposals from proposer nodes and accepts those that offer the highest payouts. The validator then countersigns the collation header, receives the full collation with its body after signing, and adds it to the VMC through PoS consensus.
4. _**Supernode Reconciles and Adds to Main Chain:**_ Supernodes that will be responsible for reconciling global state across shards into the main Ethereum blockchain. They are tasked with observing the state across the whole galaxy of shards and adding blocks to the canonical PoW main chain. Proposers get rewarded to their coinbase address for inclusion of a block (also known as a collation subsidy).
4. _**Supernode Reconciles and Adds to Main Chain:**_ Supernodes that will be responsible for reconciling global state across shards into the main Ethereum blockchain. They are tasked with observing the state across the whole galaxy of shards and adding blocks to the canonical PoW main chain. Proposers get rewarded to their coinbase address for inclusion of a block (also known as a collation subsidy).
5. _**If User Withdraws, Remove from Validator Set:**_ A user can choose to stop being a validator and then his/her ETH is withdrawn from the validator set.
5. _**If User Withdraws, Remove from Validator Set:**_ A user can choose to stop being a validator and then his/her ETH is withdrawn from the validator set.
6. _**Otherwise, Validating Client Keeps Subscribing to Block Headers:**_ If the user chooses to keep going,
6. _**Otherwise, Validating Client Keeps Subscribing to Block Headers:**_ If the user chooses to keep going,
It will be the proposer clients responsibility to listen to any new broadcasted transactions to the node and interact with validators that have staked their ETH into the VMC through an open bidding system for collation proposals. Proposer clients are the ones responsible for **state execution** of transactions in the tx pool.
@@ -309,7 +317,6 @@ At that point, the attacker has the ability to conduct 51% attacks against that
However, this problem transcends the sharding scheme itself and goes into the broader problem of fraud detection, which we have yet to comprehensively address.
### Collation Header Approval
Explains the on-chain verification of a collation header.
@@ -328,9 +335,13 @@ The main running thread of our implementation is the validator client, which ser
When we launch the client with
<<<<<<< HEAD
```
geth sharding-validator --deposit --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
```
=======
geth sharding-validator --deposit 100eth --password ~/Desktop/password.txt
>>>>>>> 5bc1276af89d6f43060be9c1324e7080aaf79d53
The instance connects to a running geth node via JSON-RPC and calls the deposit function on a deployed, Validator Manager Contract to insert the user into a validator set. Then, we subscribe for updates on incoming block headers and call `getEligibleProposer` on receiving each header. Once we are selected, our client fetches and “purchases” proposed, unsigned collations from a proposals pool created by proposer nodes. The validator client accepts a collation that offer the highest payout, countersigns it, and adds it to the VMC all within that period.
@@ -346,9 +357,13 @@ In addition to launching a validator client, our system will require a user to c
Users will launch a proposal client as another geth entrypoint as follows:
<<<<<<< HEAD
```
geth sharding-collator --datadir /path/to/your/datadir --password /path/to/your/password.txt --networkid 12345
```
=======
geth sharding-collator --password ~/Desktop/password.txt
>>>>>>> 5bc1276af89d6f43060be9c1324e7080aaf79d53
Launching this command will connect via JSON-RPC to fetch the geth nodes tx pool and see who the currently active validator node is for the period. The proposer is tasked with running transactions to create valid collations and executing their required computations, tracking used gas, and all the heavy lifting that is usually seen in full Ethereum nodes. Once a valid collation is created, the proposer broadcasts the unsigned header **(note: the body is not broadcasted)** to a proposals pool along with a guaranteed ETH deposit that is extracted from the proposers account upfront. Then, the current validator assigned for the period will find proposals for her/her assigned shard and sign the one with the highest payout.
@@ -374,17 +389,17 @@ Work in progress.
As an important aside, well take a brief detour into the EVM and what we need to understand before we modify it for a sharded blockchain. At its core, the functionality of the EVM optimizes for _security_ and not for computational power with the following restrictions:
- Every single step must be paid for upfront with gas costs to prevent DDoS
- Programs can't interact with each other without a single byte array
- This also means programs can't access other programs' state
- Sandboxed Execution - the EVM can only modify its internal state and nothing else
- Deterministic execution guarantees
- Every single step must be paid for upfront with gas costs to prevent DDoS
- Programs can't interact with each other without a single byte array
- This also means programs can't access other programs' state
- Sandboxed Execution - the EVM can only modify its internal state and nothing else
- Deterministic execution guarantees
So what exactly is the EVM? The EVM was purposely designed to be a stack based machine with memory-byte arrays and key-value stores that are kept on merkle trees
- Every single keys and storage values are 32 byte
- There are 100 total opcodes in the EVM
- The EVM comes with a temporary memory byte-array and storage tree to hold persistent memory.
- Every single keys and storage values are 32 byte
- There are 100 total opcodes in the EVM
- The EVM comes with a temporary memory byte-array and storage tree to hold persistent memory.
The only crypto operation in the EVM is the sha3 hash function. Aside from that, the EVM provides a bunch of blockchain access-level context that allows certain opcodes to fetch useful information from the external system. For example, LOG opcodes store useful information in the log bloom filter that can be synced with light clients
@@ -399,23 +414,27 @@ Work in progress.
## Sharding In-Practice
### Fork Choice Rule
In the sharding consensus mechanism, it is important to consider that we now have two layers of longest chain rules when adding a collation. When we are reaching consensus on the best shard chain, we not only have to check for the longest canonical main chain, but also the longest shard chain **within** this longest main chain. Vlad Zamfir has elaborated on this fork-choice rule in a [tweet](https://twitter.com/VladZamfir/status/945358660187893761) that is important for our sharding scheme.
### Use-Case Stories: Proposers
The primary purpose of proposers is to use their computational power for state execution of transactions and create valid collations that can then be put on an open market for validators to take. Upon offering a proposal, proposers will deposit part of their ETH as a payout to the validator that adds its collation header to the VMC, _*even if the collation gets orphaned*_. By forcing proposers to take on this risk, we prevent a certain degree of collation proposal spamming, albeit not without a few other security problems: (See: [Active Research](#active-questions-and-research)).
The primary purpose of proposers is to use their computational power for state execution of transactions and create valid collations that can then be put on an open market for validators to take. Upon offering a proposal, proposers will deposit part of their ETH as a payout to the validator that adds its collation header to the VMC, __even if the collation gets orphaned__. By forcing proposers to take on this risk, we prevent a certain degree of collation proposal spamming, albeit not without a few other security problems: (See: [Active Research](#active-questions-and-research)).
The primary incentive for proposers to generate these collations is to receive a payout to their coinbase address along with transactions fees from the ones they process once added to a block in a canonical chain.
### Use-Case Stories: Validators
The primary purpose of validators is to use Proof of Stake and reach **consensus** on valid shard chains based on the collations they process and add to the Validator Manager Contract. They have two primary options they can choose to do:
- They can deposit ETH into the VMC and become a validator. They then have to wait to be selected by the VMC on a particular period to add a collation header to the VMC.
- They can accept the collation proposals with the highest payouts from the collation pool during their eligible period.
- They can withdraw their stake and stop being a part of the validator pool
- They can deposit ETH into the VMC and become a validator. They then have to wait to be selected by the VMC on a particular period to add a collation header to the VMC.
- They can accept the collation proposals with the highest payouts from the collation pool during their eligible period.
- They can withdraw their stake and stop being a part of the validator pool
The primary incentive for validators is to earn the payouts from the proposers offering them collations within their period.
## Current Status
Currently, Prysmatic Labs is focusing its initial implementation around the logic of the validator and proposer clients. We have built the command line entrypoints as well as the minimum, required functions of the Validator Manager Contract that is deployed to a local Ethereum blockchain instance. Our validator client is able to subscribe for block headers from the running Geth node and determine when we are selected as an eligible proposer in a given period if we have deposited ETH into the contract.
You can track our progress, open issues, and projects in our repository [here](https://github.com/prysmaticlabs/geth-sharding).
@@ -453,7 +472,7 @@ On the other hand, to enforce **availability** for the windback process, a possi
Vitalik has [mentioned](https://ethresear.ch/t/detailed-analysis-of-stateless-client-witness-size-and-gains-from-batching-and-multi-state-roots/862/5?u=rauljordan) that the average amount of windback, or how many immediate periods in the past a validator has to check before adding a collation header, is around 25. In a [medium post](https://medium.com/@icebearhww/ethereum-sharding-and-finality-65248951f649) on the value of explicit finality for sharding, Hsiao-Wei Wang mentions how the finality that Casper FFG provides would mean stateless clients would be entirely confident of blocks ahead to prevent complete reshuffling and faster collation processing. In her own words:
_“Casper FFG will provide explicit finality threshold after about 2.5 “epoch times”, i.e., 125 block times [1] [7]. If validators can verify more than 125 / PERIOD_LENGTH = 25 collations during reshuffling, the shard system can benefit from explicit finality and be more confident of the 25 ahead collations from now are all finalized.”_
_“Casper FFG will provide explicit finality threshold after about 2.5 “epoch times”, i.e., 125 block times [1][7]. If validators can verify more than 125 / PERIOD_LENGTH = 25 collations during reshuffling, the shard system can benefit from explicit finality and be more confident of the 25 ahead collations from now are all finalized.”_
Casper allows us to forego some of these windback considerations and reduces the number of constraints on scalability from a collation header verification standpoint.
@@ -499,7 +518,7 @@ A problem that can occur with current distributed fork-choice ledgers is the pos
Fortunately, there is a potential method of creating a fork-free sharding mechanism that relies on what we are currently implementing through the Validator Manager Contract that has been explored by Justin Drake and Vitalik in [this](https://ethresear.ch/t/fork-free-sharding/1058) and this [other post](https://ethresear.ch/t/a-model-for-stage-4-tightly-coupled-sharding-plus-full-casper/1065), respectively.
The current spec of the Validator Manager Contract _*already does a canonical ordering of collation headers for us*_ (i.e. we can track the timestamped logs of collation headers being added). Because the data for the VMC lives on the canonical main chain, we are able to easily extract and exact ordering and validity from headers added through the contract.
The current spec of the Validator Manager Contract __already does a canonical ordering of collation headers for us__ (i.e. we can track the timestamped logs of collation headers being added). Because the data for the VMC lives on the canonical main chain, we are able to easily extract and exact ordering and validity from headers added through the contract.
To add validity to our current VMC spec, Drake mentions that we can use a succinct zkSNARK in the collation root proving validity upon construction that can be checked directly by the `addHeader` function on the the VMC.
@@ -513,7 +532,7 @@ Work in progress.
## Separation of Proposals and Consensus
In a recent [blog post](https://ethresear.ch/t/separating-proposing-and-confirmation-of-collations/1000/7), Vitalik has outlined a novel system to the sharding mechanism through better separation of concerns. In the current reference documentation for a sharding systems, validators are responsible for proposing transactions into collations and reaching consensus on these proposals. That is, this process happens *all at once*, as proposing a collation happens in tandem with consensus.
In a recent [blog post](https://ethresear.ch/t/separating-proposing-and-confirmation-of-collations/1000/7), Vitalik has outlined a novel system to the sharding mechanism through better separation of concerns. In the current reference documentation for a sharding systems, validators are responsible for proposing transactions into collations and reaching consensus on these proposals. That is, this process happens _all at once_, as proposing a collation happens in tandem with consensus.
This leads to significant computational burdens on validators that need to keep track of the state of a particular shard in the proposal process as part of the transaction packaging process. The potentially better approach outlined above is the idea of separating the transaction packaging process and the consensus mechanism into two separate nodes with different responsibilities. **Our model will be based on this separation and we will be releasing a proposer client alongside a validator client in our Ruby release**.
@@ -525,12 +544,12 @@ A key question asked in the post is whether _“this makes it easy for a propose
It is important to note a possible attack vector in this case: which is that a an attacker could spam proposals on a particular shard and take on the price of excluding certain transactions as a censorship mechanism while the validators would have no idea this is happening. However, given a competitive enough proposal economy, this would be very similar to the current problem of transaction spam in traditional blockchains.
In this system, validators would get paid the proposals deposit _*even if the collation does not get appended to the shard*_. Proposers would have to take on this risk to mitigate the possibilities of malicious intent to make an obfuscated-collation-body proposal system work. Only collations that are double signed and have an available body can be included in the main chain, fee F goes to the validator regardless whether collation gets into the main chain, but fee T only goes to the proposer if the collation gets included to the main chain.
In this system, validators would get paid the proposals deposit __even if the collation does not get appended to the shard__. Proposers would have to take on this risk to mitigate the possibilities of malicious intent to make an obfuscated-collation-body proposal system work. Only collations that are double signed and have an available body can be included in the main chain, fee F goes to the validator regardless whether collation gets into the main chain, but fee T only goes to the proposer if the collation gets included to the main chain.
In practice, we would end up with a system of 3 types of nodes to ensure the functioning of a sharded blockchain
1. Proposer nodes that are tasked with state execution and creation of unsigned collation headers, obfuscated collation bodies, data availability proofs, and an ETH deposit to be relayed to validators.
2. Validator nodes that accept proposals through an open auction system similar to way transactions fees currently work. These nodes then sign these collations and pass them through the VMC for inclusion into a shard through PoS.
1. Proposer nodes that are tasked with state execution and creation of unsigned collation headers, obfuscated collation bodies, data availability proofs, and an ETH deposit to be relayed to validators.
2. Validator nodes that accept proposals through an open auction system similar to way transactions fees currently work. These nodes then sign these collations and pass them through the VMC for inclusion into a shard through PoS.
## Selecting Eligible Validators Off-Chain
@@ -552,19 +571,19 @@ You can explore our [Current Projects](https://github.com/prysmaticlabs/geth-sha
**Contribution Steps**
- Create a folder in your `$GOPATH` and navigate to it `mkdir -p $GOPATH/src/github.com/ethereum && cd $GOPATH/src/github.com/ethereum`
- Clone our repository as `go-ethereum`, `git clone https://github.com/prysmaticlabs/geth-sharding ./go-ethereum`
- Fork the `go-ethereum` repository on Github: https://github.com/ethereum/go-ethereum
- Add a remote to your fork
`git remote add YOURNAME https://github.com/YOURNAME/go-ethereum
- Create a folder in your `$GOPATH` and navigate to it `mkdir -p $GOPATH/src/github.com/ethereum && cd $GOPATH/src/github.com/ethereum`
- Clone our repository as `go-ethereum`, `git clone https://github.com/prysmaticlabs/geth-sharding ./go-ethereum`
- Fork the `go-ethereum` repository on Github: <https://github.com/ethereum/go-ethereum>
- Add a remote to your fork
\`git remote add YOURNAME <https://github.com/YOURNAME/go-ethereum>
Now you should have a remote pointing to the `origin` repo (geth-sharding) and to your forked, go-ethereum repo on Github. To commit changes and start a Pull Request, our workflow is as follows:
- Create a new branch with a clear feature name such as `git checkout -b collations-pool`
- Issue changes with clear commit messages
- Push to your remote `git push YOURNAME collations-pool`
- Go to the [geth-sharding](https://github.com/prysmaticlabs/geth-sharding) repository on Github and start a PR comparing `geth-sharding:master` with `go-ethereum:collations-pool` (your fork on your profile).
- Add a clear PR title along with a description of what this PR encompasses, when it can be closed, and what you are currently working on. Github markdown checklists work great for this.
- Create a new branch with a clear feature name such as `git checkout -b collations-pool`
- Issue changes with clear commit messages
- Push to your remote `git push YOURNAME collations-pool`
- Go to the [geth-sharding](https://github.com/prysmaticlabs/geth-sharding) repository on Github and start a PR comparing `geth-sharding:master` with `go-ethereum:collations-pool` (your fork on your profile).
- Add a clear PR title along with a description of what this PR encompasses, when it can be closed, and what you are currently working on. Github markdown checklists work great for this.
# Acknowledgements