(This article is a deep-dive into the the mechanics and incentives that make Ethereum work. There’s no code, but the logic is fairly thorough – blockchains are complicated!)
Consensus mechanisms are tricky, with a lot of moving parts.
How does proof-of-work “secure the network” in PoW blockchains?
How is it that miners expending energy to solve cryptographic puzzles somehow enables the network to agree on the record of transactions and balances?
Here’s a breakdown of proof-of-work consensus, with Ethereum as our case-study.
Blockchain Consensus Defined
Consensus in a distributed ledger refers to the ability of the nodes in the network to each reach the same conclusion about the “truth”: which transactions happened, and in what order.
Consensus between humans implies some discussion or agreement – but there is no decision-making process between nodes on a PoW blockchain.
Nodes do communicate – they send each other valid transactions and blocks, but each node independently calculates the ledger from the information it has received.
Specifically, it looks for the heaviest valid chain of blocks – the chain with the most cumulative work done on it.
Consensus on blockchains simply means that nodes independently reach the same conclusion – not that they have engaged in some group decision-making process.
Why is Consensus Necessary?
In short: event ordering. Open, decentralized ledgers aim to provide immutable records of transactions and state changes across time. In Bitcoin, this equates to “who owns what” and “who sent what to who”. In Ethereum, it is also “how did the data in storage evolve over time”.
A ledger is only useful if everyone agrees on the order of events – present and future events are strongly conditional on past events.
You can only send funds to Bob if you didn’t already send them all to Alice. In a smart contract, you can only mint tokens if you have already been granted permission – etc. Event ordering matters, and all network participants have to agree on the order.
Consensus Emerges
Consensus is a consequence of the rules of a PoW blockchain protocol. It emerges, as an equilibrium.
There is no explicit voting or decision-making on the network. The protocol rules create an economic incentive structure for network participants at the micro-level – which in turn, yields consensus at the macro level.
Why is Consensus so Hard, Anyway?
A distributed ledger is a set-in-stone, trustworthy record of transactions, that every node has a copy of. Transactions may cause value transfer, or computation – but for the ledger to be useful, it needs to be identical on every node. There must be consensus on ‘the truth’ – the recorded history of what transactions happened, and when.
Consensus is easy when:
- Everyone is incentivized to reach the same conclusion, or
- There is a centralized source of truth, or
- There is some decision-making process that yields agreement
On decentralized peer-to-peer networks, it’s really hard to get anonymous strangers to agree on the state of the ledger – ‘who owns what’, because:
- Everyone has an incentive to order transactions in a ways that benefits themselves
- Centralization isn’t an option – we’re trying to bipass banks, not create one
- Decision-making can be gamed or cheated: bad actors can create many fake accounts, or buy votes. They could profit by steering the vote toward ledger changes that let them spend the same funds twice.
Before the first PoW blockchain (Bitcoin) came along, this problem hadn’t been solved.
Proof-of-Work is Key, But Only One Piece
Proof-of-work solves a crucial piece of the puzzle, but network consensus depends on several interlocking pieces. Proof of-work alone isn’t enough.
Consensus relies on:
- Independent verification of transactions by every full node
- Independent aggregation of transactions into blocks by mining nodes
- A blockchain data structure, whereby proposed blocks contain a reference to a parent block
- Proof-of-work mechanism that makes valid block creation costly
- Independent verification of blocks by every full node, and asssembly of blocks into a chain
- Independent selection of the heaviest chain of valid blocks by every full node
All pieces are essential.
Primer – The Ethereum Network
Ethereum is a software protocol. The Ethereum network is the set of computers/servers running the Ethereum protocol.
Every node on the network runs the same protocol. Every full node holds a complete record of the ‘accepted history’ of all transactions and data stored.
Each node holds a copy of:
- Every transaction ever sent
- The current Ether balance of every account
- The current data storage and code of every contract.
This dataset is Ethereum’s ‘state’ – the most current set of balances and data in storage.
Full nodes mostly agree on state, and hold identical copies. Any “disagreements” get quickly resolved.
So how do nodes reach agreement on the state?
The Mechanics of Consensus
Consensus is enabled by several key properties of the network. Let’s look in turn at these: 1) transaction propagation and validation, 2) block mining and valid blocks, and 3) miner incentives.
1) How Transactions Propagate
Transactions are broadcast to the network, and they propagate according to the wire protocol. Basically: nodes send and receive transactions to and from their neighbours.
Nodes only accept valid transactions. They check every transaction they see against the protocol rules. If the transaction is invalid, they reject it and do not propagate it.
Valid Transactions:
- Provably originate from the sender – they have a valid signature and transaction nonce
- Have a minimum fee attached (‘gas’)
2) Block Mining, and Valid Blocks
Mining nodes also broadcast and receive transactions.
From the set of transactions they’ve received, miners choose transactions to group in to blocks. A block is a list of transactions, and contain some additional info, such as:
- A hash of a parent block
- The sum of the transaction fees for all transactions in the block
- The resultant global state, after all transactions within have been executed
In the block, miners also increment their own account balance by the block reward, R. If the block gets included in the blockchain, the miner will be richer by amount R.
To successfully mine a block, miners try to find a nonce that satisifies this condition:
H(BlockHeader, nonce, DAG ) < Target
Where:
Targetis a numerical threshold- The nonce is some number chosen by the miner
- The DAG is a large dataset, derived from the block number
His a complex one-way hashing function – it hashes the block, the nonce and slices of the DAG.
The salient property of the hashing function H:
It’s hard for miners to find the nonce, but easy for nodes to check if a block’s nonce is valid – i.e. below the target threshold.
The only way to find a valid nonce is brute force – to trial and error many different values computationally. This ‘work’ is performed by GPU-based mining hardware. The computation is energy intensive, and financially costly. Therefore, a valid nonce constitutes a proof-of-work – proof that the miner did a lot of costly work in order to find it.
The target difficulty is adjusted algorithmically by the Ethereum protocol, in order to keep the rate of valid block production roughly constant, and to keep the miners ‘working’ hard.
Once a miner solves a block, they broadcast to the network. The block propagates around the network according to the p2p wire protocol. Every node checks the validity of a block it receives.
If the block is valid, they accept it and propagate it. If not, they reject it.
What Makes a Block Valid?
Nodes consider a block valid if:
- The PoW is valid – the miner has found a valid nonce
- The sum of the gas used by transactions matches the ‘gasUsed’ property of the block
- The state is valid – nodes execute transactions in a block, compute state changes, and check the final state corresponds to the state trie in the block header
- The miner’s account balance increases by the correct reward amount
Every Node Calculates the Heaviest Chain
Nodes running the Ethereum protocol receive many different blocks from their peers, but they always independently calculate their own view of the “heaviest” valid chain – the chain with the most accumulated work done on it.
That means each node forms it’s own view of the blockchain independently, given the blocks it has received. But it uses the same, identical validation rules as every other node to do this.
3) The Key to Consensus: Mining Incentives. Only Valid Blocks Pay Off
The blockchain data structure, the validation rules for transactions and blocks, and the proof-of-work mechanism, combine to create an incentive structure for miners.
The mining incentive structure is the crucial piece that guides the system into equilibrium, where all nodes agree on the Ethereum state.
The incentive structure for miners is thus:
- Submitting a valid block has a chance of earning the miner a reward. The block will propagate, and may be included by nodes in their view of the heaviest chain
- Submitting an invalid block is a waste of $ / energy – nodes will reject it. It has no chance of being included in the heaviest chain.
- Submitting a valid block that isn’t the child of the most recent block in the heaviest chain is likely a waste of $ / energy – nodes won’t include it in their picture of the heaviest chain
These incentives combine – miners can only get their block rewards (and thus earn a profit) if they mine and propagate valid blocks that extend the heaviest chain.
The rules are designed such that it doesn’t pay to cheat.
Honest Mining Leads To Consensus
We’ve seen above how proof-of-work in a blockchain network incentivizes miners to produce valid, current blocks.
This guarantee of honest mining is the key to consensus.
Consensus on the network emerges because:
- Valid blocks are produced at a predictable rate, and have time to propagate around the network
- Every node follows the same rules for block validation and propagation
- Every node follows the same rules for calculating the heaviest chain
- Miners tend to produce honest blocks that they believe will be added to the heaviest chain
Each node will eventually receive all valid blocks, and because they all follow the same validation rules, they will all calculate the same heaviest chain.
PoW mining ensures a guaranteed supply of current, valid blocks – and so the system finds equilibrium at consensus.
Why Miners Can’t Rewrite History
Miners could attempt to cheat and re-mine an older block, in order to ‘rewrite history’ and gain some advantage – e..g. re-spending funds they’ve already spent on goods.
For a miner to have such a block included in the longest chain, they would have to mine the older block and re-mine every subsequent block, since the blockchain structure requires that each block header contains a hash of its parent block.
A miner would need to control the majority of hash power on the network to be able to mine valid blocks faster than the rest of the network, and catch up to the chain tip – the famous 51% attack, or chain re-organization.
Assuming no miner controls 51% hash power, then no miner can ‘alter history’ and replace older blocks.
What Happens to a Ledger without PoW or a Blockchain?
To see the true breakthrough of blockchain-based PoW networks, it’s helpful to look at networks lacking one or more pieces of the puzzle. We’ll see how consensus breaks down without the combination of a blockchain and PoW:
1) Nodes validate transactions – but no blocks, reward, blockchain or PoW
The problem: How is transaction ordering decided? There is no way to reach consensus.
Trransactions benefit some over others – nodes are incentivized to order transactions such that they receive the most benefit (including payments to themselves but not those to competitors, removing ‘spend’ transactions once they’ve received goods, etc)
Maybe we can use some co-ordination, voting, or decision-making mechanism? But then we’re back to the fundamental problem of trust on open, anonymous networks. Bad actors could create millions of accounts, vote for transactions that benefit themselves over others, buy votes, etc. The ledger is worthless.
2) Block production with reward – but no blockchain or PoW
We could imagine a network with randomly chosen block producers who pick valid transactions, put them into blocks and propagate them. Nodes validate blocks.
Problem: Ordering, as above. Consensus isn’t reached, and nodes will order blocks in ways that benefit them. No consensus is reached, and the ledger is worthless.
3) Blockchain and block production – without PoW
The blockchain structure solves the problem of ordering – blocks contain a hash of a parent block, so any given block can only come after exactly one block.
However, without PoW, consensus still isn’t reached.
The problem: with no cost to block production, nodes are incentivized to flood the network with blocks in order to include transactions that favor them. Even with block production limits, bad actors may flood the network with blocks from multiple accounts.
Most detrimentally – miners are able to ‘rewrite history’ too easily – submitting older blocks and re-mining several subsequent blocks without cost. No transaction can be regarded as ‘finalized’ – any can be erased.
With the cost of rewriting history being negligible, nodes are likely to regularly receive blocks that together constitute entirely new and long valid sub-chains. Double-spend attacks are trivially easy, and consensus is impossible.
Why PoW is the Missing Piece to Consensus
- It makes block production regular and predictable
- It randomizes block production across miners – miner have only have a slim chance of ‘winning the race’ and getting any given block accepted. They mostly profit from picking transactions purely according to their fees (attached gas).
- PoW ensure miners only profit by mining valid, current blocks that extend the heaviest chain
- It is hard and unprofitable for miners to try to rewrite history – it’s much easier to get a block accepted to the chain tip, than to re-mine several previous blocks
This means that blocks a few blocks deep from the chain tip can be considered final. The deeper that block A is, the more likely it is that miners are now mining, say, block F – a descendant of block A – and the less likely block A will be replaced.
PoW, combined with the blockchain structure and validation rules, means that nodes receive regular valid blocks, most of which are the direct child of the chain tip. Hence, nodes agree on the heaviest chain, and the total history of all transactions.
The equilibrium of consensus on total history of ledger – all transactions, and their resultant changes to state – is maintained.
With the record of transactions and computation provably ‘set in stone’, Ethereum gives us a trustable ledger with no trusted central authority. This is the value of PoW blockchains.
