tlsnotary/pagesigner-cli
1
0
Fork 0
mirror of https://github.com/tlsnotary/pagesigner-cli.git synced 2026-01-08 23:47:58 -05:00
Code Issues Packages Projects Releases Wiki Activity

update to work with PageSigner v3

Browse Source
This commit is contained in:
themighty1
2021-12-08 13:00:52 +03:00
parent 3e5be722ac
commit 6b4001526d
23 changed files with 2445 additions and 3320 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.gitmodules vendored Normal file
View File

@@ -0,0 +1,3 @@
[submodule "pagesigner"]
path = pagesigner
url = https://github.com/tlsnotary/pagesigner

31
README Normal file
View File

@@ -0,0 +1,31 @@
pgsg-node.js allows you to run PageSigner with nodejs >= v.15.0
Clone this repo with:
git clone --recurse-submodules https://github.com/tlsnotary/pagesigner-cli
Install dependencies by running inside the pagesigner-cli directory:
npm ci
NB! You must supply the file with HTTP headers with --headers path/to/headers
The most basic headers file looks like this:
GET /r/worldnews/ HTTP/1.1
Host: www.reddit.com <---first /r/n
<---second /r/n
As shown above, there are two carriage returns '/r/n/r/n' after the last header's last letter.
Usage: ./pgsg-node.js <command> [arguments]
where <command> is one of notarize, verify
Examples:
./pgsg-node.js notarize example.com --headers headers.txt
Notarize example.com using HTTP headers from headers.txt
./pgsg-node.js verify imported.pgsg
Verify a Pagesigner session from imported.pgsg. This will create a session directory with the decrypted cleartext and the copy of the pgsg file.

93
README.md
View File

@@ -1,93 +0,0 @@
UPDATE November 2021. pagesigner-browserless is not yet available for PageSigner v3. When it becomes available, this repo will be changed accordingly.
UPDATE October 2020. This repository works only for pre-v2 PageSigner. You will have to set up your own oracle server and modify the values in /src/shared/tlsnotary.ini as well as change the pubkeys in /src/shared/pubkeys.txt. Also the server being notarized must support TLS 1.0 or TLS 1.1.
# Browserless PageSigner #
A set of tools to create and use PageSigner format notarization files.
Three scripts are currently provided: (1) notarize.py (2) auditor.py (3) parse-pgsg.py
* notarize - generate notarizations of webpages
* auditor - check the validity of a pgsg file and extract the html from it
* parse-pgsg - currently, just extracts the DER certificates from the file for verification.
More details on each below.
## 1. notarize.py ##
Intended to be run as a script.
Use the script notarize.py in src/auditee
Usage:
```
python notarize.py [-e headers_file] [-a] www.reddit.com/r/all
```
(no need for https, assumed)
### Important notes on usage ###
* HTTP request headers: -e option allows you to specify an **absolute** file path to a file that
contains a set of http headers in json (not including a GET). Request types other than GET have not yet
been implemented.
* Configure the notary: if necessary, edit the \[Notary\] section of the `src/shared/tlsnotary.ini` config file with
the appropriate server IP and port. Do *not* edit the remaining fields unless you know what you are doing.
Note that currently the settings are for the main tlsnotarygroup1 pagesigner oracle. This should work fine.
Bear in mind that this oracle server rate currently limits on a per-IP basis; for high frequency runs this may cause notarization to fail.
* The -a option: this flag allows you to request a check of the oracle status via AWS queries.
If this check completes successfully (note that it will run *twice*: one for the main oracle server, then again for the signing server which holds the notarizing private key).
The recommendation is to use this check periodically, but not for every run. The reason is (a) because the oracle check takes time (a few seconds)
and also to avoid swamping AWS with queries. For example, in a bash script you could configure this option to be applied to 1 out of 10 queries.
Please see the pagesigner-oracles repo for details on the setup of the oracle server on Amazon AWS.
* Verifying the certificate: the main disadvantage of operating outside the browser is we can't
directly reuse the browser's certificate store. For now, you can find the file pubkey1 in the
session directory, which prints the pubkey of the certificate used in hex format, and manually compare it
with that reported in your browser for the same website. Note, however, that this is a **RSA** certificate,
so you will have to make sure that you connect to the site (in the browser) with an RSA key-exchange ciphersuite.
A less clunky way of achieving this sanity check is sought.
## 2. auditor.py ##
Use this to extract html and examine it, in cases where import into the PageSigner addon is not possible.
Usage and output:
```
python auditor.py myfile.pgsg
Notary pubkey OK
Processing data for server: incapsula.com
Notary signature OK
Commitment hash OK
HTML decryption with correct HMACs OK.
Audit passed! You can read the html at: fullpathtohtml and check the server certificate with the data provided in: fullpathtodomainfile
```
If any of the above checks do not pass, the notarization file is **definitely** invalid.
Note that the oracle's pubkey is embedded into the code; if you don't use the tlsnotarygroup1 oracle, you'll have to change it.
To complete the checking, you should compare the contents of the created `domain_data.txt` file with the certificate public key that your browser shows for that domain.
## 3. parse-pgsg.py ##
For now this is a bare-bones script that merely outputs the certificates found in the pgsg file (which are the same
certificates that were passed over the wire during the audit) into a series of DER encoded files 0.der, 1.der ... and fullcert.der.
Note that 0.der should be the certificate of the site itself.
Usage:
```
python parse-pgsg.py myfile.pgsg
```
If you want to check their contents, it can be easily done with openssl:
```
openssl x509 -in 0.der -inform der -text -noout -fingerprint
```
This will show you cert fingerprints, common name etc, expiration date etc.

2156
package-lock.json generated Normal file
View File

File diff suppressed because it is too large Load Diff

17
package.json Normal file
View File

@@ -0,0 +1,17 @@
{
"type": "module",
"dependencies": {
"@peculiar/webcrypto": "^1.2.3",
"bigint-crypto-utils": "^2.5.3",
"bytestreamjs": "1.0.29",
"cbor-js": "^0.1.0",
"cose-js": "^0.8.2",
"fast-sha256": "^1.3.0",
"libsodium-wrappers-sumo": "^0.7.9",
"node-fetch": "2.6.6",
"pkijs": "^2.2.1",
"simple-js-ec-math": "^2.0.1",
"tweetnacl": "^1.0.3",
"universal-dom-parser": "0.0.3"
}
}

1
pagesigner Submodule

Submodule pagesigner added at fe6a75e41f

237
pgsg-node.js Executable file
View File

@@ -0,0 +1,237 @@
#!/usr/bin/env node
import {int2ba, ba2int, assert, b64decode, b64encode} from './pagesigner/core/utils.js'
import {parse_certs} from './pagesigner/core/verifychain.js';
import {verifyNotary, getURLFetcherDoc} from './pagesigner/core/oracles.js';
import {FirstTimeSetup} from './pagesigner/core/FirstTimeSetup.js';
import {globals} from './pagesigner/core/globals.js';
import * as Path from 'path'
import {fileURLToPath} from 'url'
// __dirname is the directory where we are located
const __dirname = Path.dirname(fileURLToPath(import.meta.url))
// this workaround allows to require() from ES6 modules, which is not allowed by default
import { createRequire } from 'module'
const require = createRequire(import.meta.url)
const pkijs = require("pkijs");
const { Crypto } = require("@peculiar/webcrypto");
const crypto = new Crypto();
global.crypto = crypto;
pkijs.setEngine("newEngine", crypto, new pkijs.CryptoEngine({ name: "", crypto: crypto, subtle: crypto.subtle }))
global.CBOR = require('cbor-js')
import * as COSE from './pagesigner/core/third-party/coseverify.js'
global.COSE = COSE
// replace browser's fetch
import fetch from 'node-fetch'
global.fetch = fetch
const http = require('http');
// keepAliveAgent tells fetch to reuse the same source port
global.keepAliveAgent = new http.Agent({keepAlive: true});
// replace browser's DOMParser
import DOMParser from 'universal-dom-parser'
global.DOMParser = DOMParser
const fs = require('fs')
global.fs = fs
global.sodium= require('libsodium-wrappers-sumo');
global.nacl = require('tweetnacl')
global.ECSimple = require('simple-js-ec-math')
global.bcuNode = require('bigint-crypto-utils')
global.fastsha256 = require('fast-sha256');
// this must be imported dynamically after global.bcuNode becomes available
const TLSNotarySession = (await import('./pagesigner/core/TLSNotarySession.js')).TLSNotarySession;
const Main = (await import('./pagesigner/core/Main.js')).Main;
// always return 0 when browser calls performance.now()
global.performance = {'now':function(){return 0;}};
// when browser asks for a resource URL, return a full path
global.chrome = {'extension':{'getURL': (url) => Path.join(__dirname, 'pagesigner', url)}}
// override some methods of Socket
import {Socket} from './pagesigner/core/Socket.js';
const net = require('net');
class SocketNode extends Socket{
constructor(server, port){
super(server, port)
this.sock = new net.Socket();
const that = this
this.sock.on('data', function(d) {
that.buffer = Buffer.concat([that.buffer, d])
});
}
async connect(){
await this.sock.connect(this.port, this.name);
setTimeout(function() {
if (! this.wasClosed) this.close();
}, this.lifeTime);
return 'ready';
}
async send(d) {
var data = new Buffer.from(d)
this.sock.write(data);
}
async close() {
this.sock.destroy()
}
}
global.SocketNode = SocketNode
// a drop-in replacement for HTML WebWorker
const { Worker, parentPort } = require('worker_threads');
class mWorker extends Worker{
constructor(url){
super(url)
this.onmessage = function(){}
this.on('message', function(msg){
this.onmessage(msg)
})
}
}
global.Worker = mWorker
// original parseAndAssemble is called with relative paths, we convert them into absolute paths
import './pagesigner/core/twopc/circuits/casmbundle.js'
CASM.parseAndAssembleOld = CASM.parseAndAssemble
CASM.parseAndAssemble = function(file){
const fullpath = Path.join(__dirname, 'pagesigner', 'core', 'twopc', 'circuits', file)
return CASM.parseAndAssembleOld(fullpath)
}
async function createNewSession(host, request, response, date, pgsg, is_imported=false){
const suffix = is_imported ? "_imported" : ""
const sessDir = Path.join(__dirname, 'saved_sessions', date + "_" + host + suffix)
fs.mkdirSync(sessDir, { recursive: true });
fs.writeFileSync(Path.join(sessDir, "request"), request)
fs.writeFileSync(Path.join(sessDir, "response"), response)
fs.writeFileSync(Path.join(sessDir, date+".pgsg"), Buffer.from(JSON.stringify(pgsg)))
return sessDir
}
function showUsage(){
console.log("Usage: ./pgsg-node.js <command> [options] \r\n")
console.log("where <command> is one of notarize, verify\r\n")
console.log("Examples:\r\n")
console.log("./pgsg-node.js notarize example.com --headers headers.txt")
console.log("Notarize example.com using HTTP headers from headers.txt\r\n")
console.log("./pgsg-node.js verify imported.pgsg")
console.log("Verify a Pagesigner session from imported.pgsg. This will create a session directory with the decrypted cleartext and a copy of the pgsg file.\r\n")
console.log("\r\n")
process.exit(0)
}
async function setupNotary(){
const m = new Main();
if (globals.useNotaryNoSandbox){
return await m.queryNotaryNoSandbox(globals.defaultNotaryIP);
} else {
const cacheDir = Path.join(__dirname, 'cache')
const tnPath = Path.join(cacheDir, 'trustedNotary')
if (fs.existsSync(tnPath)) {
// load notary from disk
const obj = JSON.parse(fs.readFileSync(tnPath))
obj['URLFetcherDoc'] = b64decode(obj['URLFetcherDoc'])
console.log('Using cached notary from ', tnPath)
console.log('Notary IP address: ', obj.IP);
return obj
} else {
// fetch and verify the URLFetcher doc
const URLFetcherDoc = await getURLFetcherDoc(globals.defaultNotaryIP);
const trustedPubkeyPEM = await verifyNotary(URLFetcherDoc);
assert(trustedPubkeyPEM != undefined);
const obj = {
'IP': globals.defaultNotaryIP,
'pubkeyPEM': trustedPubkeyPEM,
'URLFetcherDoc': URLFetcherDoc
};
// save the notary to disk
const objSave = {
'IP': obj.IP,
'pubkeyPEM': obj.pubkeyPEM,
'URLFetcherDoc': b64encode(obj.URLFetcherDoc)
}
fs.writeFileSync(tnPath, Buffer.from(JSON.stringify(objSave)))
return obj
}
}
}
async function main (){
const argv = process.argv
if (argv[2] === 'notarize') {
if (argv.length !== 6 || (argv.length == 6 && argv[4] !== '--headers')){
showUsage();
}
const cacheDir = Path.join(__dirname, 'cache')
if (! fs.existsSync(cacheDir)) {fs.mkdirSync(cacheDir)};
const psPath = Path.join(cacheDir, 'parsedCircuits')
const gbPath = Path.join(cacheDir, 'gatesBlob')
let circuits
if (fs.existsSync(psPath)) {
// load cached serialized circuits
circuits = JSON.parse(fs.readFileSync(psPath))
} else {
// run first time setup
circuits = await new FirstTimeSetup().start();
for (const [k, v] of Object.entries(circuits)) {
circuits[k]['gatesBlob'] = b64encode(circuits[k]['gatesBlob'])
}
fs.writeFileSync(psPath, Buffer.from(JSON.stringify(circuits)))
}
for (const [k, v] of Object.entries(circuits)) {
circuits[k]['gatesBlob'] = b64decode(circuits[k]['gatesBlob'])
}
// prepare root store certificates
const rootStorePath = Path.join(__dirname, 'pagesigner', 'core', 'third-party', 'certs.txt')
await parse_certs(fs.readFileSync(rootStorePath).toString());
var server = argv[3]
var headersfile = Path.join(__dirname, argv[5])
var headers = fs.readFileSync(headersfile).toString().replace(/\n/g, '\r\n')
const m = new Main();
m.trustedOracle = await setupNotary();
// start the actual notarization
const session = new TLSNotarySession(
server, 443, headers, m.trustedOracle, globals.sessionOptions, circuits, null);
const obj = await session.start();
obj['title'] = 'PageSigner notarization file';
obj['version'] = 6;
if (! globals.useNotaryNoSandbox){
obj['URLFetcher attestation'] = m.trustedOracle.URLFetcherDoc;
}
const [host, request, response, date] = await m.verifyPgsgV6(obj);
const serializedPgsg = m.serializePgsg(obj);
const sessDir = await createNewSession(host, request, response, date, serializedPgsg)
console.log('Session was saved in ', sessDir)
process.exit(0)
}
else if (argv[2] === 'verify') {
if (argv.length !== 4){
showUsage()
}
const pgsgBuf = fs.readFileSync(argv[3])
const serializedPgsg = JSON.parse(pgsgBuf)
const m = new Main();
const pgsg = m.deserializePgsg(serializedPgsg);
// prepare root store certificates
const rootStorePath = Path.join(__dirname, 'pagesigner', 'core', 'third-party', 'certs.txt')
await parse_certs(fs.readFileSync(rootStorePath).toString());
const [host, request, response, date] = await m.verifyPgsgV6(pgsg);
const sessDir = await createNewSession(host, request, response, date, serializedPgsg, true)
console.log('The imported session was verified and saved in ', sessDir)
process.exit(0)
}
else {
showUsage()
}
}
main()

24
src/APIexamples/bfx_README.md
View File

@@ -1,24 +0,0 @@
HOW TO USE:
===========
First, you need an API key and API secret from Bitfinex,
available on the website. These are alphanumeric strings.
Write them as the first two lines in a file anywhere on disk.
Then, run the script as:
python bitfinexAPI.py absolute-path-to-APIkeyfile absolute-path-to-new-headers-file
where the first argument is the full path for the API key file you created earlier,
and the second argument is a new file location in which the headers can be stored
(it can be anywhere, but make sure it's the **absolute** path).
For now, the script will record a proof of the balance of the account owner, which
can be found in `src/auditee/sessions/<timestamp>/commit/html-1`. The data is seen
at the bottom of the file in json.
This script is designed so it can be called from a shell script or similar, perhaps
at regular intervals. Usage is up to your imagination.
Notes
=====
Still primitive. See the TODOs in the Python script, and read the API docs on bitfinex.com.

54
src/APIexamples/bitfinexAPI.py
View File

@@ -1,54 +0,0 @@
from __future__ import print_function
import requests
import json
import base64
import hashlib
import time
import hmac
import sys
from subprocess import check_output
def build_headers(header_file, bfx_data):
payloadObject = {
'request':bfx_data['call'],
#nonce must be strictly increasing
'nonce':str(long(time.time()*1000000)),
'options':{}
}
payload_json = json.dumps(payloadObject)
payload = base64.b64encode(payload_json)
m = hmac.new(bfx_data['secret'], payload, hashlib.sha384)
m = m.hexdigest()
headers = {
'X-BFX-APIKEY' : bfx_data['key'],
'X-BFX-PAYLOAD' : payload,
'X-BFX-SIGNATURE' : m
}
with open(header_file,'wb') as f:
f.write(json.dumps(headers))
def run_pagesigner(header_file, bfx_data):
url = bfx_data['url']+bfx_data['call']
#TODO tidy up file paths
r = check_output(['python','../auditee/notarize.py','-e',header_file,url])
#TODO flag success/failure
print ("PageSigner notarization output: \n", r)
if __name__ == "__main__":
bitfinexURLroot = 'api.bitfinex.com'
#TODO: the first obvious extension is to
#add other API queries other than balance
bitfinexURLAPIcall = '/v1/balances'
if len(sys.argv) != 3:
print ("wrong args, see bfx_README.md")
exit(1)
#TODO use optparse or similar, make input more flexible, including file paths
#and other options
with open(sys.argv[1],'rb') as f:
bitfinexKey = f.readline().rstrip()
bitfinexSecret = f.readline().rstrip()
bfx_data = {'url': bitfinexURLroot, 'call': bitfinexURLAPIcall,
'key':bitfinexKey, 'secret': bitfinexSecret}
build_headers(sys.argv[2], bfx_data)
run_pagesigner(sys.argv[2], bfx_data)

161
src/auditee/auditor.py
View File

@@ -1,161 +0,0 @@
#!/usr/bin/env python
from __future__ import print_function
from base64 import b64decode, b64encode
from hashlib import md5, sha1, sha256
from os.path import join
from oracles import oracle_modulus
import binascii, hmac, os, platform, tarfile
import random, re, sys, time, zipfile
#file system setup.
data_dir = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.dirname(data_dir))
install_dir = os.path.dirname(os.path.dirname(data_dir))
sessions_dir = join(data_dir, 'sessions')
time_str = time.strftime('%d-%b-%Y-%H-%M-%S', time.gmtime())
current_session_dir = join(sessions_dir, time_str)
os.makedirs(current_session_dir)
global_tlsver = bytearray('\x03\x02')
global_use_gzip = True
global_use_slowaes = False
global_use_paillier = False
oracle_ba_modulus = None
oracle_int_modulus = None
def extract_audit_data(audit_filename):
audit_data = {}
with open(audit_filename,'rb') as f:
header = f.read(29)
if header != 'tlsnotary notarization file\n\n':
raise Exception("Invalid file format")
version = f.read(2)
if version != '\x00\x02':
raise Exception("Incompatible file version")
audit_data['cipher_suite'] = shared.ba2int(f.read(2))
audit_data['client_random'] = f.read(32)
audit_data['server_random'] = f.read(32)
audit_data['pms1'] = f.read(24)
audit_data['pms2'] = f.read(24)
audit_data['certs_len'] = shared.ba2int(f.read(3))
audit_data['certs'] = f.read(audit_data['certs_len'])
audit_data['tlsver'] = f.read(2)
audit_data['initial_tlsver'] = f.read(2)
response_len = shared.ba2int(f.read(8))
audit_data['response'] = f.read(response_len)
IV_len = shared.ba2int(f.read(2))
if IV_len not in [258,16]:
print ("IV length was: ", IV_len)
raise Exception("Wrong IV format in audit file")
audit_data['IV'] = f.read(IV_len)
audit_data['oracle_modulus_len'] = f.read(2) #TODO can check this
audit_data['signature'] = f.read(len(oracle_ba_modulus))
audit_data['commit_hash'] = f.read(32)
audit_data['oracle_modulus'] = f.read(512)
if audit_data['oracle_modulus'] != oracle_ba_modulus:
print ("file mod was: ", binascii.hexlify(audit_data['oracle_modulus']))
print ("actual was: ", binascii.hexlify(oracle_ba_modulus))
raise Exception("Unrecognized oracle")
audit_data['audit_time'] = f.read(4)
return audit_data
#unpack and check validity of Python modules
def first_run_check(modname,modhash):
if not modhash: return
mod_dir = join(data_dir, 'python', modname)
if not os.path.exists(mod_dir):
print ('Extracting '+modname + '.tar.gz...')
with open(join(data_dir, 'python', modname+'.tar.gz'), 'rb') as f: tarfile_data = f.read()
if md5(tarfile_data).hexdigest() != modhash:
raise Exception ('Wrong hash')
os.chdir(join(data_dir, 'python'))
tar = tarfile.open(join(data_dir, 'python', modname+'.tar.gz'), 'r:gz')
tar.extractall()
tar.close()
if __name__ == "__main__":
audit_filename = sys.argv[1]
#for md5 hash, see https://pypi.python.org/pypi/<module name>/<module version>
modules_to_load = {'rsa-3.1.4':'b6b1c80e1931d4eba8538fd5d4de1355',\
'pyasn1-0.1.7':'2cbd80fcd4c7b1c82180d3d76fee18c8',\
'slowaes':'','requests-2.3.0':'7449ffdc8ec9ac37bbcd286003c80f00'}
for x,h in modules_to_load.iteritems():
first_run_check(x,h)
sys.path.append(join(data_dir, 'python', x))
import rsa
import pyasn1
import requests
from pyasn1.type import univ
from pyasn1.codec.der import encoder, decoder
from slowaes import AESModeOfOperation
import shared
oracle_ba_modulus = bytearray('').join(map(chr,oracle_modulus))
oracle_int_modulus = shared.ba2int(oracle_ba_modulus)
shared.load_program_config()
if int(shared.config.get("General","gzip_disabled")) == 1:
global_use_gzip = False
audit_data = extract_audit_data(audit_filename)
#1. Verify notary pubkey - done in extract_audit_data
print ('Notary pubkey OK')
#2. Verify signature
#First, extract the cert in DER form from the notarization file
#Then, extract from the cert the modulus and server name (common name field)
#To do this, we need to initialise the TLSNClientSession
audit_session = shared.TLSNClientSession(ccs=audit_data['cipher_suite'],tlsver=audit_data['initial_tlsver'])
first_cert_len = shared.ba2int(audit_data['certs'][:3])
server_mod, server_exp = audit_session.extract_mod_and_exp(certDER=audit_data['certs'][3:3+first_cert_len], sn=True)
print ('Processing data for server: ', audit_session.server_name)
data_to_be_verified = audit_data['commit_hash'] + audit_data['pms2'] + shared.bi2ba(server_mod) + audit_data['audit_time']
data_to_be_verified = sha256(data_to_be_verified).digest()
if not shared.verify_signature(data_to_be_verified, audit_data['signature'],oracle_int_modulus):
print ('Audit FAILED. Signature is not verified.')
exit()
print ('Notary signature OK')
#3. Verify commitment hash.
if not sha256(audit_data['response']).digest() == audit_data['commit_hash']:
print ('Audit FAILED. Commitment hash does not match encrypted server response.')
exit()
print ('Commitment hash OK')
#4 Decrypt html and check for mac errors.
audit_session.unexpected_server_app_data_count = shared.ba2int(audit_data['response'][0])
audit_session.tlsver = audit_data['tlsver']
audit_session.client_random = audit_data['client_random']
audit_session.server_random = audit_data['server_random']
audit_session.pms1 = audit_data['pms1']
audit_session.pms2 = audit_data['pms2']
audit_session.p_auditee = shared.tls_10_prf('master secret'+audit_session.client_random+audit_session.server_random,
first_half=audit_session.pms1)[0]
audit_session.p_auditor = shared.tls_10_prf('master secret'+audit_session.client_random+audit_session.server_random,
second_half=audit_session.pms2)[1]
audit_session.set_master_secret_half()
audit_session.do_key_expansion()
audit_session.store_server_app_data_records(audit_data['response'][1:])
audit_session.IV_after_finished = (map(ord,audit_data['IV'][:256]),ord(audit_data['IV'][256]), \
ord(audit_data['IV'][257])) if audit_data['cipher_suite'] in [4,5] else audit_data['IV']
plaintext, bad_mac = audit_session.process_server_app_data_records(is_for_auditor=True)
if bad_mac:
print ('Audit FAILED. Decrypted data has bad HMACs.')
print ('HTML decryption with correct HMACs OK.')
plaintext = shared.dechunk_http(plaintext)
plaintext = shared.gunzip_http(plaintext)
#5 Display html + success.
with open(join(current_session_dir,'audited.html'),'wb') as f:
f.write(plaintext)
#print out the info about the domain
n_hexlified = binascii.hexlify(shared.bi2ba(server_mod))
#pubkey in the format 09 56 23 ....
n_write = " ".join(n_hexlified[i:i+2] for i in range(0, len(n_hexlified), 2))
with open(join(current_session_dir,'domain_data.txt'), 'wb') as f:
f.write('Server name: '+audit_session.server_name + '\n\n'+'Server pubkey:' + '\n\n' + n_write+'\n')
print ("Audit passed! You can read the html at: ",
join(current_session_dir,'audited.html'),
'and check the server certificate with the data provided in ',
join(current_session_dir,'domain_data.txt'))

412
src/auditee/notarize.py
View File

@@ -1,412 +0,0 @@
#!/usr/bin/env python
from __future__ import print_function
from base64 import b64decode, b64encode
from hashlib import md5, sha1, sha256
from os.path import join
import binascii, hmac, os, platform, tarfile
import Queue, random, re, shutil, signal, sys, time
import SimpleHTTPServer, socket, threading, zipfile
import string, json
from optparse import OptionParser
from oracles import check_oracle, oracle, oracle_modulus
try: import wingdbstub
except: pass
#file system setup.
data_dir = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.dirname(data_dir))
install_dir = os.path.dirname(os.path.dirname(data_dir))
sessions_dir = join(data_dir, 'sessions')
time_str = time.strftime('%d-%b-%Y-%H-%M-%S', time.gmtime())
current_session_dir = join(sessions_dir, time_str)
os.makedirs(current_session_dir)
#Globals
audit_no = 0 #we may be auditing multiple URLs. This var keeps track of how many
#successful audits there were so far and is used to index html files audited.
#Default values from the config file. Will be overridden after configfile is parsed
global_tlsver = bytearray('\x03\x02')
global_use_gzip = True
global_use_slowaes = False
global_use_paillier = False
random_uid = ''
def probe_server_modulus(server):
probe_session = shared.TLSNClientSession(server, tlsver=global_tlsver)
print ('ssl port is: ', probe_session.ssl_port)
tls_sock = shared.create_sock(probe_session.server_name,probe_session.ssl_port)
probe_session.start_handshake(tls_sock)
server_mod, server_exp = probe_session.extract_mod_and_exp()
tls_sock.close()
return shared.bi2ba(server_mod)
def start_audit(server_name, headers, server_modulus):
global global_tlsver
global global_use_gzip
global global_use_slowaes
tlsn_session = shared.TLSNClientSession(server_name, tlsver=global_tlsver)
tlsn_session.server_modulus = shared.ba2int(server_modulus)
tlsn_session.server_mod_length = shared.bi2ba(len(server_modulus))
print ('Preparing encrypted pre-master secret')
prepare_pms(tlsn_session)
for i in range(10):
try:
print ('Performing handshake with server')
tls_sock = shared.create_sock(tlsn_session.server_name,tlsn_session.ssl_port)
tlsn_session.start_handshake(tls_sock)
retval = negotiate_crippled_secrets(tlsn_session, tls_sock)
if not retval == 'success':
raise shared.TLSNSSLError('Failed to negotiate secrets: '+retval)
#TODO: cert checking; how to do it for browserless mode?
#========================================================
#before sending any data to server compare this connection's cert to the
#one which FF already validated earlier
#if sha256(tlsn_session.server_certificate.asn1cert).hexdigest() != certhash:
# raise Exception('Certificate mismatch')
#========================================================
print ('Getting data from server')
response = make_tlsn_request(headers,tlsn_session,tls_sock)
#prefix response with number of to-be-ignored records,
#note: more than 256 unexpected records will cause a failure of audit. Just as well!
response = shared.bi2ba(tlsn_session.unexpected_server_app_data_count,fixed=1) + response
break
except shared.TLSNSSLError:
shared.ssl_dump(tlsn_session)
raise
except Exception as e:
print ('Exception caught while getting data from server, retrying...', e)
if i == 9:
raise Exception('Notarization failed')
continue
global audit_no
audit_no += 1 #we want to increase only after server responded with data
sf = str(audit_no)
commit_hash, pms2, signature, audit_time = commit_session(tlsn_session, response,sf)
with open(join(current_session_dir,'sigfile'+sf),'wb') as f:
f.write(signature)
with open(join(current_session_dir,'commit_hash_pms2_servermod'+sf),'wb') as f:
f.write(commit_hash+pms2+shared.bi2ba(tlsn_session.server_modulus))
msg = sha256(commit_hash+pms2+shared.bi2ba(tlsn_session.server_modulus)+audit_time).digest()
oracle_ba_modulus = bytearray('').join(map(chr,oracle_modulus))
oracle_int_modulus = shared.ba2int(oracle_ba_modulus)
if not shared.verify_signature(msg, signature, oracle_int_modulus):
raise Exception("Notarization FAILED, notary signature invalid.")
print ('Verified OK')
audit_data = 'tlsnotary notarization file\n\n'
audit_data += '\x00\x02' #2 version bytes
audit_data += shared.bi2ba(tlsn_session.chosen_cipher_suite,fixed=2) # 2 bytes
audit_data += tlsn_session.client_random + tlsn_session.server_random # 64 bytes
audit_data += tlsn_session.pms1 + pms2 #48 bytes
audit_data += shared.bi2ba(len(tlsn_session.server_certificate.certs),fixed=3)
audit_data += tlsn_session.server_certificate.certs
audit_data += tlsn_session.tlsver #2 bytes
audit_data += tlsn_session.initial_tlsver #2 bytes
audit_data += shared.bi2ba(len(response),fixed=8) #8 bytes
audit_data += response #note that it includes unexpected pre-request app data, 10s of kB
IV = tlsn_session.IV_after_finished if tlsn_session.chosen_cipher_suite in [47,53] \
else shared.rc4_state_to_bytearray(tlsn_session.IV_after_finished)
audit_data += shared.bi2ba(len(IV),fixed=2) #2 bytes
audit_data += IV #16 bytes or 258 bytes for RC4.
audit_data += shared.bi2ba(len(oracle_ba_modulus),fixed=2)
audit_data += signature #512 bytes RSA PKCS 1 v1.5 padding
audit_data += commit_hash #32 bytes sha256 hash
audit_data += oracle_ba_modulus
audit_data += audit_time
with open(join(current_session_dir,sf+".pgsg"),"wb") as f:
f.write(audit_data)
#for later verification, write out the server modulus in hex
#(e.g. to be compared against what's in browser)
n_hexlified = binascii.hexlify(shared.bi2ba(tlsn_session.server_modulus))
#pubkey in the format 09 56 23 ....
n_write = " ".join(n_hexlified[i:i+2] for i in range(0, len(n_hexlified), 2))
with open(join(current_session_dir,'pubkey'+sf), 'wb') as f: f.write(n_write)
print ("\n\n NOTARIZATION SUCCEEDED. \n ",
"You can pass the file(s) " , join(current_session_dir, "1.pgsg"),
" to an auditor for verification, or import into the PageSigner extension.")
rv = decrypt_html(pms2, tlsn_session, sf)
html_paths = b64encode(rv[1])
return True
#Because there is a 1 in ? chance that the encrypted PMS will contain zero bytes in its
#padding, we first try the encrypted PMS with a reliable site and see if it gets rejected.
def prepare_pms(tlsn_session):
n = shared.bi2ba(tlsn_session.server_modulus)
rs_choice = random.choice(shared.reliable_sites.keys())
for i in range(10): #keep trying until reliable site check succeeds
try:
pms_session = shared.TLSNClientSession(rs_choice,shared.reliable_sites[rs_choice][0], ccs=53, tlsver=global_tlsver)
if not pms_session:
raise Exception("Client session construction failed in prepare_pms")
tls_sock = shared.create_sock(pms_session.server_name,pms_session.ssl_port)
pms_session.start_handshake(tls_sock)
reply = send_and_recv('rcr_rsr_rsname_n',
pms_session.client_random+pms_session.server_random+rs_choice[:5]+n)
if reply[0] != 'success':
raise Exception ('Failed to receive a reply for rcr_rsr_rsname_n:')
if not reply[1]=='rrsapms_rhmac_rsapms':
raise Exception ('bad reply. Expected rrsapms_rhmac_rsapms:')
reply_data = reply[2]
rrsapms2 = reply_data[:256]
pms_session.p_auditor = reply_data[256:304]
rsapms2 = reply_data[304:]
response = pms_session.complete_handshake(tls_sock,rrsapms2)
tls_sock.close()
if not response:
print ("PMS trial failed")
continue
#judge success/fail based on whether a properly encoded
#Change Cipher Spec record is returned by the server (we could
#also check the server finished, but it isn't necessary)
if not response.count(shared.TLSRecord(shared.chcis,f='\x01', tlsver=global_tlsver).serialized):
print ("PMS trial failed, retrying. (",binascii.hexlify(response),")")
continue
tlsn_session.auditee_secret = pms_session.auditee_secret
tlsn_session.auditee_padding_secret = pms_session.auditee_padding_secret
tlsn_session.enc_second_half_pms = shared.ba2int(rsapms2)
tlsn_session.set_enc_first_half_pms()
tlsn_session.set_encrypted_pms()
return
except shared.TLSNSSLError:
shared.ssl_dump(pms_session,fn='preparepms_ssldump')
shared.ssl_dump(tlsn_session)
raise
#except Exception,e:
# print ('Exception caught in prepare_pms, retrying...', e)
# continue
raise Exception ('Could not prepare PMS with ', rs_choice, ' after 10 tries. Please '+\
'double check that you are using a valid public key modulus for this site; '+\
'it may have expired.')
def send_and_recv (cmd, dat,timeout=5):
headers = {'Request':cmd,"Data":b64encode(dat),"UID":random_uid}
url = 'http://'+shared.config.get("Notary","server_name")+":"+shared.config.get("Notary","server_port")
r = requests.head(url,headers=headers)
r_response_headers = r.headers #case insensitive dict
received_cmd, received_dat = (r_response_headers['response'],r_response_headers['data'])
return ('success', received_cmd, b64decode(received_dat))
#reconstruct correct http headers
#for passing to TLSNotary custom ssl session
#TODO not yet implemented in browserless mode; should
#add standard headers, esp. gzip according to prefs
def parse_headers(headers):
header_lines = headers.split('\r\n') #no new line issues; it was constructed like that
server = header_lines[1].split(':')[1].strip()
if not global_use_gzip:
modified_headers = '\r\n'.join([x for x in header_lines if 'gzip' not in x])
else:
modified_headers = '\r\n'.join(header_lines)
return (server,modified_headers)
def negotiate_crippled_secrets(tlsn_session, tls_sock):
'''Negotiate with auditor in order to create valid session keys
(except server mac is garbage as auditor withholds it)'''
assert tlsn_session.handshake_hash_md5
assert tlsn_session.handshake_hash_sha
tlsn_session.set_auditee_secret()
cs_cr_sr_hmacms_verifymd5sha = chr(tlsn_session.chosen_cipher_suite) + tlsn_session.client_random + \
tlsn_session.server_random + tlsn_session.p_auditee[:24] + tlsn_session.handshake_hash_md5 + \
tlsn_session.handshake_hash_sha
reply = send_and_recv('cs_cr_sr_hmacms_verifymd5sha',cs_cr_sr_hmacms_verifymd5sha)
if reply[0] != 'success':
raise Exception ('Failed to receive a reply for cs_cr_sr_hmacms_verifymd5sha:')
if not reply[1]=='hmacms_hmacek_hmacverify':
raise Exception ('bad reply. Expected hmacms_hmacek_hmacverify but got', reply[1])
reply_data = reply[2]
expanded_key_len = shared.tlsn_cipher_suites[tlsn_session.chosen_cipher_suite][-1]
if len(reply_data) != 24+expanded_key_len+12:
raise Exception('unexpected reply length in negotiate_crippled_secrets')
hmacms = reply_data[:24]
hmacek = reply_data[24:24 + expanded_key_len]
hmacverify = reply_data[24 + expanded_key_len:24 + expanded_key_len+12]
tlsn_session.set_master_secret_half(half=2,provided_p_value = hmacms)
tlsn_session.p_master_secret_auditor = hmacek
tlsn_session.do_key_expansion()
tlsn_session.send_client_finished(tls_sock,provided_p_value=hmacverify)
sha_digest2,md5_digest2 = tlsn_session.set_handshake_hashes(server=True)
reply = send_and_recv('verify_md5sha2',md5_digest2+sha_digest2)
if reply[0] != 'success':
raise Exception("Failed to receive a reply for verify_md5sha2")
if not reply[1]=='verify_hmac2':
raise Exception("bad reply. Expected verify_hmac2:")
if not tlsn_session.check_server_ccs_finished(provided_p_value = reply[2]):
raise Exception ("Could not finish handshake with server successfully. Audit aborted")
return 'success'
def make_tlsn_request(headers,tlsn_session,tls_sock):
'''Send TLS request including http headers and receive server response.'''
try:
tlsn_session.build_request(tls_sock,headers)
response = shared.recv_socket(tls_sock) #not handshake flag means we wait on timeout
if not response:
raise Exception ("Received no response to request, cannot continue audit.")
tlsn_session.store_server_app_data_records(response)
except shared.TLSNSSLError:
shared.ssl_dump(tlsn_session)
raise
tls_sock.close()
#we return the full record set, not only the response to our request
return tlsn_session.unexpected_server_app_data_raw + response
def commit_session(tlsn_session,response,sf):
'''Commit the encrypted server response and other data to auditor'''
commit_dir = join(current_session_dir, 'commit')
if not os.path.exists(commit_dir): os.makedirs(commit_dir)
#Serialization of RC4 'IV' requires concatenating the box,x,y elements of the RC4 state tuple
IV = shared.rc4_state_to_bytearray(tlsn_session.IV_after_finished) \
if tlsn_session.chosen_cipher_suite in [4,5] else tlsn_session.IV_after_finished
stuff_to_be_committed = {'response':response,'IV':IV,
'cs':str(tlsn_session.chosen_cipher_suite),
'pms_ee':tlsn_session.pms1,'domain':tlsn_session.server_name,
'certificate.der':tlsn_session.server_certificate.asn1cert,
'origtlsver':tlsn_session.initial_tlsver, 'tlsver':tlsn_session.tlsver}
for k,v in stuff_to_be_committed.iteritems():
with open(join(commit_dir,k+sf),'wb') as f: f.write(v)
commit_hash = sha256(response).digest()
reply = send_and_recv('commit_hash',commit_hash)
#TODO: changed response from webserver
if reply[0] != 'success':
raise Exception ('Failed to receive a reply')
if not reply[1]=='pms2':
raise Exception ('bad reply. Expected pms2')
return (commit_hash, reply[2][:24], reply[2][24:536], reply[2][536:540] )
def decrypt_html(pms2, tlsn_session,sf):
'''Receive correct server mac key and then decrypt server response (html),
(includes authentication of response). Submit resulting html for browser
for display (optionally render by stripping http headers).'''
print ("\nStarting decryption of content, may take a few seconds...")
try:
tlsn_session.auditor_secret = pms2[:tlsn_session.n_auditor_entropy]
tlsn_session.set_auditor_secret()
tlsn_session.set_master_secret_half() #without arguments sets the whole MS
tlsn_session.do_key_expansion() #also resets encryption connection state
except shared.TLSNSSLError:
shared.ssl_dump(tlsn_session)
raise
#either using slowAES or a RC4 ciphersuite
try:
plaintext,bad_mac = tlsn_session.process_server_app_data_records()
except shared.TLSNSSLError:
shared.ssl_dump(tlsn_session)
raise
if bad_mac:
raise Exception("ERROR! Audit not valid! Plaintext is not authenticated.")
return decrypt_html_stage2(plaintext, tlsn_session, sf)
def decrypt_html_stage2(plaintext, tlsn_session, sf):
plaintext = shared.dechunk_http(plaintext)
if global_use_gzip:
plaintext = shared.gunzip_http(plaintext)
#write a session dump for checking even in case of success
with open(join(current_session_dir,'session_dump'+sf),'wb') as f: f.write(tlsn_session.dump())
commit_dir = join(current_session_dir, 'commit')
html_path = join(commit_dir,'html-'+sf)
with open(html_path,'wb') as f: f.write('\xef\xbb\xbf'+plaintext) #see "Byte order mark"
if not int(shared.config.get("General","prevent_render")):
html_path = join(commit_dir,'forbrowser-'+sf+'.html')
with open(html_path,'wb') as f:
f.write('\r\n\r\n'.join(plaintext.split('\r\n\r\n')[1:]))
print ("Decryption complete.")
return ('success',html_path)
def get_headers(hpath):
#assumed in json format
with open(hpath,'rb') as f:
json_headers = json.loads(f.read())
print (json_headers)
headers_as_string = ''
for h in json_headers:
headers_as_string += bytearray(h,'utf-8') +':'+bytearray(json_headers[h],'utf-8')+' \r\n'
return headers_as_string
#unpack and check validity of Python modules
def first_run_check(modname,modhash):
if not modhash: return
mod_dir = join(data_dir, 'python', modname)
if not os.path.exists(mod_dir):
print ('Extracting '+modname + '.tar.gz...')
with open(join(data_dir, 'python', modname+'.tar.gz'), 'rb') as f: tarfile_data = f.read()
if md5(tarfile_data).hexdigest() != modhash:
raise Exception ('Wrong hash')
os.chdir(join(data_dir, 'python'))
tar = tarfile.open(join(data_dir, 'python', modname+'.tar.gz'), 'r:gz')
tar.extractall()
tar.close()
if __name__ == "__main__":
#for md5 hash, see https://pypi.python.org/pypi/<module name>/<module version>
modules_to_load = {'rsa-3.1.4':'b6b1c80e1931d4eba8538fd5d4de1355',\
'pyasn1-0.1.7':'2cbd80fcd4c7b1c82180d3d76fee18c8',\
'slowaes':'','requests-2.3.0':'7449ffdc8ec9ac37bbcd286003c80f00'}
for x,h in modules_to_load.iteritems():
first_run_check(x,h)
sys.path.append(join(data_dir, 'python', x))
import rsa
import pyasn1
import requests
from pyasn1.type import univ
from pyasn1.codec.der import encoder, decoder
from slowaes import AESModeOfOperation
import shared
shared.load_program_config()
shared.import_reliable_sites(join(install_dir,'src','shared'))
random_uid = ''.join(random.choice(string.ascii_lowercase+string.digits) for x in range(10))
#override default config values
if int(shared.config.get("General","tls_11")) == 0:
global_tlsver = bytearray('\x03\x01')
if int(shared.config.get("General","decrypt_with_slowaes")) == 1:
global_use_slowaes = True
if int(shared.config.get("General","gzip_disabled")) == 1:
global_use_gzip = False
if int(shared.config.get("General","use_paillier_scheme")) == 1:
global_use_paillier = True
parser = OptionParser(usage='usage: %prog [options] url',
description='Automated notarization of the response to an https'
+ ' request made to the url \'url\' , with https:// omitted.'
)
parser.add_option('-e', '--header-file', action="store", type="string", dest='header_path',
help='if specified, the path to the file containing the HTTP headers to'
+' be used in the request, in json format.')
parser.add_option('-a', '--aws-query-check', action='store_true', dest='awscheck',
help='if set, %prog will perform a check of the PageSigner AWS oracle to verify it.'
+ 'This takes a few seconds.')
(options, args) = parser.parse_args()
if len(args) != 1:
parser.error('Need a url to notarize')
exit(1)
url_raw = args[0]
if options.awscheck:
check_oracle(oracle)
host = url_raw.split('/')[0]
url = '/'.join(url_raw.split('/')[1:])
print ('using host', host)
server_mod = probe_server_modulus(host)
headers = "GET" + " /" + url + " HTTP/1.1" + "\r\n" + "Host: " + host + "\r\n"
x = get_headers(options.header_path) if options.header_path else ''
headers += x + "\r\n"
if start_audit(host, headers, server_mod):
print ('successfully finished')
exit(0)
else:
print ('failed to complete notarization')
exit(1)

205
src/auditee/oracles.py
View File

@@ -1,205 +0,0 @@
import re
from xml.dom import minidom
import urllib2
from base64 import b64decode,b64encode
snapshotID = 'snap-03bae56722ceec3f0'
imageID = 'ami-1f447c65'
oracle_modulus = [186,187,68,57,92,215,243,62,188,248,16,13,3,29,40,217,208,206,78,13,202,184,82,121,26,51,203,41,169,11,4,102,228,127,110,117,170,48,210,212,160,51,175,246,110,178,43,106,94,255,69,0,217,91,225,7,84,133,193,43,177,254,75,191,109,50,212,190,177,61,64,230,188,105,56,252,40,3,91,190,117,1,52,30,210,137,136,13,216,110,83,21,164,56,248,215,33,159,129,149,85,236,130,194,79,227,184,135,133,61,85,201,243,225,121,233,36,84,207,218,86,68,99,21,150,252,28,220,4,93,81,57,214,94,147,56,234,236,0,178,93,39,48,143,21,120,241,33,73,239,185,255,255,79,112,194,72,226,84,158,182,96,159,33,111,57,212,27,23,133,223,152,101,240,98,181,94,38,147,195,187,245,226,158,11,102,91,91,47,146,178,65,180,73,176,209,32,27,99,183,254,161,115,38,186,31,132,165,252,189,226,72,152,219,177,52,47,178,121,45,30,143,78,142,223,133,112,136,72,165,166,225,18,62,249,119,157,198,68,114,69,199,32,121,201,72,159,13,37,66,160,210,83,163,131,128,54,178,219,5,74,94,214,244,43,123,140,156,192,89,120,211,61,192,76,70,176,122,247,198,21,220,79,212,200,192,88,126,200,115,71,102,66,92,102,60,179,213,125,123,86,195,67,204,71,222,249,46,242,179,11,111,12,158,91,189,215,72,190,15,165,11,102,51,1,91,116,127,31,12,55,193,249,170,15,231,13,189,60,73,8,239,238,18,44,131,78,190,164,46,41,169,139,43,230,105,2,170,231,202,203,126,74,202,172,112,217,194,26,202,140,71,183,45,239,213,254,213,139,27,95,163,172,27,176,189,233,59,181,49,225,220,125,90,182,120,183,236,62,100,170,130,122,202,206,193,77,130,250,167,187,238,39,197,216,183,56,203,72,122,168,64,217,225,8,233,13,164,224,23,255,239,230,44,90,31,149,106,207,28,9,249,154,163,84,231,149,167,59,194,193,41,106,239,30,137,188,78,45,66,30,224,233,181,132,146,106,227,135,229,106,71,168,69,149,167,154,150,106,29,130,114,109,11,66,120,42,128,247,166,248,152,103,131,56,88,37,46,19,240,110,135,15,234,44,39,87,65,232,105,2,163]
oracle = {'name':'tlsnotarygroup5',
"IP":"54.158.251.14",
"port":"10011",
'DI':'https://ec2.us-east-1.amazonaws.com/?AWSAccessKeyId=AKIAIHZGACNJKBHFWOTQ&Action=DescribeInstances&Expires=2025-01-01&InstanceId=i-0858c02ad9a33c579&SignatureMethod=HmacSHA256&SignatureVersion=2&Version=2014-10-01&Signature=AWkxF%2FlBVL%2FBl2WhQC62qGJ80qhL%2B%2B%2FJXvSp8mm5sIg%3D',
'DV':'https://ec2.us-east-1.amazonaws.com/?AWSAccessKeyId=AKIAIHZGACNJKBHFWOTQ&Action=DescribeVolumes&Expires=2025-01-01&SignatureMethod=HmacSHA256&SignatureVersion=2&Version=2014-10-01&VolumeId=vol-056223d4e1ce55d9c&Signature=DCYnV1vNqE3cyTm6bmtNS1idGdBT7DcbeLtZfcm3ljo%3D',
'GCO':'https://ec2.us-east-1.amazonaws.com/?AWSAccessKeyId=AKIAIHZGACNJKBHFWOTQ&Action=GetConsoleOutput&Expires=2025-01-01&InstanceId=i-0858c02ad9a33c579&SignatureMethod=HmacSHA256&SignatureVersion=2&Version=2014-10-01&Signature=I%2F1kp7oSli9GvYrrP5HD52D6nOy7yCq9dowaDomSAOQ%3D',
'GU':'https://iam.amazonaws.com/?AWSAccessKeyId=AKIAIHZGACNJKBHFWOTQ&Action=GetUser&Expires=2025-01-01&SignatureMethod=HmacSHA256&SignatureVersion=2&Version=2010-05-08&Signature=N%2BsdNA6z3QReVsHsf7RV4uZLzS5Pqi0n3QSfqBAMs8o%3D',
'DIA':'https://ec2.us-east-1.amazonaws.com/?AWSAccessKeyId=AKIAIHZGACNJKBHFWOTQ&Action=DescribeInstanceAttribute&Attribute=userData&Expires=2025-01-01&InstanceId=i-0858c02ad9a33c579&SignatureMethod=HmacSHA256&SignatureVersion=2&Version=2014-10-01&Signature=ENM%2Bw9WkB4U4kYDMN6kowJhZenuCEX3c1G7xSuu6GZA%3D',
'instanceId': 'i-0858c02ad9a33c579',
"modulus":[186,187,68,57,92,215,243,62,188,248,16,13,3,29,40,217,208,206,78,13,202,184,82,121,26,51,203,41,169,11,4,102,228,127,110,117,170,48,210,212,160,51,175,246,110,178,43,106,94,255,69,0,217,91,225,7,84,133,193,43,177,254,75,191,109,50,212,190,177,61,64,230,188,105,56,252,40,3,91,190,117,1,52,30,210,137,136,13,216,110,83,21,164,56,248,215,33,159,129,149,85,236,130,194,79,227,184,135,133,61,85,201,243,225,121,233,36,84,207,218,86,68,99,21,150,252,28,220,4,93,81,57,214,94,147,56,234,236,0,178,93,39,48,143,21,120,241,33,73,239,185,255,255,79,112,194,72,226,84,158,182,96,159,33,111,57,212,27,23,133,223,152,101,240,98,181,94,38,147,195,187,245,226,158,11,102,91,91,47,146,178,65,180,73,176,209,32,27,99,183,254,161,115,38,186,31,132,165,252,189,226,72,152,219,177,52,47,178,121,45,30,143,78,142,223,133,112,136,72,165,166,225,18,62,249,119,157,198,68,114,69,199,32,121,201,72,159,13,37,66,160,210,83,163,131,128,54,178,219,5,74,94,214,244,43,123,140,156,192,89,120,211,61,192,76,70,176,122,247,198,21,220,79,212,200,192,88,126,200,115,71,102,66,92,102,60,179,213,125,123,86,195,67,204,71,222,249,46,242,179,11,111,12,158,91,189,215,72,190,15,165,11,102,51,1,91,116,127,31,12,55,193,249,170,15,231,13,189,60,73,8,239,238,18,44,131,78,190,164,46,41,169,139,43,230,105,2,170,231,202,203,126,74,202,172,112,217,194,26,202,140,71,183,45,239,213,254,213,139,27,95,163,172,27,176,189,233,59,181,49,225,220,125,90,182,120,183,236,62,100,170,130,122,202,206,193,77,130,250,167,187,238,39,197,216,183,56,203,72,122,168,64,217,225,8,233,13,164,224,23,255,239,230,44,90,31,149,106,207,28,9,249,154,163,84,231,149,167,59,194,193,41,106,239,30,137,188,78,45,66,30,224,233,181,132,146,106,227,135,229,106,71,168,69,149,167,154,150,106,29,130,114,109,11,66,120,42,128,247,166,248,152,103,131,56,88,37,46,19,240,110,135,15,234,44,39,87,65,232,105,2,163]
}
def get_xhr(url):
xml = urllib2.urlopen(url).read()
return minidom.parseString(xml)
def getChildNodes(x):
return [a for a in x[0].childNodes if not a.nodeName=='#text']
#assuming both events happened on the same day, get the time
#difference between them in seconds
#the time string looks like "2015-04-15T19:00:59.000Z"
def getSecondsDelta (later, sooner):
assert (len(later) == 24)
if (later[:11] != sooner[:11]):
return 999999; #not on the same day
laterTime = later[11:19].split(':')
soonerTime = sooner[11:19].split(':')
laterSecs = int(laterTime[0])*3600+int(laterTime[1])*60+int(laterTime[2])
soonerSecs = int(soonerTime[0])*3600+int(soonerTime[1])*60+int(soonerTime[2])
return laterSecs - soonerSecs
def modulus_from_pubkey(pem_pubkey):
b64_str = ''
lines = pem_pubkey.split('\n')
#omit header and footer lines
for i in range(1,len(lines)-1):
b64_str += lines[i]
der = b64decode(b64_str)
#last 5 bytes are 2 DER bytes and 3 bytes exponent, our pubkey is the preceding 512 bytes
pubkey = der[len(der)-517:len(der)-5]
return pubkey
def checkDescribeInstances(xmlDoc, instanceId, IP):
try:
rs = xmlDoc.getElementsByTagName('reservationSet')
assert rs.length == 1
rs_items = getChildNodes(rs)
assert len(rs_items) == 1
ownerId = rs_items[0].getElementsByTagName('ownerId')[0].firstChild.data
isets = rs_items[0].getElementsByTagName('instancesSet')
assert isets.length == 1
instances = getChildNodes(isets)
assert len(instances) == 1
parent = instances[0]
assert parent.getElementsByTagName('instanceId')[0].firstChild.data == instanceId
assert parent.getElementsByTagName('imageId')[0].firstChild.data == imageID
assert parent.getElementsByTagName('instanceState')[0].getElementsByTagName('name')[0].firstChild.data == 'running'
launchTime = parent.getElementsByTagName('launchTime')[0].firstChild.data
assert parent.getElementsByTagName('ipAddress')[0].firstChild.data == IP
assert parent.getElementsByTagName('rootDeviceType')[0].firstChild.data == 'ebs'
assert parent.getElementsByTagName('rootDeviceName')[0].firstChild.data == '/dev/xvda'
devices = parent.getElementsByTagName('blockDeviceMapping')[0].getElementsByTagName('item')
assert devices.length == 1
assert devices[0].getElementsByTagName('deviceName')[0].firstChild.data == '/dev/xvda'
assert devices[0].getElementsByTagName('ebs')[0].getElementsByTagName('status')[0].firstChild.data == 'attached'
volAttachTime = devices[0].getElementsByTagName('ebs')[0].getElementsByTagName('attachTime')[0].firstChild.data
volumeId = devices[0].getElementsByTagName('ebs')[0].getElementsByTagName('volumeId')[0].firstChild.data
#get seconds from "2015-04-15T19:00:59.000Z"
assert getSecondsDelta(volAttachTime, launchTime) <= 3
assert parent.getElementsByTagName('virtualizationType')[0].firstChild.data == 'hvm'
return {'ownerId':ownerId, 'volumeId':volumeId, 'volAttachTime':volAttachTime, 'launchTime':launchTime}
except:
return False
def checkDescribeVolumes(xmlDoc, instanceId, volumeId, volAttachTime):
try:
v = xmlDoc.getElementsByTagName('volumeSet')
volumes = getChildNodes(v)
assert len(volumes) == 1
volume = volumes[0]
assert volume.getElementsByTagName('volumeId')[0].firstChild.data == volumeId
assert volume.getElementsByTagName('snapshotId')[0].firstChild.data == snapshotID
assert volume.getElementsByTagName('status')[0].firstChild.data == 'in-use'
volCreateTime = volume.getElementsByTagName('createTime')[0].firstChild.data
attVolumes = volume.getElementsByTagName('attachmentSet')[0].getElementsByTagName('item')
assert attVolumes.length == 1
attVolume = attVolumes[0];
assert attVolume.getElementsByTagName('volumeId')[0].firstChild.data == volumeId
assert attVolume.getElementsByTagName('instanceId')[0].firstChild.data == instanceId
assert attVolume.getElementsByTagName('device')[0].firstChild.data == '/dev/xvda'
assert attVolume.getElementsByTagName('status')[0].firstChild.data == 'attached'
attTime = attVolume.getElementsByTagName('attachTime')[0].firstChild.data
assert volAttachTime == attTime
#Crucial: volume was created from snapshot and attached at the same instant
#this guarantees that there was no time window to modify it
assert getSecondsDelta(attTime, volCreateTime) == 0
return True
except:
return False
def checkGetConsoleOutput(xmlDoc, instanceId, launchTime):
try:
assert xmlDoc.getElementsByTagName('instanceId')[0].firstChild.data == instanceId
timestamp = xmlDoc.getElementsByTagName('timestamp')[0].firstChild.data
#prevent funny business: last consoleLog entry no later than 5 minutes after instance starts
assert getSecondsDelta(timestamp, launchTime) <= 300
b64data = xmlDoc.getElementsByTagName('output')[0].firstChild.data
logstr = b64decode(b64data)
sigmark = 'PageSigner public key for verification'
pkstartmark = '-----BEGIN PUBLIC KEY-----'
pkendmark = '-----END PUBLIC KEY-----'
mark_start = logstr.index(sigmark)
assert mark_start != -1
pubkey_start = mark_start + logstr[mark_start:].index(pkstartmark)
pubkey_end = pubkey_start+ logstr[pubkey_start:].index(pkendmark) + len(pkendmark)
pk = logstr[pubkey_start:pubkey_end]
assert len(pk) > 0
return pk
except:
return False
# "userData" allows to pass an arbitrary script to the instance at launch. It MUST be empty.
# This is a sanity check because the instance is stripped of the code which parses userData.
def checkDescribeInstanceAttribute(xmlDoc, instanceId):
try:
assert xmlDoc.getElementsByTagName('instanceId')[0].firstChild.data == instanceId
assert not xmlDoc.getElementsByTagName('userData')[0].firstChild
return True
except:
return False
def checkGetUser(xmlDoc, ownerId):
#try:
assert xmlDoc.getElementsByTagName('UserId')[0].firstChild.data == ownerId
assert xmlDoc.getElementsByTagName('Arn')[0].firstChild.data[-(len(ownerId) + len(':root')):] == ownerId+':root'
#except:
#return False
return True
def check_oracle(o):
print ('Verifying PageSigner AWS oracle server, this may take up to a minute...')
xmlDoc = get_xhr(o['DI'])
CDIresult = checkDescribeInstances(xmlDoc, o['instanceId'], o['IP'])
if not CDIresult:
raise Exception('checkDescribeInstances')
print ('check 1 of 5 successful')
xmlDoc = get_xhr(o['DV'])
CDVresult = checkDescribeVolumes(xmlDoc, o['instanceId'], CDIresult['volumeId'], CDIresult['volAttachTime'])
if not CDVresult:
raise Exception('checkDescribeVolumes')
print ('check 2 of 5 successful')
xmlDoc = get_xhr(o['GU'])
CGUresult = checkGetUser(xmlDoc, CDIresult['ownerId'])
if not CGUresult:
raise Exception('checkGetUser')
print ('check 3 of 5 successful')
xmlDoc = get_xhr(o['GCO'])
GCOresult = checkGetConsoleOutput(xmlDoc, o['instanceId'], CDIresult['launchTime'])
if not GCOresult:
raise Exception('checkGetConsoleOutput')
print ('check 4 of 5 successful')
sigmod_binary = bytearray('').join(map(chr,o['modulus']))
if modulus_from_pubkey(GCOresult) != sigmod_binary:
raise Exception("modulus from pubkey")
xmlDoc = get_xhr(o['DIA'])
DIAresult = checkDescribeInstanceAttribute(xmlDoc, o['instanceId'])
if not DIAresult:
raise Exception('checkDescribeInstanceAttribute')
print ('check 5 of 5 successful')
mark = 'AWSAccessKeyId=';
ids = [];
#"AWSAccessKeyId" should be the same to prove that the queries are made on behalf of AWS user "root".
#The attacker can be a user with limited privileges for whom the API would report only partial information.
for url in [o['DI'], o['DV'], o['GU'], o['GCO'], o['DIA']]:
start = url.index(mark)+len(mark);
ids.append(url[start:start + url[start:].index('&')])
assert len(set(ids)) == 1
print('oracle verification successfully finished')

104
src/auditee/parse-pgsg.py
View File

@@ -1,104 +0,0 @@
#!/usr/bin/env python
from __future__ import print_function
import tarfile
from hashlib import md5, sha1, sha256
from os.path import join
import sys, os
data_dir = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.dirname(data_dir))
install_dir = os.path.dirname(os.path.dirname(data_dir))
def extract_audit_data(audit_filename):
audit_data = {}
with open(audit_filename,'rb') as f:
header = f.read(29)
print (header)
if header != 'tlsnotary notarization file\n\n':
raise Exception("Invalid file format")
version = f.read(2)
if version != '\x00\x02':
raise Exception("Incompatible file version")
audit_data['cipher_suite'] = shared.ba2int(f.read(2))
audit_data['client_random'] = f.read(32)
audit_data['server_random'] = f.read(32)
audit_data['pms1'] = f.read(24)
audit_data['pms2'] = f.read(24)
full_cert = f.read(3)
chain_serialized_len = shared.ba2int(full_cert)
chain_serialized = f.read(chain_serialized_len)
full_cert += chain_serialized
audit_data['tlsver'] = f.read(2)
audit_data['initial_tlsver'] = f.read(2)
response_len = shared.ba2int(f.read(8))
audit_data['response'] = f.read(response_len)
IV_len = shared.ba2int(f.read(2))
if IV_len not in [258,16]:
print ("IV length was: ", IV_len)
raise Exception("Wrong IV format in audit file")
audit_data['IV'] = f.read(IV_len)
sig_len = shared.ba2int(f.read(2))
audit_data['signature'] = f.read(sig_len)
audit_data['commit_hash'] = f.read(32)
audit_data['pubkey_pem'] = f.read(sig_len)
offset = 0
chain=[]
while (offset < chain_serialized_len):
l = shared.ba2int(chain_serialized[offset:offset+3])
offset += 3
cert = chain_serialized[offset:offset+l]
offset += l
chain.append(cert)
audit_data['certs'] = chain
audit_data['fullcert'] = full_cert
return audit_data
#unpack and check validity of Python modules
def first_run_check(modname,modhash):
if not modhash: return
mod_dir = join(data_dir, 'python', modname)
if not os.path.exists(mod_dir):
print ('Extracting '+modname + '.tar.gz...')
with open(join(data_dir, 'python', modname+'.tar.gz'), 'rb') as f: tarfile_data = f.read()
if md5(tarfile_data).hexdigest() != modhash:
raise Exception ('Wrong hash')
os.chdir(join(data_dir, 'python'))
tar = tarfile.open(join(data_dir, 'python', modname+'.tar.gz'), 'r:gz')
tar.extractall()
tar.close()
if __name__ == "__main__":
#for md5 hash, see https://pypi.python.org/pypi/<module name>/<module version>
modules_to_load = {'rsa-3.1.4':'b6b1c80e1931d4eba8538fd5d4de1355',\
'pyasn1-0.1.7':'2cbd80fcd4c7b1c82180d3d76fee18c8',\
'slowaes':'','requests-2.3.0':'7449ffdc8ec9ac37bbcd286003c80f00'}
for x,h in modules_to_load.iteritems():
first_run_check(x,h)
sys.path.append(join(data_dir, 'python', x))
import rsa
import pyasn1
import requests
from pyasn1.type import univ
from pyasn1.codec.der import encoder, decoder
from slowaes import AESModeOfOperation
import shared
shared.load_program_config()
shared.import_reliable_sites(join(install_dir,'src','shared'))
print (sys.argv)
if (len(sys.argv)!=2):
raise Exception("Invalid argument")
audit_data = extract_audit_data(sys.argv[1])
for i,c in enumerate(audit_data['certs']):
with open(str(i)+'.der','wb') as f:
f.write(c)
with open('fullcert','wb') as f:
f.write(shared.bi2ba(len(audit_data['fullcert']),fixed=3))
f.write(audit_data['fullcert'])

BIN
src/auditee/python/pyasn1-0.1.7.tar.gz
View File

Binary file not shown.

BIN
src/auditee/python/requests-2.3.0.tar.gz
View File

Binary file not shown.

BIN
src/auditee/python/rsa-3.1.4.tar.gz
View File

Binary file not shown.

0
src/auditee/python/slowaes/__init__.py
View File

656
src/auditee/python/slowaes/slowaes.py
View File

@@ -1,656 +0,0 @@
#!/usr/bin/python
#
# aes.py: implements AES - Advanced Encryption Standard
# from the SlowAES project, http://code.google.com/p/slowaes/
#
# Copyright (c) 2008 Josh Davis ( http://www.josh-davis.org ),
# Alex Martelli ( http://www.aleax.it )
#
# Ported from C code written by Laurent Haan ( http://www.progressive-coding.com )
#
# Licensed under the Apache License, Version 2.0
# http://www.apache.org/licenses/
#
import os
import sys
import math
def append_PKCS7_padding(s):
"""return s padded to a multiple of 16-bytes by PKCS7 padding"""
numpads = 16 - (len(s)%16)
return s + numpads*chr(numpads)
def strip_PKCS7_padding(s):
"""return s stripped of PKCS7 padding"""
if len(s)%16 or not s:
raise ValueError("String of len %d can't be PCKS7-padded" % len(s))
numpads = ord(s[-1])
if numpads > 16:
raise ValueError("String ending with %r can't be PCKS7-padded" % s[-1])
return s[:-numpads]
class AES(object):
# valid key sizes
keySize = dict(SIZE_128=16, SIZE_192=24, SIZE_256=32)
# Rijndael S-box
sbox = [0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, 0x30, 0x01, 0x67,
0x2b, 0xfe, 0xd7, 0xab, 0x76, 0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59,
0x47, 0xf0, 0xad, 0xd4, 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0, 0xb7,
0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, 0xcc, 0x34, 0xa5, 0xe5, 0xf1,
0x71, 0xd8, 0x31, 0x15, 0x04, 0xc7, 0x23, 0xc3, 0x18, 0x96, 0x05,
0x9a, 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75, 0x09, 0x83,
0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, 0x52, 0x3b, 0xd6, 0xb3, 0x29,
0xe3, 0x2f, 0x84, 0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b,
0x6a, 0xcb, 0xbe, 0x39, 0x4a, 0x4c, 0x58, 0xcf, 0xd0, 0xef, 0xaa,
0xfb, 0x43, 0x4d, 0x33, 0x85, 0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c,
0x9f, 0xa8, 0x51, 0xa3, 0x40, 0x8f, 0x92, 0x9d, 0x38, 0xf5, 0xbc,
0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2, 0xcd, 0x0c, 0x13, 0xec,
0x5f, 0x97, 0x44, 0x17, 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19,
0x73, 0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, 0x46, 0xee,
0xb8, 0x14, 0xde, 0x5e, 0x0b, 0xdb, 0xe0, 0x32, 0x3a, 0x0a, 0x49,
0x06, 0x24, 0x5c, 0xc2, 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79,
0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, 0x4e, 0xa9, 0x6c, 0x56, 0xf4,
0xea, 0x65, 0x7a, 0xae, 0x08, 0xba, 0x78, 0x25, 0x2e, 0x1c, 0xa6,
0xb4, 0xc6, 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a, 0x70,
0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, 0x61, 0x35, 0x57, 0xb9,
0x86, 0xc1, 0x1d, 0x9e, 0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e,
0x94, 0x9b, 0x1e, 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf, 0x8c, 0xa1,
0x89, 0x0d, 0xbf, 0xe6, 0x42, 0x68, 0x41, 0x99, 0x2d, 0x0f, 0xb0,
0x54, 0xbb, 0x16]
# Rijndael Inverted S-box
rsbox = [0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38, 0xbf, 0x40, 0xa3,
0x9e, 0x81, 0xf3, 0xd7, 0xfb , 0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f,
0xff, 0x87, 0x34, 0x8e, 0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb , 0x54,
0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23, 0x3d, 0xee, 0x4c, 0x95, 0x0b,
0x42, 0xfa, 0xc3, 0x4e , 0x08, 0x2e, 0xa1, 0x66, 0x28, 0xd9, 0x24,
0xb2, 0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25 , 0x72, 0xf8,
0xf6, 0x64, 0x86, 0x68, 0x98, 0x16, 0xd4, 0xa4, 0x5c, 0xcc, 0x5d,
0x65, 0xb6, 0x92 , 0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda,
0x5e, 0x15, 0x46, 0x57, 0xa7, 0x8d, 0x9d, 0x84 , 0x90, 0xd8, 0xab,
0x00, 0x8c, 0xbc, 0xd3, 0x0a, 0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3,
0x45, 0x06 , 0xd0, 0x2c, 0x1e, 0x8f, 0xca, 0x3f, 0x0f, 0x02, 0xc1,
0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b , 0x3a, 0x91, 0x11, 0x41,
0x4f, 0x67, 0xdc, 0xea, 0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6,
0x73 , 0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85, 0xe2, 0xf9,
0x37, 0xe8, 0x1c, 0x75, 0xdf, 0x6e , 0x47, 0xf1, 0x1a, 0x71, 0x1d,
0x29, 0xc5, 0x89, 0x6f, 0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b ,
0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2, 0x79, 0x20, 0x9a, 0xdb, 0xc0,
0xfe, 0x78, 0xcd, 0x5a, 0xf4 , 0x1f, 0xdd, 0xa8, 0x33, 0x88, 0x07,
0xc7, 0x31, 0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f , 0x60,
0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d, 0x2d, 0xe5, 0x7a, 0x9f,
0x93, 0xc9, 0x9c, 0xef , 0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5,
0xb0, 0xc8, 0xeb, 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61 , 0x17, 0x2b,
0x04, 0x7e, 0xba, 0x77, 0xd6, 0x26, 0xe1, 0x69, 0x14, 0x63, 0x55,
0x21, 0x0c, 0x7d]
def getSBoxValue(self,num):
"""Retrieves a given S-Box Value"""
return self.sbox[num]
def getSBoxInvert(self,num):
"""Retrieves a given Inverted S-Box Value"""
return self.rsbox[num]
def rotate(self, word):
""" Rijndael's key schedule rotate operation.
Rotate a word eight bits to the left: eg, rotate(1d2c3a4f) == 2c3a4f1d
Word is an char list of size 4 (32 bits overall).
"""
return word[1:] + word[:1]
# Rijndael Rcon
Rcon = [0x8d, 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36,
0x6c, 0xd8, 0xab, 0x4d, 0x9a, 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97,
0x35, 0x6a, 0xd4, 0xb3, 0x7d, 0xfa, 0xef, 0xc5, 0x91, 0x39, 0x72,
0xe4, 0xd3, 0xbd, 0x61, 0xc2, 0x9f, 0x25, 0x4a, 0x94, 0x33, 0x66,
0xcc, 0x83, 0x1d, 0x3a, 0x74, 0xe8, 0xcb, 0x8d, 0x01, 0x02, 0x04,
0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, 0x6c, 0xd8, 0xab, 0x4d,
0x9a, 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, 0x35, 0x6a, 0xd4, 0xb3,
0x7d, 0xfa, 0xef, 0xc5, 0x91, 0x39, 0x72, 0xe4, 0xd3, 0xbd, 0x61,
0xc2, 0x9f, 0x25, 0x4a, 0x94, 0x33, 0x66, 0xcc, 0x83, 0x1d, 0x3a,
0x74, 0xe8, 0xcb, 0x8d, 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40,
0x80, 0x1b, 0x36, 0x6c, 0xd8, 0xab, 0x4d, 0x9a, 0x2f, 0x5e, 0xbc,
0x63, 0xc6, 0x97, 0x35, 0x6a, 0xd4, 0xb3, 0x7d, 0xfa, 0xef, 0xc5,
0x91, 0x39, 0x72, 0xe4, 0xd3, 0xbd, 0x61, 0xc2, 0x9f, 0x25, 0x4a,
0x94, 0x33, 0x66, 0xcc, 0x83, 0x1d, 0x3a, 0x74, 0xe8, 0xcb, 0x8d,
0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, 0x6c,
0xd8, 0xab, 0x4d, 0x9a, 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, 0x35,
0x6a, 0xd4, 0xb3, 0x7d, 0xfa, 0xef, 0xc5, 0x91, 0x39, 0x72, 0xe4,
0xd3, 0xbd, 0x61, 0xc2, 0x9f, 0x25, 0x4a, 0x94, 0x33, 0x66, 0xcc,
0x83, 0x1d, 0x3a, 0x74, 0xe8, 0xcb, 0x8d, 0x01, 0x02, 0x04, 0x08,
0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, 0x6c, 0xd8, 0xab, 0x4d, 0x9a,
0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, 0x35, 0x6a, 0xd4, 0xb3, 0x7d,
0xfa, 0xef, 0xc5, 0x91, 0x39, 0x72, 0xe4, 0xd3, 0xbd, 0x61, 0xc2,
0x9f, 0x25, 0x4a, 0x94, 0x33, 0x66, 0xcc, 0x83, 0x1d, 0x3a, 0x74,
0xe8, 0xcb ]
def getRconValue(self, num):
"""Retrieves a given Rcon Value"""
return self.Rcon[num]
def core(self, word, iteration):
"""Key schedule core."""
# rotate the 32-bit word 8 bits to the left
word = self.rotate(word)
# apply S-Box substitution on all 4 parts of the 32-bit word
for i in range(4):
word[i] = self.getSBoxValue(word[i])
# XOR the output of the rcon operation with i to the first part
# (leftmost) only
word[0] = word[0] ^ self.getRconValue(iteration)
return word
def expandKey(self, key, size, expandedKeySize):
"""Rijndael's key expansion.
Expands an 128,192,256 key into an 176,208,240 bytes key
expandedKey is a char list of large enough size,
key is the non-expanded key.
"""
# current expanded keySize, in bytes
currentSize = 0
rconIteration = 1
expandedKey = [0] * expandedKeySize
# set the 16, 24, 32 bytes of the expanded key to the input key
for j in range(size):
expandedKey[j] = key[j]
currentSize += size
while currentSize < expandedKeySize:
# assign the previous 4 bytes to the temporary value t
t = expandedKey[currentSize-4:currentSize]
# every 16,24,32 bytes we apply the core schedule to t
# and increment rconIteration afterwards
if currentSize % size == 0:
t = self.core(t, rconIteration)
rconIteration += 1
# For 256-bit keys, we add an extra sbox to the calculation
if size == self.keySize["SIZE_256"] and ((currentSize % size) == 16):
for l in range(4): t[l] = self.getSBoxValue(t[l])
# We XOR t with the four-byte block 16,24,32 bytes before the new
# expanded key. This becomes the next four bytes in the expanded
# key.
for m in range(4):
expandedKey[currentSize] = expandedKey[currentSize - size] ^ \
t[m]
currentSize += 1
return expandedKey
def addRoundKey(self, state, roundKey):
"""Adds (XORs) the round key to the state."""
for i in range(16):
state[i] ^= roundKey[i]
return state
def createRoundKey(self, expandedKey, roundKeyPointer):
"""Create a round key.
Creates a round key from the given expanded key and the
position within the expanded key.
"""
roundKey = [0] * 16
for i in range(4):
for j in range(4):
roundKey[j*4+i] = expandedKey[roundKeyPointer + i*4 + j]
return roundKey
def galois_multiplication(self, a, b):
"""Galois multiplication of 8 bit characters a and b."""
p = 0
for counter in range(8):
if b & 1: p ^= a
hi_bit_set = a & 0x80
a <<= 1
# keep a 8 bit
a &= 0xFF
if hi_bit_set:
a ^= 0x1b
b >>= 1
return p
#
# substitute all the values from the state with the value in the SBox
# using the state value as index for the SBox
#
def subBytes(self, state, isInv):
if isInv: getter = self.getSBoxInvert
else: getter = self.getSBoxValue
for i in range(16): state[i] = getter(state[i])
return state
# iterate over the 4 rows and call shiftRow() with that row
def shiftRows(self, state, isInv):
for i in range(4):
state = self.shiftRow(state, i*4, i, isInv)
return state
# each iteration shifts the row to the left by 1
def shiftRow(self, state, statePointer, nbr, isInv):
for i in range(nbr):
if isInv:
state[statePointer:statePointer+4] = \
state[statePointer+3:statePointer+4] + \
state[statePointer:statePointer+3]
else:
state[statePointer:statePointer+4] = \
state[statePointer+1:statePointer+4] + \
state[statePointer:statePointer+1]
return state
# galois multiplication of the 4x4 matrix
def mixColumns(self, state, isInv):
# iterate over the 4 columns
for i in range(4):
# construct one column by slicing over the 4 rows
column = state[i:i+16:4]
# apply the mixColumn on one column
column = self.mixColumn(column, isInv)
# put the values back into the state
state[i:i+16:4] = column
return state
# galois multiplication of 1 column of the 4x4 matrix
def mixColumn(self, column, isInv):
if isInv: mult = [14, 9, 13, 11]
else: mult = [2, 1, 1, 3]
cpy = list(column)
g = self.galois_multiplication
column[0] = g(cpy[0], mult[0]) ^ g(cpy[3], mult[1]) ^ \
g(cpy[2], mult[2]) ^ g(cpy[1], mult[3])
column[1] = g(cpy[1], mult[0]) ^ g(cpy[0], mult[1]) ^ \
g(cpy[3], mult[2]) ^ g(cpy[2], mult[3])
column[2] = g(cpy[2], mult[0]) ^ g(cpy[1], mult[1]) ^ \
g(cpy[0], mult[2]) ^ g(cpy[3], mult[3])
column[3] = g(cpy[3], mult[0]) ^ g(cpy[2], mult[1]) ^ \
g(cpy[1], mult[2]) ^ g(cpy[0], mult[3])
return column
# applies the 4 operations of the forward round in sequence
def aes_round(self, state, roundKey):
state = self.subBytes(state, False)
state = self.shiftRows(state, False)
state = self.mixColumns(state, False)
state = self.addRoundKey(state, roundKey)
return state
# applies the 4 operations of the inverse round in sequence
def aes_invRound(self, state, roundKey):
state = self.shiftRows(state, True)
state = self.subBytes(state, True)
state = self.addRoundKey(state, roundKey)
state = self.mixColumns(state, True)
return state
# Perform the initial operations, the standard round, and the final
# operations of the forward aes, creating a round key for each round
def aes_main(self, state, expandedKey, nbrRounds):
state = self.addRoundKey(state, self.createRoundKey(expandedKey, 0))
i = 1
while i < nbrRounds:
state = self.aes_round(state,
self.createRoundKey(expandedKey, 16*i))
i += 1
state = self.subBytes(state, False)
state = self.shiftRows(state, False)
state = self.addRoundKey(state,
self.createRoundKey(expandedKey, 16*nbrRounds))
return state
# Perform the initial operations, the standard round, and the final
# operations of the inverse aes, creating a round key for each round
def aes_invMain(self, state, expandedKey, nbrRounds):
state = self.addRoundKey(state,
self.createRoundKey(expandedKey, 16*nbrRounds))
i = nbrRounds - 1
while i > 0:
state = self.aes_invRound(state,
self.createRoundKey(expandedKey, 16*i))
i -= 1
state = self.shiftRows(state, True)
state = self.subBytes(state, True)
state = self.addRoundKey(state, self.createRoundKey(expandedKey, 0))
return state
# encrypts a 128 bit input block against the given key of size specified
def encrypt(self, iput, key, size):
output = [0] * 16
# the number of rounds
nbrRounds = 0
# the 128 bit block to encode
block = [0] * 16
# set the number of rounds
if size == self.keySize["SIZE_128"]: nbrRounds = 10
elif size == self.keySize["SIZE_192"]: nbrRounds = 12
elif size == self.keySize["SIZE_256"]: nbrRounds = 14
else: return None
# the expanded keySize
expandedKeySize = 16*(nbrRounds+1)
# Set the block values, for the block:
# a0,0 a0,1 a0,2 a0,3
# a1,0 a1,1 a1,2 a1,3
# a2,0 a2,1 a2,2 a2,3
# a3,0 a3,1 a3,2 a3,3
# the mapping order is a0,0 a1,0 a2,0 a3,0 a0,1 a1,1 ... a2,3 a3,3
#
# iterate over the columns
for i in range(4):
# iterate over the rows
for j in range(4):
block[(i+(j*4))] = iput[(i*4)+j]
# expand the key into an 176, 208, 240 bytes key
# the expanded key
expandedKey = self.expandKey(key, size, expandedKeySize)
# encrypt the block using the expandedKey
block = self.aes_main(block, expandedKey, nbrRounds)
# unmap the block again into the output
for k in range(4):
# iterate over the rows
for l in range(4):
output[(k*4)+l] = block[(k+(l*4))]
return output
# decrypts a 128 bit input block against the given key of size specified
def decrypt(self, iput, key, size):
output = [0] * 16
# the number of rounds
nbrRounds = 0
# the 128 bit block to decode
block = [0] * 16
# set the number of rounds
if size == self.keySize["SIZE_128"]: nbrRounds = 10
elif size == self.keySize["SIZE_192"]: nbrRounds = 12
elif size == self.keySize["SIZE_256"]: nbrRounds = 14
else: return None
# the expanded keySize
expandedKeySize = 16*(nbrRounds+1)
# Set the block values, for the block:
# a0,0 a0,1 a0,2 a0,3
# a1,0 a1,1 a1,2 a1,3
# a2,0 a2,1 a2,2 a2,3
# a3,0 a3,1 a3,2 a3,3
# the mapping order is a0,0 a1,0 a2,0 a3,0 a0,1 a1,1 ... a2,3 a3,3
# iterate over the columns
for i in range(4):
# iterate over the rows
for j in range(4):
block[(i+(j*4))] = iput[(i*4)+j]
# expand the key into an 176, 208, 240 bytes key
expandedKey = self.expandKey(key, size, expandedKeySize)
# decrypt the block using the expandedKey
block = self.aes_invMain(block, expandedKey, nbrRounds)
# unmap the block again into the output
for k in range(4):
# iterate over the rows
for l in range(4):
output[(k*4)+l] = block[(k+(l*4))]
return output
class AESModeOfOperation(object):
aes = AES()
# structure of supported modes of operation
modeOfOperation = dict(OFB=0, CFB=1, CBC=2)
# converts a 16 character string into a number array
def convertString(self, string, start, end, mode):
if end - start > 16: end = start + 16
if mode == self.modeOfOperation["CBC"]: ar = [0] * 16
else: ar = []
i = start
j = 0
while len(ar) < end - start:
ar.append(0)
while i < end:
ar[j] = ord(string[i])
j += 1
i += 1
return ar
# Mode of Operation Encryption
# stringIn - Input String
# mode - mode of type modeOfOperation
# hexKey - a hex key of the bit length size
# size - the bit length of the key
# hexIV - the 128 bit hex Initilization Vector
def encrypt(self, stringIn, mode, key, size, IV):
if len(key) % size:
return None
if len(IV) % 16:
return None
# the AES input/output
plaintext = []
iput = [0] * 16
output = []
ciphertext = [0] * 16
# the output cipher string
cipherOut = []
# char firstRound
firstRound = True
if stringIn != None:
for j in range(int(math.ceil(float(len(stringIn))/16))):
start = j*16
end = j*16+16
if end > len(stringIn):
end = len(stringIn)
plaintext = self.convertString(stringIn, start, end, mode)
# print 'PT@%s:%s' % (j, plaintext)
if mode == self.modeOfOperation["CFB"]:
if firstRound:
output = self.aes.encrypt(IV, key, size)
firstRound = False
else:
output = self.aes.encrypt(iput, key, size)
for i in range(16):
if len(plaintext)-1 < i:
ciphertext[i] = 0 ^ output[i]
elif len(output)-1 < i:
ciphertext[i] = plaintext[i] ^ 0
elif len(plaintext)-1 < i and len(output) < i:
ciphertext[i] = 0 ^ 0
else:
ciphertext[i] = plaintext[i] ^ output[i]
for k in range(end-start):
cipherOut.append(ciphertext[k])
iput = ciphertext
elif mode == self.modeOfOperation["OFB"]:
if firstRound:
output = self.aes.encrypt(IV, key, size)
firstRound = False
else:
output = self.aes.encrypt(iput, key, size)
for i in range(16):
if len(plaintext)-1 < i:
ciphertext[i] = 0 ^ output[i]
elif len(output)-1 < i:
ciphertext[i] = plaintext[i] ^ 0
elif len(plaintext)-1 < i and len(output) < i:
ciphertext[i] = 0 ^ 0
else:
ciphertext[i] = plaintext[i] ^ output[i]
for k in range(end-start):
cipherOut.append(ciphertext[k])
iput = output
elif mode == self.modeOfOperation["CBC"]:
for i in range(16):
if firstRound:
iput[i] = plaintext[i] ^ IV[i]
else:
iput[i] = plaintext[i] ^ ciphertext[i]
# print 'IP@%s:%s' % (j, iput)
firstRound = False
ciphertext = self.aes.encrypt(iput, key, size)
# always 16 bytes because of the padding for CBC
for k in range(16):
cipherOut.append(ciphertext[k])
return mode, len(stringIn), cipherOut
# Mode of Operation Decryption
# cipherIn - Encrypted String
# originalsize - The unencrypted string length - required for CBC
# mode - mode of type modeOfOperation
# key - a number array of the bit length size
# size - the bit length of the key
# IV - the 128 bit number array Initilization Vector
def decrypt(self, cipherIn, originalsize, mode, key, size, IV):
# cipherIn = unescCtrlChars(cipherIn)
if len(key) % size:
return None
if len(IV) % 16:
return None
# the AES input/output
ciphertext = []
iput = []
output = []
plaintext = [0] * 16
# the output plain text string
stringOut = ''
# char firstRound
firstRound = True
if cipherIn != None:
for j in range(int(math.ceil(float(len(cipherIn))/16))):
start = j*16
end = j*16+16
if j*16+16 > len(cipherIn):
end = len(cipherIn)
ciphertext = cipherIn[start:end]
if mode == self.modeOfOperation["CFB"]:
if firstRound:
output = self.aes.encrypt(IV, key, size)
firstRound = False
else:
output = self.aes.encrypt(iput, key, size)
for i in range(16):
if len(output)-1 < i:
plaintext[i] = 0 ^ ciphertext[i]
elif len(ciphertext)-1 < i:
plaintext[i] = output[i] ^ 0
elif len(output)-1 < i and len(ciphertext) < i:
plaintext[i] = 0 ^ 0
else:
plaintext[i] = output[i] ^ ciphertext[i]
for k in range(end-start):
stringOut += chr(plaintext[k])
iput = ciphertext
elif mode == self.modeOfOperation["OFB"]:
if firstRound:
output = self.aes.encrypt(IV, key, size)
firstRound = False
else:
output = self.aes.encrypt(iput, key, size)
for i in range(16):
if len(output)-1 < i:
plaintext[i] = 0 ^ ciphertext[i]
elif len(ciphertext)-1 < i:
plaintext[i] = output[i] ^ 0
elif len(output)-1 < i and len(ciphertext) < i:
plaintext[i] = 0 ^ 0
else:
plaintext[i] = output[i] ^ ciphertext[i]
for k in range(end-start):
stringOut += chr(plaintext[k])
iput = output
elif mode == self.modeOfOperation["CBC"]:
output = self.aes.decrypt(ciphertext, key, size)
for i in range(16):
if firstRound:
plaintext[i] = IV[i] ^ output[i]
else:
plaintext[i] = iput[i] ^ output[i]
firstRound = False
if originalsize is not None and originalsize < end:
for k in range(originalsize-start):
stringOut += chr(plaintext[k])
else:
for k in range(end-start):
stringOut += chr(plaintext[k])
iput = ciphertext
return stringOut
def encryptData(key, data, mode=AESModeOfOperation.modeOfOperation["CBC"]):
"""encrypt `data` using `key`
`key` should be a string of bytes.
returned cipher is a string of bytes prepended with the initialization
vector.
"""
key = map(ord, key)
if mode == AESModeOfOperation.modeOfOperation["CBC"]:
data = append_PKCS7_padding(data)
keysize = len(key)
assert keysize in AES.keySize.values(), 'invalid key size: %s' % keysize
# create a new iv using random data
iv = [ord(i) for i in os.urandom(16)]
moo = AESModeOfOperation()
(mode, length, ciph) = moo.encrypt(data, mode, key, keysize, iv)
# With padding, the original length does not need to be known. It's a bad
# idea to store the original message length.
# prepend the iv.
return ''.join(map(chr, iv)) + ''.join(map(chr, ciph))
def decryptData(key, data, mode=AESModeOfOperation.modeOfOperation["CBC"]):
"""decrypt `data` using `key`
`key` should be a string of bytes.
`data` should have the initialization vector prepended as a string of
ordinal values.
"""
key = map(ord, key)
keysize = len(key)
assert keysize in AES.keySize.values(), 'invalid key size: %s' % keysize
# iv is first 16 bytes
iv = map(ord, data[:16])
data = map(ord, data[16:])
moo = AESModeOfOperation()
decr = moo.decrypt(data, None, mode, key, keysize, iv)
if mode == AESModeOfOperation.modeOfOperation["CBC"]:
decr = strip_PKCS7_padding(decr)
return decr
def generateRandomKey(keysize):
"""Generates a key from random data of length `keysize`.
The returned key is a string of bytes.
"""
if keysize not in (16, 24, 32):
emsg = 'Invalid keysize, %s. Should be one of (16, 24, 32).'
raise ValueError, emsg % keysize
return os.urandom(keysize)
if __name__ == "__main__":
moo = AESModeOfOperation()
cleartext = "This is a test!"
cypherkey = [143,194,34,208,145,203,230,143,177,246,97,206,145,92,255,84]
iv = [103,35,148,239,76,213,47,118,255,222,123,176,106,134,98,92]
mode, orig_len, ciph = moo.encrypt(cleartext, moo.modeOfOperation["CBC"],
cypherkey, moo.aes.keySize["SIZE_128"], iv)
print 'm=%s, ol=%s (%s), ciph=%s' % (mode, orig_len, len(cleartext), ciph)
decr = moo.decrypt(ciph, orig_len, mode, cypherkey,
moo.aes.keySize["SIZE_128"], iv)
print decr

2
src/shared/__init__.py
View File

@@ -1,2 +0,0 @@
from tlsn_common import *
from tlsn_ssl import *

209
src/shared/pubkeys.txt
View File

@@ -1,209 +0,0 @@
#This is a comment.
#We use such reliable site here which have 2048 bit RSA modulus
#and which don't serve different certificates depending on
#your location. This rules out using very large sites like google/facebook/twitter.
#Also make sure they don't expire too soon.
#Some websites e.g. apple.com, microsoft.com will not even respond with a TLS alert.
#You can copy-paste the modulus straight from Firefox's
#View Certificate - Details - Subject's Public Key
#There must be a blank line between entries.
#
#Name=whatsapp.com
#Expires=09/09/2019
#Modulus=
d3 6a fd 68 f8 ec f1 3a 2b 6d 01 7c 92 f7 98 37
f6 fc 3a 19 91 4a 19 a8 7f 85 a7 6b 46 b2 e1 a3
66 f6 63 4f e7 90 83 cf 00 00 a7 7a 68 48 38 04
0a e7 c3 d5 96 f4 2a 95 21 b5 04 e4 c3 23 cd 56
81 92 cd 34 0f 32 67 7e 64 d9 41 52 08 2e 43 2d
2a d3 c2 95 7a f6 20 36 10 fa 36 f2 18 2e e5 f1
6a 40 79 ae 38 5d 1a 24 8e 64 a1 85 a2 c4 df dd
89 9b a9 02 e9 cc a4 5c 4a bc a2 5a 89 c5 74 6d
14 e5 7e 71 b8 97 f9 d0 d1 06 f0 cc a4 cb 66 d5
a8 8c e1 25 27 4b 3f 80 db 1c 14 ca ec b8 35 89
f6 e6 2e 36 e3 3b 9a 9c 03 94 18 e2 e9 1d 66 4d
4f 74 d2 58 16 9b 44 cd 9d 82 53 ad 85 10 bf 73
3a bf 1c 9e 45 cc a7 8a c0 3e ab c7 31 ce ef 60
ac 19 9e 03 bd ab 7d 4c 08 b6 d2 8a 0b 3a 84 cf
50 a8 69 a0 d7 29 04 65 5f 8e 65 4f 3d a4 5c 64
8d 81 0f e3 61 e0 b0 6b 9f 2d 29 94 ce 74 fd 73
#
#Name=wordpress.org
#Expires=12/15/2020
#Modulus=
b7 ee 9f ad 32 5f 3b 77 0b d7 f6 f1 9b 4d 88 15
90 78 92 84 e0 4e 77 f0 9a a3 cf 25 11 d9 bf 00
14 12 20 bd 96 a6 6f da 53 e7 c5 0b 40 8c a6 44
7a 2f 53 4c 54 90 44 b2 22 87 9d 02 01 93 53 f1
c7 ba 1a bb 8d 4d 8f 79 bc b2 c9 00 71 78 0a c8
ba 9b 93 99 7f 1e 98 93 f9 66 ce c4 65 03 07 53
f3 91 8b 9e 5e 23 72 01 f4 b9 4f db 65 3c 33 d4
c9 42 6b 17 43 71 5b b6 cc 02 b5 3e 72 e3 fb 07
04 2f 01 8f 89 fa ad 2e 76 15 4a 19 4b 3f 36 b6
f7 95 2f 4c e6 95 9a 9f 95 61 c5 af 4b 02 ed 9f
e3 a2 a9 77 78 d3 93 d6 a3 73 35 dd 8a 21 e7 f3
f2 1d a8 c7 74 17 43 5a 10 38 86 b3 ef 1e f8 db
7c ac 79 cd ca 7f 6e ea 18 dd d4 4b 1e 83 87 fa
71 34 93 74 17 ad a1 03 4d 8d 8c cc e7 4c 0c a4
2b 8a a8 51 af cd 86 b3 01 0d a2 3b 77 e4 ce 33
ec c0 3d 29 1d 45 c2 84 94 74 66 da 8a a1 b0 f5
#
#Name=vimeo.com
#Expires=03/20/2020
#Modulus=
c7 08 83 5a 6d b9 8c 67 5e db bb 3f 14 cb 7b 93
ba 48 c7 bb 0d 8a 4b a5 df cc 02 8a 9b 80 93 65
b0 c7 e5 0f 04 3a 48 b9 ec ff 3e dc 8c 7a 90 cc
cf e5 22 79 30 3b a5 87 8b dc ee 1b 49 cb ea 91
57 67 1d fc 88 71 29 80 51 1d 6f 49 33 a2 34 b6
78 6c 33 7c ff ec 89 60 c6 ca 46 c1 b7 92 00 cc
74 bf 33 03 58 e0 42 eb 29 a1 22 52 16 36 c1 18
ad 30 3b af 5c cf 77 77 fe 01 3e 05 8a 42 6d be
5d 60 b0 78 44 c8 06 5d a8 88 2d 65 34 32 36 a8
3c 97 3b 3d e7 1f ca cf 67 fa fc 4e 76 36 ee 35
37 7b bf ba e3 f9 51 00 e9 79 49 21 e6 24 07 cc
89 11 dc 54 b7 dc e4 93 9c c9 8a ef 42 d4 77 b2
eb bc 67 82 21 de 57 d2 bb a4 bd b9 2f 49 b9 96
f2 54 72 02 d2 7d 77 c4 33 a2 db 0f 36 37 89 a2
e9 55 17 97 20 64 e7 00 b1 3e f6 53 db 95 22 2d
0a 1d b9 23 61 42 2b 44 e6 5b 8a 1a f6 f5 6c fb
#
#Name=w3.org
#Expires=06/02/2019
#Modulus=
b7 e1 f2 60 0c e4 cf 2d 23 ed 29 10 a0 b4 53 be
35 ed 8d 3b f9 8f b5 45 41 ce ee 51 ed d5 2b 42
fe ea 04 12 d5 f5 1a ad 71 b1 4a 8e 53 21 90 04
d0 b4 2c 9d 08 b3 0d 5a 6e 45 19 1b de 96 78 0d
22 5d 0a 5e 85 bd dc a0 fb 45 e8 45 04 9f 58 06
3e 82 f7 e7 f8 68 05 e2 69 43 87 ce 14 e6 d0 aa
0e 5f d5 51 c2 55 0e fd 95 5e 3f 44 22 7c f8 87
96 00 c6 7a cb f8 43 a8 fc 45 c7 69 53 c5 8a a9
86 78 27 d7 de f4 61 a3 e3 af 7e fa df 56 e2 a0
5b cd 53 a8 88 32 98 73 40 68 db ea 9a 76 33 ae
25 ae b5 7a c6 ab ce ae 3c a9 fc 73 df a4 74 52
90 a2 51 e0 95 eb 0f 59 cd ea 46 47 88 48 0a ea
1a 91 ee 8e 2c a4 ae 1c a1 4a 2a ce b4 ad 23 d9
d5 7a 36 3a 2b 9d 7c 34 64 9a c7 43 b0 1b 19 64
23 7c fa ee 38 57 6a c3 3b 61 b4 6c 28 ea 8e b4
41 ca 54 79 e3 e3 97 90 74 f7 49 8f 5d 06 45 2b
#
#Name=digg.com
#Expires=06/08/2019
#Modulus=
e1 21 04 3d f2 6d a8 f2 cc e9 b6 1a b7 22 71 c3
a3 19 4d 2f d0 b0 eb be 86 84 0e e1 fa 07 63 a1
77 81 50 fa b5 4f f1 ce 39 cd 48 33 8c 70 0b 7e
93 8f 00 24 1c 4c fe 94 45 cc 0d bc de 26 d3 74
5e 70 38 52 9e bc 67 57 89 b1 1b af 3d c0 38 cf
23 27 cc 69 36 83 03 dc ec 33 67 21 d4 19 e3 02
df a8 76 86 95 9d d5 f1 5a 04 0e 08 1e 68 f7 9d
c3 c4 ec ae 1d 50 3b 79 ab fd 3f 8d 2b 3c 70 c5
1e 6c 5e 0c b7 d6 fa a9 c1 2e 88 e7 53 31 32 6a
dc 98 c1 e4 25 0b 6d aa cb 56 33 9b 4a d7 38 39
d5 36 52 27 53 c0 8f f6 d2 97 6f d1 40 33 21 7a
5d f0 6b 69 ef c1 8e ae 8c 71 1b b8 46 bd ed c0
86 4e 74 31 96 93 7b fc d2 0a cc ba f2 31 78 f6
58 e7 0d fa ad b1 35 c0 d8 3d be 88 55 08 92 87
3b b5 88 61 37 df 2a b0 69 5b 3a cf a3 28 c6 3a
49 da 6d cf 08 1c 95 75 cb 99 3c 2d 91 19 d4 8b
#
#Name=dropbox.com
#Expires=11/02/2020
#Modulus=
bc 9e 8d 01 9d d4 c0 ad 20 40 eb d5 45 eb 66 84
2a 28 be 22 21 6f 2b 8a 14 74 4c 40 17 c5 be fd
33 c4 1c cf 4a e8 02 7a 63 f5 bf 6a 5e 72 55 ba
11 e3 b8 ef ef 82 17 67 ab 25 f3 45 7a 56 b7 1c
15 32 43 57 6f a0 16 d0 57 08 c7 c3 e3 13 6d 03
d7 2f 85 73 46 3e 1c 19 62 1f 5a 93 50 15 ce 83
4c 1e 50 90 95 0e 14 bf 1e 64 14 ec 02 0d 63 31
de 46 39 1c 53 d0 4e 17 8a 07 bb ca bd 4c 88 3f
4f 8b df e1 6b be f6 3a f5 8b bd bb 77 2a ee 16
81 86 71 d4 77 cf 0e 46 c9 ae fa 28 9a 97 44 3b
e3 49 ad 58 f5 bc 2f 2e 08 c3 8e 0e b6 a3 17 0d
f8 87 37 3c 8a 07 39 6c aa 42 91 b8 b2 60 93 d7
0b 3d fb d0 0b dc 56 6a 07 cf cf b1 9f b8 54 ba
73 90 93 0b 6b 33 d6 4c ec ad e3 fe af fd da b9
f4 0b 48 f5 c6 f4 c2 33 c2 46 6e d0 11 7d fe 9b
19 b9 1f dd 33 3f 8b 92 38 38 f9 d8 a4 66 39 3b
#
#Name=archive.org
#Expires=02/22/2020
#Modulus=
d2 ba 32 ad 20 0d 8a e5 05 34 ff 93 af 34 83 45
11 e3 95 87 2d 53 f2 14 d9 a4 af e1 ac 4a 86 e8
e0 fb 08 9c d0 66 2e c7 f9 e2 a2 9b a8 d3 dd d8
38 3e f1 67 89 6c 69 db fa 36 bb 2b 73 66 e8 5f
a2 df c0 62 bf 14 35 43 e3 3b 3c 28 d3 76 5d 09
93 a1 22 db 61 cf b6 8a 62 6f e0 43 23 4f 2e 28
b1 9f c5 31 9e bf b1 a4 b7 a3 b8 a5 e8 64 59 01
bf 18 db b2 37 b4 c3 42 58 65 79 73 85 16 ad 7c
6e c4 25 ab b0 7c 84 7d a8 be b3 68 8c d9 3a 67
ed b8 bc 71 1a 68 c8 05 fe 5c 53 72 ff d5 ba fc
e2 da 0b 08 66 22 34 c7 b9 76 6d c4 9b 98 5e 0e
ec 68 d6 8b e8 5f 54 0a ef 51 8f 89 a9 93 60 ac
ee 1a 45 06 a4 4e 19 c6 38 35 59 ef d1 4d 22 cd
01 fe 51 26 9e c7 1b fb e8 6d 46 86 01 dc b5 23
1a 63 5a 9c 1b 08 6a 36 4b b7 58 ea bc f8 b4 49
c6 51 f5 5e b0 ca 55 36 3c df 2a 16 24 4e e1 ab
#
#Name=mail.ru
#Expires=10/08/2020
#Modulus=
a5 46 ec 16 f0 19 9b ca 25 64 a5 97 8e 87 79 ee
ed c3 83 81 84 c2 a2 19 cb 3b e9 e8 f7 28 7e b8
8e 52 07 b8 dc 08 75 14 15 1f 69 c5 5c 77 91 10
ac e9 29 36 97 93 ca 05 5b 6c db e6 1d 78 b4 91
41 ce 10 89 c9 aa fa 1f fe e6 21 0e f4 03 ab 70
a9 94 64 c1 2b 29 5a c9 de 1c 1b f8 f4 95 20 99
59 79 97 f1 52 de 55 0b c0 c9 73 21 ad 28 5a 8a
96 ee 87 cf 09 2d d1 5a e5 57 20 74 3a fe a5 ac
38 d5 26 b3 61 9a 5f 32 44 d4 ef bd 68 a4 9d 1d
a4 a9 3f 05 af 23 8f ce 70 a5 72 58 36 3f 50 c5
8b 38 5a e0 e9 2c 37 c9 8a 8e 91 5e 77 c9 75 0c
43 bd 1d 70 33 70 41 e2 eb f4 e9 6f 9b bf 37 dc
18 af 43 04 93 02 d7 5c 97 b7 a5 7c 96 f1 5f 0c
d7 4c ed b5 b6 db 4f ff 4b 5a 21 cb 70 d5 cf 75
cb 06 c9 a3 31 eb 36 fe 79 17 a5 2d c8 0f 23 67
3d 9c 56 9d 82 b2 f3 bb b5 7e 2b 9a 7b c2 65 69
#
#Name=360.cn
#Expires=02/12/2019
#Modulus=
81 5d df fd 0f 33 bb ff 34 9a 64 d4 64 2f af 7d
77 e9 ac 27 a7 33 3d d1 cc 3e cb c0 ab fb ae 1c
c4 de 96 38 39 a6 ab ed 42 83 35 68 55 d4 91 fd
15 38 94 4a fb ee 84 f1 5f 21 28 a3 e3 62 79 d3
25 f9 88 34 4a aa c9 ff d5 fa 17 51 75 f4 57 56
8d a3 d9 66 db bc 68 02 e8 e4 a1 25 78 3f 37 b7
ed b0 8a ca b8 34 d7 b9 df 09 51 a2 9a 86 e6 b5
37 4c 9b 5b 4d 66 b5 0a 8c 17 b9 70 16 e5 82 c8
7e b9 11 1a 93 19 b5 ed d8 f3 e7 41 2e 3f 22 88
31 fa 30 b8 6b 9e 08 e1 0f ec af 70 02 d9 b6 0a
49 af 45 66 43 02 be 75 89 af db f1 5e 7f f0 7a
07 06 a7 0c 46 46 45 d4 cd 1f ae 16 f5 d3 26 08
5a d5 95 23 91 04 31 a9 d7 9a 4d 61 cb 23 b1 42
e5 25 b2 72 0f d8 54 86 6c 0f 98 e5 59 db fb 08
34 4e 82 1d 96 0e 77 36 f3 f5 91 c0 67 45 de 33
82 0d 4f 22 d3 e8 0e f6 1b 90 82 2f bc 23 ef fd
#
#Name=netflix.com
#Expires=25/07/2019
#Modulus=
a6 4b e4 63 b2 47 cb c5 3b 11 63 b7 6a af b9 1d
e0 90 c2 2c 56 2e 31 97 47 6a e4 f8 1e e5 a6 82
96 c0 8d 27 f4 db ab ae e0 2a 5a eb 11 6d b7 26
30 4c e9 6a f6 fe 0b 50 ab 1f f6 b2 2c ce c7 68
c3 bf 38 48 aa a6 5f 6d 99 a2 22 69 1b e3 ae 10
ea 0f 25 36 be a8 50 a2 ba 0f 10 28 67 8a 20 b3
11 76 fd 84 65 06 11 31 91 4a ba 8f d5 4f 42 22
2f c3 f7 46 a3 15 8e 4a 6d b9 30 4e d3 cc e0 35
f3 27 b8 4e 56 54 ba f4 62 3e 6f 13 fa 19 2d 0c
d1 f0 42 32 bb 76 76 92 40 1d b0 4b 9d 87 55 d9
83 86 c4 e3 60 54 ba fc 56 39 b5 46 ed 72 44 46
cd 54 1b d0 fb cc f2 75 44 f8 8d 21 be 5c 03 cb
88 d2 3b 59 78 3e 1b c3 7b c0 d7 23 be f1 bd d5
21 71 65 3d 8d 5b 51 88 21 fe 0f 2e b4 37 d8 18
5e 22 7b bb 7c fd 07 05 d2 b9 9c ad a7 90 2f 35
6a bc 85 17 fa b0 28 aa ec 54 3f 3d e5 81 00 97

187
src/shared/tlsn_common.py
View File

@@ -1,187 +0,0 @@
from __future__ import print_function
from ConfigParser import SafeConfigParser
from SocketServer import ThreadingMixIn
from struct import pack
import os, binascii, itertools, re, random
import threading, BaseHTTPServer
import select, time, socket
from subprocess import check_output
#General utility objects used by both auditor and auditee.
config = SafeConfigParser()
config_location = os.path.join(os.path.dirname(os.path.realpath(__file__)),'tlsnotary.ini')
#required_options = {'Notary':['notary_server','notary_port']}
required_options = {}
reliable_sites = {}
def verify_signature(msg, signature, modulus):
'''RSA verification is sig^e mod n, drop the padding and get the last 32 bytes
Args: msg as sha256 digest, signature as bytearray, modulus as (big) int
'''
sig = ba2int(signature)
exponent = 65537
result = pow(sig,exponent,modulus)
padded_hash = bi2ba(result,fixed=512) #4096 bit key
unpadded_hash = padded_hash[512-32:]
if msg==unpadded_hash:
return True
else:
return False
def load_program_config():
loadedFiles = config.read([config_location])
#detailed sanity checking :
#did the file exist?
if len(loadedFiles) != 1:
raise Exception("Could not find config file: "+config_location)
#check for sections
for s in required_options:
if s not in config.sections():
raise Exception("Config file does not contain the required section: "+s)
#then check for specific options
for k,v in required_options.iteritems():
for o in v:
if o not in config.options(k):
raise Exception("Config file does not contain the required option: "+o)
def import_reliable_sites(d):
'''Read in the site names and ssl ports from the config file,
and then read in the corresponding pubkeys in browser hex format from
the file pubkeys.txt in directory d. Then combine this data into the reliable_sites global dict'''
sites = [x.strip() for x in config.get('SSL','reliable_sites').split(',')]
ports = [int(x.strip()) for x in config.get('SSL','reliable_sites_ssl_ports').split(',')]
assert len(sites) == len(ports), "Error, tlsnotary.ini file contains a mismatch between reliable sites and ports"
#import hardcoded pubkeys
with open(os.path.join(d,'pubkeys.txt'),'rb') as f: plines = f.readlines()
raw_pubkeys= []
pubkeys = []
while len(plines):
next_raw_pubkey = list(itertools.takewhile(lambda x: x.startswith('#') != True,plines))
k = len(next_raw_pubkey)
plines = plines[k+1:]
if k > 0 : raw_pubkeys.append(''.join(next_raw_pubkey))
for rp in raw_pubkeys:
pubkeys.append(re.sub(r'\s+','',rp))
for i,site in enumerate(sites):
reliable_sites[site] = [ports[i]]
reliable_sites[site].append(pubkeys[i])
def check_complete_records(d):
'''Given a response d from a server,
we want to know if its contents represents
a complete set of records, however many.'''
l = ba2int(d[3:5])
if len(d)< l+5: return False
elif len(d)==l+5: return True
else: return check_complete_records(d[l+5:])
def create_sock(server,prt):
returned_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
returned_sock.settimeout(int(config.get("General","tcp_socket_timeout")))
returned_sock.connect((server, prt))
return returned_sock
def recv_socket(sckt,is_handshake=False):
last_time_data_was_seen_from_server = 0
data_from_server_seen = False
databuffer=''
while True:
rlist, wlist, xlist = select.select((sckt,), (), (sckt,), 1)
if len(rlist) == len(xlist) == 0: #timeout
#TODO dont rely on a fixed timeout
delta = int(time.time()) - last_time_data_was_seen_from_server
if not data_from_server_seen: continue
if delta < int(config.get("General","server_response_timeout")): continue
return databuffer #we timed out on the socket read
if len(xlist) > 0:
print ('Socket exceptional condition. Terminating connection')
return ''
if len(rlist) == 0:
print ('Python internal socket error: rlist should not be empty. Please investigate. Terminating connection')
return ''
for rsckt in rlist:
data = rsckt.recv(1024*32)
if not data:
if not databuffer:
raise Exception ("Server closed the socket and sent no data")
else:
return databuffer
data_from_server_seen = True
databuffer += data
if is_handshake:
if check_complete_records(databuffer): return databuffer #else, just continue loop
last_time_data_was_seen_from_server = int(time.time())
def bi2ba(bigint,fixed=None):
m_bytes = []
while bigint != 0:
b = bigint%256
m_bytes.insert( 0, b )
bigint //= 256
if fixed:
padding = fixed - len(m_bytes)
if padding > 0: m_bytes = [0]*padding + m_bytes
return bytearray(m_bytes)
def xor(a,b):
return bytearray([ord(a) ^ ord(b) for a,b in zip(a,b)])
def bigint_to_list(bigint):
m_bytes = []
while bigint != 0:
b = bigint%256
m_bytes.insert( 0, b )
bigint //= 256
return m_bytes
#convert bytearray into int
def ba2int(byte_array):
return int(str(byte_array).encode('hex'), 16)
def gunzip_http(http_data):
import gzip
import StringIO
http_header = http_data[:http_data.find('\r\n\r\n')+len('\r\n\r\n')]
#\s* below means any amount of whitespaces
if re.search(r'content-encoding:\s*deflate', http_header, re.IGNORECASE):
#TODO manually resend the request with compression disabled
raise Exception('Please set gzip_disabled = 1 in tlsnotary.ini and rerun the audit')
if not re.search(r'content-encoding:\s*gzip', http_header, re.IGNORECASE):
return http_data #nothing to gunzip
http_body = http_data[len(http_header):]
ungzipped = http_header
gzipped = StringIO.StringIO(http_body)
f = gzip.GzipFile(fileobj=gzipped, mode="rb")
ungzipped += f.read()
return ungzipped
def dechunk_http(http_data):
'''Dechunk only if http_data is chunked otherwise return http_data unmodified'''
http_header = http_data[:http_data.find('\r\n\r\n')+len('\r\n\r\n')]
#\s* below means any amount of whitespaces
if not re.search(r'transfer-encoding:\s*chunked', http_header, re.IGNORECASE):
return http_data #nothing to dechunk
http_body = http_data[len(http_header):]
dechunked = http_header
cur_offset = 0
chunk_len = -1 #initialize with a non-zero value
while True:
new_offset = http_body[cur_offset:].find('\r\n')
if new_offset==-1: #pre-caution against endless looping
#pinterest.com is known to not send the last 0 chunk when HTTP gzip is disabled
return dechunked
chunk_len_hex = http_body[cur_offset:cur_offset+new_offset]
chunk_len = int(chunk_len_hex, 16)
if chunk_len ==0: break #for properly-formed html we should break here
cur_offset += new_offset+len('\r\n')
dechunked += http_body[cur_offset:cur_offset+chunk_len]
cur_offset += chunk_len+len('\r\n')
return dechunked

1191
src/shared/tlsn_ssl.py
View File

File diff suppressed because it is too large Load Diff

22
src/shared/tlsnotary.ini
View File

@@ -1,22 +0,0 @@
[Notary]
server_name = 54.158.251.14
server_port = 10011
[SSL]
#NOTE: Do *not* modify this list of websites unless
#your auditor tells you to. They are meant
#to be highly trusted websites that both sides can use
#to test out the secret negotiation process. If you
#modify this list, your initial connection to the auditor
#may fail.
reliable_sites = whatsapp.com,wordpress.org,vimeo.com,w3.org,digg.com,dropbox.com,archive.org,mail.ru,360.cn,netflix.com
reliable_sites_ssl_ports = 443,443,443,443,443,443,443,443,443,443
[General]
msg_chunk_size = 110
tcp_socket_timeout = 20
server_response_timeout = 3
gzip_disabled = 0
prevent_render = 0
decrypt_with_slowaes = 1
use_paillier_scheme = 0
tls_11 = 1