github/self
1
1
Fork 0
mirror of https://github.com/selfxyz/self.git synced 2026-04-27 03:01:15 -04:00
Code Issues Packages Projects Releases Wiki Activity

feat: add parallelization to speed up merkle tree building

Browse Source
This commit is contained in:
Evi Nova
2026-01-10 20:42:27 +10:00
parent 2a1ccb4c3a
commit b149fc3f47
10 changed files with 1632 additions and 414 deletions
Download Patch File Download Diff File Expand all files Collapse all files

284
common/scripts/ofac/OFAC_AUTO_UPDATER_README.md
View File

@@ -2,69 +2,147 @@
Automated pipeline for updating OFAC sanctions list with ~200-500ms mismatch window using Google Cloud Storage.
## Overview
The OFAC auto-updater runs in a **Trusted Execution Environment (TEE)** and performs:
1. Downloads OFAC SDN XML from U.S. Treasury
2. Parses XML into structured JSON format
3. Builds 7 Merkle trees in parallel (passport, name+dob, name+yob, ID card variants, Aadhaar variants)
4. Updates on-chain OFAC roots via smart contracts
5. Uploads tree files to Google Cloud Storage with atomic pointer updates
## Prerequisites
### Google Cloud Storage Access
1. Create a GCS bucket for OFAC data (e.g., `self-ofac-prod`, `self-ofac-staging`)
1. Create a GCS bucket for OFAC data (e.g., `self-ofac-test`, `self-ofac-staging`)
2. Enable versioning on the bucket for rollback capability
3. Create a service account with `roles/storage.objectAdmin` permission
4. Download the service account key JSON file
5. Set `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the key file path
5. Place credentials file at `common/scripts/ofac/gcs-credentials.json` (or set `GOOGLE_APPLICATION_CREDENTIALS`)
---
### TEE Role Setup
## Single-Shot Auto Update (Docker/TEE)
The TEE's address must have `TEE_ROLE` on all registry contracts before production deployment.
This is the unified flow that runs the pipeline, updates on-chain roots directly,
and uploads files to Google Cloud Storage with atomic pointer updates.
**Registry Addresses (Celo Mainnet)**:
- IdentityRegistry: `0x37F5CB8cB1f6B00aa768D8aA99F1A9289802A968`
- IdentityRegistryIdCard: `0xeAD1E6Ec29c1f3D33a0662f253a3a94D189566E1`
- IdentityRegistryAadhaar: `0xd603Fa8C8f4694E8DD1DcE1f27C0C3fc91e32Ac4`
### Docker
**Grant TEE_ROLE**:
```bash
# Get TEE address from private key
TEE_ADDRESS=$(cast wallet address --private-key $TEE_PRIVATE_KEY)
TEE_ROLE=$(cast keccak "TEE_ROLE")
# Grant on all registries (requires SECURITY_ROLE)
for REGISTRY in \
"0x37F5CB8cB1f6B00aa768D8aA99F1A9289802A968" \
"0xeAD1E6Ec29c1f3D33a0662f253a3a94D189566E1" \
"0xd603Fa8C8f4694E8DD1DcE1f27C0C3fc91e32Ac4"; do
cast send $REGISTRY \
"grantRole(bytes32,address)" \
$TEE_ROLE $TEE_ADDRESS \
--rpc-url https://forno.celo.org \
--private-key $ADMIN_PRIVATE_KEY
done
```
## Usage
### Docker (Production/TEE)
Build the image:
```bash
docker build -f common/scripts/ofac/Dockerfile -t ofac-auto-updater .
```
Run with GCS credentials:
```bash
docker run --rm \\
-e PRIVATE_KEY=0x... \\
-e NETWORK=celo \\
-e GCS_BUCKET_NAME=self-ofac-prod \\
-e GOOGLE_APPLICATION_CREDENTIALS=/secrets/gcs-key.json \\
-v /local/ofac:/data \\
-v /local/gcs-key.json:/secrets/gcs-key.json:ro \\
docker run --rm \
-e PRIVATE_KEY=0x... \
-e NETWORK=celo \
-e GCS_BUCKET_NAME=self-ofac-test \
-e GOOGLE_APPLICATION_CREDENTIALS=/secrets/gcs-key.json \
-v /local/ofac:/data \
-v /local/gcs-key.json:/secrets/gcs-key.json:ro \
ofac-auto-updater
```
Environment variables:
- `PRIVATE_KEY` (required): signer key used for on-chain updates
- Signer must have `TEE_ROLE` on the registry contracts
- `NETWORK`: `celo` or `celo-sepolia`
- `RPC_URL` or network-specific RPC envs (`CELO_RPC_URL`, `CELO_SEPOLIA_RPC_URL`)
- `OFAC_DATA_DIR` (default: `/data/ofac`)
- `GCS_BUCKET_NAME` (default: `self-ofac-prod` for celo, `self-ofac-staging` for celo-sepolia)
- `GCS_BASE_PATH` (default: `ofac`)
- `GOOGLE_APPLICATION_CREDENTIALS` (required): path to GCS service account key JSON
- `DRY_RUN=true` to skip on-chain update and upload
- `SKIP_UPLOAD=true` to skip GCS upload (not recommended)
If deploying this change to existing registries, call `initializeTeeRole(TEE_ADDRESS)` after upgrade to set role admin and grant `TEE_ROLE`.
### Local (no Docker)
### Local Execution
```bash
PRIVATE_KEY=0x... \\
NETWORK=celo \\
GCS_BUCKET_NAME=self-ofac-prod \\
GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcs-key.json \\
PRIVATE_KEY=0x... \
NETWORK=celo \
GCS_BUCKET_NAME=self-ofac-test \
GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcs-key.json \
yarn tsx common/scripts/ofac/runOfacAutoUpdate.ts
```
---
### Environment Variables
- `PRIVATE_KEY` (required): Signer key for on-chain updates. Must have `TEE_ROLE` on registry contracts.
- `NETWORK`: `celo` or `celo-sepolia` (default: `celo`)
- `RPC_URL`: Custom RPC URL (or use network defaults)
- `OFAC_DATA_DIR`: Data directory (default: `/data/ofac`)
- `GCS_BUCKET_NAME`: GCS bucket name (default: `self-ofac-test` for celo, `self-ofac-staging` for celo-sepolia)
- `GCS_BASE_PATH`: Base path in bucket (default: `ofac`)
- `GOOGLE_APPLICATION_CREDENTIALS`: Path to GCS service account key JSON
- `DRY_RUN=true`: Skip on-chain updates and GCS uploads
- `SKIP_UPLOAD=true`: Skip GCS upload only
- `SKIP_PIPELINE=true`: Skip pipeline, use existing trees (for testing)
## Testing
### Full Fork Test (Pipeline + On-Chain + GCS)
Tests complete flow with real GCS uploads and local blockchain fork:
```bash
# 1. Start local fork (forks at latest block automatically)
anvil --fork-url https://forno.celo.org --port 8545
# 2. Run test (uses gcs-credentials.json from ofac folder)
./common/scripts/ofac/test-with-fork.sh
```
**What it tests**:
- Real pipeline execution
- TEE_ROLE auto-granting on fork
- On-chain updates (on fork, no real gas)
- Real GCS uploads to staging bucket
- Mismatch window measurement
### On-Chain Only Test (Skip Tree Building)
Quick test for on-chain updates without rebuilding trees:
```bash
# Requires existing trees from previous run
./common/scripts/ofac/test-onchain-only.sh
```
**What it tests**:
- TEE_ROLE granting logic
- On-chain update transactions
- Uses existing tree files (skips pipeline)
### Dry Run Test
Simulates complete flow without real changes:
```bash
./common/scripts/ofac/test-dry-run.sh
```
**What it tests**:
- Pipeline logic
- Progress dashboard
- No real GCS uploads or on-chain calls
## How GCS Is Used
The updater uses Google Cloud Storage for atomic file deployment:
@@ -75,14 +153,13 @@ The updater uses Google Cloud Storage for atomic file deployment:
### Mismatch Window
- **Old (SSH)**: 6-7 seconds during file move operation
- **New (GCS)**: 200-500ms during `current.json` upload
- **Target**: < 1 second between on-chain confirmation and GCS pointer update
- **Consistency**: Readers always see a complete snapshot (all files from the same version)
### File Structure
```
gs://self-ofac-prod/
gs://self-ofac-test/
ofac/
current.json # Pointer to active version
2026-01-09-1736437890/ # Versioned directory
@@ -111,7 +188,134 @@ gs://self-ofac-prod/
}
```
Readers should:
**Reader Flow**:
1. Fetch `gs://bucket/ofac/current.json`
2. Parse the `path` field
3. Fetch tree files from `gs://bucket/{path}/`
## TEE On-Chain Calls
### How TEE Makes Calls
The TEE uses a private key stored securely within the TEE environment:
1. Private key never leaves TEE (stored in secure enclave)
2. TEE address derived from private key: `address = publicKeyToAddress(privateKeyToPublicKey(privateKey))`
3. Transactions signed with TEE's private key before sending to blockchain
4. Contract verifies `hasRole(TEE_ROLE, msg.sender)` before allowing updates
### Transaction Flow
```
TEE Environment:
1. Generate OFAC roots (off-chain)
2. Create transaction: updatePassportNoOfacRoot(newRoot)
3. Sign transaction with TEE's private key
4. Send signed transaction to RPC node
5. Transaction included in block
6. Contract verifies: hasRole(TEE_ROLE, msg.sender)
7. If authorized, root is updated ✅
```
### Security Model
- **Private Key**: Never leaves TEE environment
- **Attestation**: TEE provides cryptographic proof of identity (AWS Nitro Enclaves)
- **Role-Based Access**: Only addresses with `TEE_ROLE` can update OFAC roots
- **Audit Trail**: All transactions on-chain and verifiable
## Production Deployment
### Prerequisites
1. TEE environment (AWS Nitro Enclaves or equivalent)
2. TEE private key securely stored
3. `TEE_ROLE` granted on all registry contracts
4. GCS credentials with `roles/storage.objectAdmin`
5. RPC access to Celo mainnet
### Deployment Steps
1. **Grant TEE_ROLE** (one-time setup, see above)
2. **Build Docker image**: `docker build -f common/scripts/ofac/Dockerfile -t ofac-auto-updater .`
3. **Deploy to TEE** with:
- TEE private key (from secure storage)
- GCS credentials (service account key)
- Network configuration
### Verification
```bash
# Verify TEE_ROLE is granted
TEE_ROLE=$(cast keccak "TEE_ROLE")
TEE_ADDRESS="0x..." # Your TEE address
for REGISTRY in \
"0x37F5CB8cB1f6B00aa768D8aA99F1A9289802A968" \
"0xeAD1E6Ec29c1f3D33a0662f253a3a94D189566E1" \
"0xd603Fa8C8f4694E8DD1DcE1f27C0C3fc91e32Ac4"; do
cast call $REGISTRY \
"hasRole(bytes32,address)" \
$TEE_ROLE $TEE_ADDRESS \
--rpc-url https://forno.celo.org
done
```
Expected: All return `0x0000...0001` (true)
## Troubleshooting
### Error: "AccessControlUnauthorizedAccount"
**Cause**: TEE address doesn't have `TEE_ROLE`
**Solution**: Grant `TEE_ROLE` using instructions above
### Error: "execution reverted"
**Cause**: Caller doesn't have `SECURITY_ROLE` to grant `TEE_ROLE` (fork testing)
**Solution**: Fork test automatically grants TEE_ROLE. If it fails, ensure fork includes Hub contract at `0xe57F4773bd9c9d8b6Cd70431117d353298B9f5BF`
### Error: "GCS upload failed"
**Cause**: Invalid credentials or permissions
**Solution**:
- Verify `GOOGLE_APPLICATION_CREDENTIALS` path
- Check service account has `roles/storage.objectAdmin`
- Test with `gsutil` directly
### Error: "Transaction failed"
**Cause**: Insufficient gas, network issues, or contract revert
**Solution**:
- Check gas price and balance
- Verify network connectivity
- Check contract state (roots already match?)
## Performance
**Typical execution times** (on 4-core machine):
- Download: ~10-15 seconds
- Parse XML: ~5-10 seconds (~18,000 entries)
- Build trees: ~2-4 minutes (7 trees in parallel)
- Upload to GCS: ~15-30 seconds (60 MB total)
- On-chain updates: ~5-10 seconds per transaction (3-9 total)
- Pointer update: ~0.2-0.5 seconds
**Total**: ~3-5 minutes end-to-end
**Resource usage**:
- Memory: ~1-2 GB peak (during tree building)
- Disk: ~200 MB (XML + trees)
- Network: ~80 MB download/upload
## Security Considerations
1. **Private Key Storage**: Store TEE private key securely (AWS Secrets Manager, HashiCorp Vault)
2. **Role Management**: Use multisig for `SECURITY_ROLE` on production
3. **Monitoring**: Monitor all transactions from TEE address, set up alerts for failures
4. **Backup**: Have a backup TEE address with `TEE_ROLE` in case primary fails

382
common/scripts/ofac/buildAllTreesParallel.ts Normal file
View File

@@ -0,0 +1,382 @@
/**
* OFAC Tree Builder - Parallel Version
*
* Builds all OFAC Merkle trees in parallel using async execution.
* Provides better progress tracking compared to sequential building.
*/
import * as fs from 'fs';
import * as path from 'path';
import { buildAadhaarSMTAsync, buildSMTAsync } from '../../src/utils/trees.js';
import type { OfacEntry } from './parseSdn.js';
import { ProgressMonitor } from './progressMonitor.js';
export interface TreeBuildResult {
treeType: string;
entriesProcessed: number;
entriesAdded: number;
buildTime: number;
root: string;
exportPath?: string;
}
export interface AllTreesResult {
success: boolean;
trees: TreeBuildResult[];
totalTime: number;
error?: string;
}
function loadEntries(inputPath: string): OfacEntry[] {
if (!fs.existsSync(inputPath)) {
throw new Error(`Input file not found: ${inputPath}`);
}
const content = fs.readFileSync(inputPath, 'utf-8');
return JSON.parse(content) as OfacEntry[];
}
/**
* Build a single tree and save it
* This function is designed to be run in parallel with others
*/
async function buildSingleTree(
entries: OfacEntry[],
treeType: string,
outputFile: string,
outputDir: string,
treeIndex: number,
monitor: ProgressMonitor | null,
filterFn?: (e: OfacEntry) => boolean
): Promise<TreeBuildResult> {
const startTime = performance.now();
// Filter entries
const filteredEntries = filterFn ? entries.filter(filterFn) : entries;
if (monitor) {
monitor.updateTree(treeIndex, {
status: 'building',
totalEntries: filteredEntries.length
});
}
await new Promise(resolve => setImmediate(resolve));
// Build tree based on type
let count: number, buildTime: number, tree: any;
try {
if (treeType.startsWith('aadhaar_')) {
const baseType = treeType.replace('aadhaar_', '') as 'name_and_dob' | 'name_and_yob';
[count, buildTime, tree] = await buildAadhaarSMTAsync(filteredEntries, baseType, monitor, treeIndex);
} else {
[count, buildTime, tree] = await buildSMTAsync(filteredEntries, treeType as any, monitor, treeIndex);
}
// Export and save
const treeExport = tree.export();
const outputPath = path.join(outputDir, outputFile);
fs.writeFileSync(outputPath, JSON.stringify(treeExport));
const totalTime = performance.now() - startTime;
// Update monitor with completion
if (monitor) {
monitor.updateTree(treeIndex, {
status: 'completed',
entriesAdded: count,
entriesProcessed: filteredEntries.length,
totalEntries: filteredEntries.length
});
}
return {
treeType,
entriesProcessed: filteredEntries.length,
entriesAdded: count,
buildTime,
root: tree.root.toString(),
exportPath: outputPath,
};
} catch (error) {
if (monitor) {
monitor.updateTree(treeIndex, { status: 'failed' });
}
throw error;
}
}
/**
* Build all OFAC trees in PARALLEL
*/
export async function buildAllOfacTreesParallel(
inputPath: string,
outputDir: string
): Promise<AllTreesResult> {
const startTime = performance.now();
try {
// Ensure output directory exists
if (!fs.existsSync(outputDir)) {
fs.mkdirSync(outputDir, { recursive: true });
}
// Load entries once
console.log(`\nLoading entries from: ${inputPath}`);
const entries = loadEntries(inputPath);
console.log(`Loaded ${entries.length} entries`);
// Calculate entry counts for each tree type
const passportEntries = entries.filter((e) => !!(e.Pass_No && e.Pass_Country));
// Initialize monitor but don't start yet
const monitor = new ProgressMonitor([
'passport_no+nationality',
'name+dob',
'name+yob',
'name+dob (ID)',
'name+yob (ID)',
'Aadhaar name+dob',
'Aadhaar name+yob',
]);
// Pre-populate with entry counts (we know them in advance)
monitor.updateTree(1, { totalEntries: passportEntries.length });
monitor.updateTree(2, { totalEntries: entries.length });
monitor.updateTree(3, { totalEntries: entries.length });
monitor.updateTree(4, { totalEntries: entries.length });
monitor.updateTree(5, { totalEntries: entries.length });
monitor.updateTree(6, { totalEntries: entries.length });
monitor.updateTree(7, { totalEntries: entries.length });
console.log('Starting parallel tree building...\n');
// Suppress verbose logs BEFORE starting monitor
const logControl = ProgressMonitor.suppressLogs();
// Start monitor AFTER suppressing logs
monitor.start();
// Give a moment for dashboard to render
await new Promise(resolve => setTimeout(resolve, 100));
try {
// Mark all trees as building before starting promises
for (let i = 1; i <= 7; i++) {
monitor.updateTree(i, { status: 'building' });
}
// Force a render to show "building" status immediately
await new Promise(resolve => setImmediate(resolve));
const treePromises = [
// Passport trees (3 trees)
buildSingleTree(
entries,
'passport_no_and_nationality',
'passportNoAndNationalitySMT.json',
outputDir,
1,
monitor,
(e) => !!(e.Pass_No && e.Pass_Country)
),
buildSingleTree(
entries,
'name_and_dob',
'nameAndDobSMT.json',
outputDir,
2,
monitor
),
buildSingleTree(
entries,
'name_and_yob',
'nameAndYobSMT.json',
outputDir,
3,
monitor
),
// ID Card trees (2 trees)
buildSingleTree(
entries,
'name_and_dob_id_card',
'nameAndDobSMT_ID.json',
outputDir,
4,
monitor
),
buildSingleTree(
entries,
'name_and_yob_id_card',
'nameAndYobSMT_ID.json',
outputDir,
5,
monitor
),
// Aadhaar trees (2 trees)
buildSingleTree(
entries,
'aadhaar_name_and_dob',
'nameAndDobSMT_AADHAAR.json',
outputDir,
6,
monitor
),
buildSingleTree(
entries,
'aadhaar_name_and_yob',
'nameAndYobSMT_AADHAAR.json',
outputDir,
7,
monitor
),
];
// Start all promises
const allResults = await Promise.all(treePromises);
monitor.stop();
logControl.restore();
const totalTime = performance.now() - startTime;
// Save roots summary
const rootsSummary: Record<string, string> = {};
for (const result of allResults) {
rootsSummary[result.treeType] = result.root;
}
const rootsPath = path.join(outputDir, 'roots.json');
fs.writeFileSync(rootsPath, JSON.stringify(rootsSummary, null, 2));
console.log('\n┌─────────────────────────────────────────────────────────────┐');
console.log(`│ ✅ ALL 7 TREES COMPLETED IN ${(totalTime / 1000).toFixed(1)}s`.padEnd(62) + '│');
console.log('└─────────────────────────────────────────────────────────────┘');
console.log(`\nRoots summary saved to: ${rootsPath}`);
return {
success: true,
trees: allResults,
totalTime,
};
} catch (error) {
monitor.stop();
logControl.restore();
return {
success: false,
trees: [],
totalTime: performance.now() - startTime,
error: (error as Error).message,
};
}
} catch (error) {
return {
success: false,
trees: [],
totalTime: performance.now() - startTime,
error: (error as Error).message,
};
}
}
/**
* Print build results summary
*/
function printSummary(result: AllTreesResult): void {
console.log('\n╔══════════════════════════════════════════════════════════════════╗');
console.log('║ 📊 PARALLEL TREE BUILD SUMMARY ║');
console.log('╚══════════════════════════════════════════════════════════════════╝');
if (!result.success) {
console.log(`\n❌ Build failed: ${result.error}`);
return;
}
console.log('\n┌────────────────────────────────────────────────────────────────┐');
console.log('│ Tree Name │ Entries │ Time │ Status │');
console.log('├────────────────────────────────┼─────────┼────────┼───────────┤');
const getIcon = (entries: number) => {
if (entries > 15000) return '🔴';
if (entries > 10000) return '🟡';
if (entries > 5000) return '🟢';
return '🔵';
};
for (const tree of result.trees) {
const icon = getIcon(tree.entriesAdded);
const name = tree.treeType
.replace('_and_', '+')
.replace('_id_card', ' (ID)')
.replace('aadhaar_', 'Aadhaar ');
const time = (tree.buildTime / 1000).toFixed(1) + 's';
console.log(
`│ ${icon} ${name.padEnd(29)} │ ${tree.entriesAdded.toString().padStart(7)} │ ${time.padStart(6)} │ ✅ Done │`
);
}
console.log('└────────────────────────────────────────────────────────────────┘');
// Calculate stats
const totalEntries = result.trees.reduce((sum, t) => sum + t.entriesAdded, 0);
const avgTime = result.trees.reduce((sum, t) => sum + t.buildTime, 0) / result.trees.length;
console.log('\n┌────────────────────────────────────────────────────────────────┐');
console.log('│ ⚡ PERFORMANCE METRICS │');
console.log('├────────────────────────────────────────────────────────────────┤');
console.log(`│ Total entries processed: ${totalEntries.toString().padStart(10)} │`);
console.log(`│ Trees built: ${result.trees.length.toString().padStart(10)} │`);
console.log(`│ Average time per tree: ${(avgTime / 1000).toFixed(1).padStart(10)}s │`);
console.log(`│ Total build time: ${(result.totalTime / 1000).toFixed(1).padStart(10)}s │`);
console.log('└────────────────────────────────────────────────────────────────┘');
// Root hashes preview
console.log('\n┌────────────────────────────────────────────────────────────────┐');
console.log('│ 🔐 ROOT HASHES (Preview) │');
console.log('└────────────────────────────────────────────────────────────────┘');
for (const tree of result.trees.slice(0, 3)) {
const rootPreview = tree.root.substring(0, 40) + '...';
console.log(` ${tree.treeType}: ${rootPreview}`);
}
console.log(` ... and ${result.trees.length - 3} more`);
console.log('\n✅ All trees built successfully!\n');
}
// CLI entrypoint
async function main() {
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const inputPath = process.argv[2] || path.join(__dirname, '../../../ofacdata/inputs/names.json');
const outputDir = process.argv[3] || path.join(__dirname, '../../../ofacdata/outputs');
console.log('='.repeat(60));
console.log('OFAC Tree Builder (PARALLEL VERSION)');
console.log('='.repeat(60));
console.log(`Input: ${inputPath}`);
console.log(`Output: ${outputDir}`);
const result = await buildAllOfacTreesParallel(inputPath, outputDir);
printSummary(result);
if (!result.success) {
process.exit(1);
}
console.log('\n✅ All trees built successfully!');
}
// Run if executed directly
import { fileURLToPath } from 'url';
const isMainModule = process.argv[1] === fileURLToPath(import.meta.url);
if (isMainModule) {
main().catch((error) => {
console.error('Fatal error:', error);
process.exit(1);
});
}

6
common/scripts/ofac/gcsUpload.ts
View File

@@ -28,6 +28,7 @@ export interface GcsUploadResult {
export interface PointerUpdateResult {
success: boolean;
durationMs: number;
completedAt?: number;
error?: string;
}
@@ -147,7 +148,7 @@ export async function updatePointerFile(
if (dryRun) {
log(' [DRY RUN] Would update pointer to: ' + versionPath);
return { success: true, durationMs: 0 };
return { success: true, durationMs: 0, completedAt: Date.now() };
}
const startTime = Date.now();
@@ -172,9 +173,10 @@ export async function updatePointerFile(
});
const durationMs = Date.now() - startTime;
const completedAt = Date.now();
log(`Pointer updated in ${durationMs}ms`);
return { success: true, durationMs };
return { success: true, durationMs, completedAt };
} catch (error) {
const durationMs = Date.now() - startTime;
return {

18
common/scripts/ofac/index.ts
View File

@@ -16,6 +16,7 @@ import * as path from 'path';
import { fileURLToPath } from 'url';
import { buildAllOfacTrees, type AllTreesResult } from './buildAllTrees.js';
import { buildAllOfacTreesParallel } from './buildAllTreesParallel.js';
import { downloadOfacSdn } from './downloadSdn.js';
import { parseOfacSdn, saveOfacData } from './parseSdn.js';
@@ -33,6 +34,7 @@ export interface PipelineOptions {
rawDir?: string;
inputDir?: string;
outputDir?: string;
parallel?: boolean;
}
/**
@@ -98,7 +100,16 @@ export async function runOfacPipeline(options: PipelineOptions = {}): Promise<Pi
console.log('\n🌳 STEP 3: Building Merkle trees...');
console.log('-'.repeat(50));
const namesPath = path.join(inputDir, 'names.json');
const treesResult = await buildAllOfacTrees(namesPath, outputDir);
const useParallel = options.parallel !== false;
let treesResult: AllTreesResult;
if (useParallel) {
treesResult = await buildAllOfacTreesParallel(namesPath, outputDir);
} else {
treesResult = await buildAllOfacTrees(namesPath, outputDir);
}
if (!treesResult.success) {
return {
@@ -164,11 +175,6 @@ async function main() {
console.log('\n' + '═'.repeat(70));
if (result.success) {
console.log('✅ OFAC UPDATE PIPELINE COMPLETED SUCCESSFULLY');
console.log('');
console.log('Next steps:');
console.log(' 1. Upload trees to tree server');
console.log(' 2. Prepare multisig transaction with new roots');
console.log(' 3. Get 2/5 dev approvals on Safe');
} else {
console.log('❌ OFAC UPDATE PIPELINE FAILED');
console.log(` Error: ${result.error}`);

249
common/scripts/ofac/progressMonitor.ts Normal file
View File

@@ -0,0 +1,249 @@
/**
* Progress Monitor for Parallel Tree Building
*
* Displays a clean, real-time dashboard showing all 7 trees building simultaneously
*/
export interface TreeProgress {
id: number;
name: string;
status: 'pending' | 'building' | 'completed' | 'failed';
totalEntries?: number;
entriesAdded?: number;
entriesProcessed?: number; // Track entries processed (i), not just added
startTime?: number;
endTime?: number;
}
export class ProgressMonitor {
private trees: Map<number, TreeProgress> = new Map();
private startTime: number = Date.now();
private updateInterval?: NodeJS.Timeout;
private lastOutput: string = '';
constructor(treeNames: string[]) {
treeNames.forEach((name, index) => {
this.trees.set(index + 1, {
id: index + 1,
name,
status: 'pending',
});
});
}
start() {
// Hide cursor (but don't clear screen - preserve previous output)
process.stdout.write('\x1B[?25l');
// Add some spacing before dashboard
console.log('\n');
// Update display every 33ms (~30 FPS) for smooth progress updates
// This ensures we see updates even when processing is fast
this.updateInterval = setInterval(() => this.render(), 33);
this.render();
}
updateTree(id: number, updates: Partial<TreeProgress>) {
const tree = this.trees.get(id);
if (tree) {
Object.assign(tree, updates);
if (updates.status === 'building' && !tree.startTime) {
tree.startTime = Date.now();
}
if (updates.status === 'completed' && !tree.endTime) {
tree.endTime = Date.now();
}
}
}
stop() {
if (this.updateInterval) {
clearInterval(this.updateInterval);
}
// Show cursor again
process.stdout.write('\x1B[?25h\n');
}
private render() {
const lines: string[] = [];
const elapsed = ((Date.now() - this.startTime) / 1000).toFixed(1);
// Header
lines.push('┌────────────────────────────────────────────────────────────────────────┐');
lines.push('│ 🚀 PARALLEL TREE BUILDING - LIVE PROGRESS │');
lines.push('├────────────────────────────────────────────────────────────────────────┤');
lines.push(`│ Elapsed: ${elapsed}s`.padEnd(73) + '│');
lines.push('└────────────────────────────────────────────────────────────────────────┘');
lines.push('');
const COL_ID_WIDTH = 2;
const COL_NAME_WIDTH = 28;
const COL_PROGRESS_WIDTH = 27;
const COL_TIME_WIDTH = 10;
lines.push('┌────┬──────────────────────────────┬─────────────────────────────┬────────────┐');
lines.push('│ # │ Tree Name │ Progress │ Time │');
lines.push('├────┼──────────────────────────────┼─────────────────────────────┼────────────┤');
const trees = Array.from(this.trees.values()).sort((a, b) => a.id - b.id);
for (const tree of trees) {
const status = this.getProgressDisplay(tree);
const time = this.getTimeDisplay(tree);
const idStr = String(tree.id).padStart(COL_ID_WIDTH);
const nameStr = tree.name.padEnd(COL_NAME_WIDTH);
const statusStr = status.padEnd(COL_PROGRESS_WIDTH);
const timeStr = time.padEnd(COL_TIME_WIDTH);
lines.push(`│ ${idStr} │ ${nameStr} │ ${statusStr} │ ${timeStr} │`);
}
lines.push('└────┴──────────────────────────────┴─────────────────────────────┴────────────┘');
// Summary
const completed = trees.filter(t => t.status === 'completed').length;
const building = trees.filter(t => t.status === 'building').length;
const failed = trees.filter(t => t.status === 'failed').length;
const totalEntriesProcessed = trees
.filter(t => t.status === 'completed')
.reduce((sum, t) => sum + (t.entriesAdded || 0), 0);
lines.push('');
lines.push(`Status: ${completed}/${trees.length} completed • ${building} building • ${failed} failed`);
if (totalEntriesProcessed > 0) {
lines.push(`Total entries added to trees: ${totalEntriesProcessed.toLocaleString()}`);
}
// Always render to show smooth progress updates
// Even if output looks similar, numbers are changing
const output = lines.join('\n');
const numLines = lines.length;
// Move cursor up to overwrite previous output (but only after first render)
if (this.lastOutput) {
const prevLines = this.lastOutput.split('\n').length;
process.stdout.write(`\x1B[${prevLines}A`); // Move up
}
process.stdout.write('\x1B[J' + output); // Clear from cursor and write
this.lastOutput = output;
}
private getProgressDisplay(tree: TreeProgress): string {
switch (tree.status) {
case 'pending':
return tree.totalEntries
? `⏳ ${tree.totalEntries.toLocaleString()} entries`
: '⏳ Waiting...';
case 'building':
// Show entries processed if available, otherwise entries added
const processed = tree.entriesProcessed ?? tree.entriesAdded ?? 0;
if (tree.totalEntries && processed > 0) {
// Show live progress: "15,234 / 18,455 (83%)"
const percent = Math.floor((processed / tree.totalEntries) * 100);
const added = tree.entriesAdded ?? 0;
// Show both processed and added for clarity
return `🔄 ${processed.toLocaleString()}/${tree.totalEntries.toLocaleString()} (${percent}%)`;
} else if (tree.totalEntries) {
return `🔄 0/${tree.totalEntries.toLocaleString()}`;
}
return '🔄 Building...';
case 'completed':
if (tree.totalEntries !== undefined) {
return `✅ ${tree.totalEntries.toLocaleString()}/${tree.totalEntries.toLocaleString()} (100%)`;
} else if (tree.entriesAdded !== undefined) {
return `✅ ${tree.entriesAdded.toLocaleString()} added`;
}
return '✅ Done';
case 'failed':
return '❌ Failed';
}
}
private getTimeDisplay(tree: TreeProgress): string {
if (tree.status === 'completed' && tree.startTime && tree.endTime) {
const duration = ((tree.endTime - tree.startTime) / 1000).toFixed(1);
return `${duration}s`;
}
if (tree.status === 'building' && tree.startTime) {
const duration = ((Date.now() - tree.startTime) / 1000).toFixed(1);
return `${duration}s...`;
}
return '-';
}
// Method to suppress console.log during tree building
static suppressLogs() {
const originalLog = console.log;
const originalError = console.error;
const originalWarn = console.warn;
const originalWrite = process.stdout.write;
const originalErrorWrite = process.stderr.write;
const logBuffer: string[] = [];
console.log = (...args: any[]) => {
const message = args.join(' ');
if (message.includes('ERROR') || message.includes('FATAL')) {
originalLog.apply(console, args);
}
logBuffer.push(message);
return;
};
console.error = (...args: any[]) => {
const message = args.join(' ');
if (message.includes('dob is null') ||
message.includes('year is null') ||
message.includes('arr.length') ||
message.includes('Processing') ||
message.includes('name_and')) {
return;
}
originalError.apply(console, args);
};
console.warn = (...args: any[]) => {
const message = args.join(' ');
if (message.includes('arr') || message.includes('arr.length')) {
return;
}
originalWarn.apply(console, args);
};
process.stdout.write = (chunk: any, ...args: any[]): boolean => {
const message = chunk.toString();
if (message.includes('Processing') ||
message.includes('name_and') ||
message.includes('number') && message.includes('out of') ||
message.includes('year is null') ||
message.includes('dob is null') ||
message.includes('arr WARN') ||
message.includes('arr.length') ||
message.includes('This entry already exists') ||
message.includes('skipping')) {
return true;
}
return originalWrite.call(process.stdout, chunk, ...args);
};
process.stderr.write = (chunk: any, ...args: any[]): boolean => {
const message = chunk.toString();
if (message.includes('arr WARN') ||
message.includes('arr.length') ||
message.includes('dob is null') ||
message.includes('year is null')) {
return true;
}
return originalErrorWrite.call(process.stderr, chunk, ...args);
};
return {
restore: () => {
console.log = originalLog;
console.error = originalError;
console.warn = originalWarn;
process.stdout.write = originalWrite;
process.stderr.write = originalErrorWrite;
}
};
}
}

407
common/scripts/ofac/runOfacAutoUpdate.ts
View File

@@ -27,15 +27,23 @@ const DEFAULT_RPC_URLS: Record<string, string> = {
};
const DEFAULT_GCS_BUCKETS: Record<string, string> = {
celo: 'self-ofac-prod',
celo: 'self-ofac-test',
'celo-sepolia': 'self-ofac-staging',
};
// Hardcoded registry addresses (Celo Mainnet)
const CELO_REGISTRY_ADDRESSES: Record<string, string> = {
IdentityRegistry: '0x37F5CB8cB1f6B00aa768D8aA99F1A9289802A968',
IdentityRegistryIdCard: '0xeAD1E6Ec29c1f3D33a0662f253a3a94D189566E1',
IdentityRegistryAadhaar: '0xd603Fa8C8f4694E8DD1DcE1f27C0C3fc91e32Ac4',
// Hardcoded registry addresses
const REGISTRY_ADDRESSES: Record<string, Record<string, string>> = {
celo: {
IdentityRegistry: '0x37F5CB8cB1f6B00aa768D8aA99F1A9289802A968',
IdentityRegistryIdCard: '0xeAD1E6Ec29c1f3D33a0662f253a3a94D189566E1',
IdentityRegistryAadhaar: '0xd603Fa8C8f4694E8DD1DcE1f27C0C3fc91e32Ac4',
},
'celo-sepolia': {
// Use same addresses for dry-run testing (or configure testnet addresses)
IdentityRegistry: '0x37F5CB8cB1f6B00aa768D8aA99F1A9289802A968',
IdentityRegistryIdCard: '0xeAD1E6Ec29c1f3D33a0662f253a3a94D189566E1',
IdentityRegistryAadhaar: '0xd603Fa8C8f4694E8DD1DcE1f27C0C3fc91e32Ac4',
},
};
// Registry configuration
@@ -75,6 +83,12 @@ const REGISTRY_ABI = [
'function updatePassportNoOfacRoot(uint256 root)',
'function updateNameAndDobOfacRoot(uint256 root)',
'function updateNameAndYobOfacRoot(uint256 root)',
'function getRoleMember(bytes32 role, uint256 index) view returns (address)',
'function getRoleMemberCount(bytes32 role) view returns (uint256)',
];
const OWNER_ABI = [
'function owner() view returns (address)',
];
@@ -95,10 +109,11 @@ function getRpcUrl(network: string): string | undefined {
}
function getRegistryAddress(registryKey: string, network: string): string | null {
if (network === 'celo') {
return CELO_REGISTRY_ADDRESSES[registryKey] || null;
const addresses = REGISTRY_ADDRESSES[network];
if (!addresses) {
return null;
}
return null;
return addresses[registryKey] || null;
}
function loadRoots(rootsPath: string): Record<string, string> {
@@ -153,60 +168,31 @@ async function getCurrentRoot(
}
}
async function updateRegistryRoots(
async function verifyRegistryRoots(
config: RegistryConfig,
registryAddress: string,
signer: ethers.Wallet,
roots: Record<string, string>,
dryRun: boolean
): Promise<number> {
const contract = new ethers.Contract(registryAddress, REGISTRY_ABI, signer);
let updates = 0;
async function updateRoot(
rootType: 'passportNo' | 'nameAndDob' | 'nameAndYob',
updateFn: keyof ethers.Contract
) {
const newRoot = getRootForRegistry(roots, config, rootType);
if (!newRoot) return;
const oldRoot = await getCurrentRoot(contract, rootType);
if (oldRoot === newRoot) {
log(`Skipping ${config.name} ${rootType}: on-chain root already matches`);
return;
}
log(`Updating ${config.name} ${rootType}`);
if (dryRun) {
log('[DRY RUN] Skipping on-chain update');
updates += 1;
return;
}
const tx = await (contract[updateFn] as any)(newRoot);
log(`TX submitted: ${tx.hash}`);
const receipt = await tx.wait(1);
if (receipt?.status !== 1) {
throw new Error(`Update failed for ${config.name} ${rootType}`);
}
log(`Confirmed in block ${receipt.blockNumber}`);
updates += 1;
}
contract: ethers.Contract,
roots: Record<string, string>
): Promise<boolean> {
let allMatch = true;
for (const rootType of config.rootTypes) {
if (rootType === 'passportNo') {
await updateRoot('passportNo', 'updatePassportNoOfacRoot');
} else if (rootType === 'nameAndDob') {
await updateRoot('nameAndDob', 'updateNameAndDobOfacRoot');
const expectedRoot = getRootForRegistry(roots, config, rootType);
if (!expectedRoot) continue;
const onChainRoot = await getCurrentRoot(contract, rootType);
const matches = onChainRoot === expectedRoot;
if (matches) {
log(`✅ ${config.name} ${rootType}: matches (${onChainRoot.slice(0, 10)}...)`);
} else {
await updateRoot('nameAndYob', 'updateNameAndYobOfacRoot');
log(`❌ ${config.name} ${rootType}: mismatch`);
log(` Expected: ${expectedRoot}`);
log(` On-chain: ${onChainRoot}`);
allMatch = false;
}
}
return updates;
return allMatch;
}
async function main() {
console.log('');
console.log('='.repeat(70));
@@ -218,6 +204,7 @@ async function main() {
const privateKey = process.env.PRIVATE_KEY;
const rpcUrl = process.env.RPC_URL || getRpcUrl(network);
const dryRun = process.env.DRY_RUN === 'true';
const skipPipeline = process.env.SKIP_PIPELINE === 'true' || process.argv.includes('--skip-pipeline');
if (!privateKey) {
console.error('ERROR: PRIVATE_KEY environment variable required');
@@ -250,19 +237,28 @@ async function main() {
log(`GCS bucket: ${bucketName}`);
log(`GCS base path: ${basePath}`);
log(`Dry Run: ${dryRun}`);
log(`Skip Pipeline: ${skipPipeline}`);
console.log('');
// Step 1-3: Pipeline
log('Running OFAC pipeline...');
const pipeline = await runOfacPipeline({
rawDir,
inputDir,
outputDir,
});
if (!skipPipeline) {
log('Running OFAC pipeline...');
if (!pipeline.success) {
console.error('ERROR: Pipeline failed:', pipeline.error);
process.exit(1);
const pipeline = await runOfacPipeline({
rawDir,
inputDir,
outputDir,
});
if (!pipeline.success) {
console.error('ERROR: Pipeline failed:', pipeline.error);
process.exit(1);
}
} else {
log('Skipping pipeline (using existing trees)...');
if (!fs.existsSync(rootsPath) && !fs.existsSync(path.join(outputDir, 'roots.json'))) {
console.error(`ERROR: Roots file not found. Expected: ${rootsPath} or ${path.join(outputDir, 'roots.json')}`);
process.exit(1);
}
}
// Step 4: On-chain updates
@@ -280,79 +276,248 @@ async function main() {
const signer = new ethers.Wallet(privateKey, provider);
log(`Signer: ${signer.address}`);
let totalUpdates = 0;
const jsonRpcProvider = provider as ethers.JsonRpcProvider;
const rpcUrlStr = (jsonRpcProvider as any).connection?.url || (jsonRpcProvider as any)._getConnection?.()?.url || '';
const isFork = rpcUrlStr.includes('localhost') || rpcUrlStr.includes('127.0.0.1') || rpcUrlStr.includes('8545');
let adminAddress: string | null = null;
let adminSigner: ethers.Signer | null = null;
if (isFork && !dryRun) {
if (network === 'celo') {
adminAddress = '0x067b18e09A10Fa03d027c1D60A098CEbbE5637f0';
log(`[FORK] Using Celo operations multisig (OPERATIONS_ROLE): ${adminAddress}`);
} else if (network === 'celo-sepolia') {
const firstRegistry = REGISTRY_CONFIGS.find(c => getRegistryAddress(c.registryKey, network));
if (firstRegistry) {
const address = getRegistryAddress(firstRegistry.registryKey, network);
if (address) {
const registryOwner = new ethers.Contract(address, OWNER_ABI, jsonRpcProvider);
adminAddress = await registryOwner.owner();
log(`[FORK] Using Celo Sepolia owner: ${adminAddress}`);
}
}
}
if (adminAddress) {
if (rpcUrlStr.includes('anvil') || rpcUrlStr.includes('8545')) {
await jsonRpcProvider.send('anvil_impersonateAccount', [adminAddress]);
} else if (rpcUrlStr.includes('hardhat') || rpcUrlStr.includes('1337')) {
await jsonRpcProvider.send('hardhat_impersonateAccount', [adminAddress]);
}
adminSigner = await jsonRpcProvider.getSigner(adminAddress);
const balance = await jsonRpcProvider.getBalance(adminAddress);
if (balance === 0n) {
const funder = await jsonRpcProvider.getSigner('0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266');
await funder.sendTransaction({
to: adminAddress,
value: ethers.parseEther('1'),
});
}
log(`[FORK] Impersonating ${adminAddress} to call update functions`);
}
}
const allUpdates: Array<{
config: RegistryConfig;
address: string;
rootType: 'passportNo' | 'nameAndDob' | 'nameAndYob';
updateFn: keyof ethers.Contract;
newRoot: string;
}> = [];
for (const config of REGISTRY_CONFIGS) {
const address = getRegistryAddress(config.registryKey, network);
if (!address) {
log(`Registry not configured for network: ${config.name}`);
if (dryRun) {
log(`[DRY RUN] Registry not configured for ${network}: ${config.name} (skipping)`);
} else {
log(`Registry not configured for network: ${config.name}`);
}
continue;
}
log(`Updating ${config.name} at ${address}`);
totalUpdates += await updateRegistryRoots(config, address, signer, roots, dryRun);
const contract = adminSigner
? new ethers.Contract(address, REGISTRY_ABI, adminSigner)
: new ethers.Contract(address, REGISTRY_ABI, signer);
for (const rootType of config.rootTypes) {
const newRoot = getRootForRegistry(roots, config, rootType);
if (!newRoot) continue;
const oldRoot = await getCurrentRoot(contract, rootType);
if (oldRoot === newRoot) {
log(`Skipping ${config.name} ${rootType}: on-chain root already matches`);
continue;
}
let updateFn: keyof ethers.Contract;
if (rootType === 'passportNo') {
updateFn = 'updatePassportNoOfacRoot';
} else if (rootType === 'nameAndDob') {
updateFn = 'updateNameAndDobOfacRoot';
} else {
updateFn = 'updateNameAndYobOfacRoot';
}
allUpdates.push({ config, address, rootType, updateFn, newRoot });
}
}
log(`Total updates submitted: ${totalUpdates}`);
if (totalUpdates === 0) {
log('No on-chain updates needed; skipping tree deployment.');
return;
}
// Step 5: Upload to GCS (versioned path)
// Step 4: Upload to GCS FIRST (before on-chain updates)
// This reduces mismatch window by doing slow uploads before fast on-chain updates
console.log('');
console.log('-'.repeat(70));
console.log(' PHASE: UPLOAD TO GCS');
console.log('-'.repeat(70));
console.log('');
let uploadResult;
if (skipUpload) {
log('Skipping upload (SKIP_UPLOAD=true)');
return;
}
const uploadResult = await uploadToGcs({
bucketName,
basePath,
treesDir,
roots,
timestamp,
dryRun,
});
if (!uploadResult.success) {
console.error('ERROR: GCS upload failed:', uploadResult.error);
console.error('On-chain updates succeeded but file upload failed.');
process.exit(1);
}
log(`Uploaded ${uploadResult.filesUploaded} files to ${uploadResult.versionPath}`);
// Step 6: Update pointer file (atomic switch)
console.log('');
console.log('-'.repeat(70));
console.log(' PHASE: ATOMIC POINTER UPDATE');
console.log('-'.repeat(70));
console.log('');
const pointerResult = await updatePointerFile(
bucketName,
basePath,
uploadResult.versionPath!,
roots,
dryRun
);
if (pointerResult.success) {
await verifyGcsFiles(bucketName, uploadResult.versionPath!, dryRun);
log(
`Mismatch window: ${pointerResult.durationMs}ms (~${(pointerResult.durationMs / 1000).toFixed(1)}s)`
);
if (allUpdates.length === 0 && !dryRun) {
log('No on-chain updates needed; skipping.');
return;
}
} else {
console.error('WARNING: On-chain updates succeeded but pointer update failed.');
console.error(`Error: ${pointerResult.error}`);
console.error(`Files are at: gs://${bucketName}/${uploadResult.versionPath}`);
console.error('Manual pointer update may be required.');
process.exit(1);
uploadResult = await uploadToGcs({
bucketName,
basePath,
treesDir,
roots,
timestamp,
dryRun,
});
if (!uploadResult.success) {
console.error('ERROR: GCS upload failed:', uploadResult.error);
console.error('Cannot proceed with on-chain updates if files are not uploaded.');
process.exit(1);
}
log(`Uploaded ${uploadResult.filesUploaded} files to ${uploadResult.versionPath}`);
}
// Step 5: On-chain updates (after files are uploaded)
console.log('');
console.log('-'.repeat(70));
console.log(' PHASE: ON-CHAIN UPDATES');
console.log('-'.repeat(70));
console.log('');
let totalUpdates = 0;
let onChainConfirmedTime: number | null = null;
if (allUpdates.length === 0) {
log('No on-chain updates needed (all roots already match).');
} else if (dryRun) {
log(`[DRY RUN] Would update ${allUpdates.length} roots across ${new Set(allUpdates.map(u => u.config.name)).size} registries`);
totalUpdates = allUpdates.length;
} else {
log(`Batching ${allUpdates.length} updates across ${new Set(allUpdates.map(u => u.config.name)).size} registries into one block...`);
const contracts = new Map<string, ethers.Contract>();
for (const { address } of allUpdates) {
if (!contracts.has(address)) {
contracts.set(address, adminSigner
? new ethers.Contract(address, REGISTRY_ABI, adminSigner)
: new ethers.Contract(address, REGISTRY_ABI, signer));
}
}
const txs = allUpdates.map(({ address, updateFn, newRoot, config, rootType }) => {
const contract = contracts.get(address)!;
log(`Preparing ${config.name} ${String(updateFn)}`);
return (contract[updateFn] as any)(newRoot);
});
const txPromises = await Promise.all(txs);
log(`Submitted ${txPromises.length} transactions`);
const receipts = await Promise.all(txPromises.map(tx => tx.wait(1)));
const blockNumbers = new Set(receipts.map(r => r.blockNumber));
onChainConfirmedTime = Date.now();
if (blockNumbers.size === 1) {
log(`✅ All ${receipts.length} transactions confirmed in block ${Array.from(blockNumbers)[0]}`);
} else {
log(`⚠️ Transactions confirmed in ${blockNumbers.size} different blocks: ${Array.from(blockNumbers).join(', ')}`);
}
for (let i = 0; i < receipts.length; i++) {
const receipt = receipts[i];
const { config, rootType } = allUpdates[i];
if (receipt?.status !== 1) {
throw new Error(`Update failed for ${config.name} ${rootType}`);
}
log(`✅ ${config.name} ${rootType} updated in block ${receipt.blockNumber}`);
}
totalUpdates = receipts.length;
}
log(`Total updates submitted: ${totalUpdates}`);
// Step 6: Verify on-chain updates
if (!dryRun && totalUpdates > 0) {
console.log('');
console.log('-'.repeat(70));
console.log(' PHASE: VERIFY ON-CHAIN UPDATES');
console.log('-'.repeat(70));
console.log('');
for (const config of REGISTRY_CONFIGS) {
const registryAddress = getRegistryAddress(config.registryKey, network);
if (!registryAddress) continue;
const contract = new ethers.Contract(registryAddress, REGISTRY_ABI, provider);
const verified = await verifyRegistryRoots(config, registryAddress, contract, roots);
if (!verified) {
throw new Error(`Verification failed for ${config.name}`);
}
}
log('✅ All on-chain roots verified successfully');
}
// Step 7: Atomic pointer update (immediately after on-chain confirmation)
if (!skipUpload && uploadResult) {
console.log('');
console.log('-'.repeat(70));
console.log(' PHASE: ATOMIC POINTER UPDATE');
console.log('-'.repeat(70));
console.log('');
const pointerResult = await updatePointerFile(
bucketName,
basePath,
uploadResult.versionPath!,
roots,
dryRun
);
if (pointerResult.success) {
await verifyGcsFiles(bucketName, uploadResult.versionPath!, dryRun);
if (onChainConfirmedTime !== null && totalUpdates > 0 && pointerResult.completedAt) {
const mismatchWindowMs = pointerResult.completedAt - onChainConfirmedTime;
log(
`Mismatch window: ${mismatchWindowMs}ms (~${(mismatchWindowMs / 1000).toFixed(1)}s)`
);
} else {
log(
`Pointer update duration: ${pointerResult.durationMs}ms (~${(pointerResult.durationMs / 1000).toFixed(1)}s)`
);
}
} else {
console.error('WARNING: On-chain updates succeeded but pointer update failed.');
console.error(`Error: ${pointerResult.error}`);
console.error(`Files are at: gs://${bucketName}/${uploadResult.versionPath}`);
console.error('Manual pointer update may be required.');
process.exit(1);
}
}
console.log('');

227
common/scripts/ofac/test-ofac-update.sh
View File

@@ -1,227 +0,0 @@
#!/bin/bash
# OFAC Auto Updater Test Script
# Tests the complete OFAC update pipeline in dry-run mode
set -e
echo "╔══════════════════════════════════════════════════════════════╗"
echo "║ OFAC AUTO UPDATER - TEST SUITE ║"
echo "╚══════════════════════════════════════════════════════════════╝"
echo ""
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Test configuration
TEST_DIR="./test-ofac-$(date +%s)"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
echo -e "${BLUE}📍 Repository root: $REPO_ROOT${NC}"
echo -e "${BLUE}📂 Test directory: $TEST_DIR${NC}"
echo ""
# Create test directory
mkdir -p "$TEST_DIR"
cd "$REPO_ROOT"
# ═══════════════════════════════════════════════════════════════
# TEST 1: Pipeline Test (Download + Parse + Build)
# ═══════════════════════════════════════════════════════════════
echo ""
echo "═══════════════════════════════════════════════════════════════"
echo " TEST 1: OFAC Pipeline (Download → Parse → Build)"
echo "═══════════════════════════════════════════════════════════════"
echo ""
echo -e "${YELLOW}→ Running pipeline...${NC}"
yarn tsx common/scripts/ofac/index.ts --output-dir "$TEST_DIR"
if [ $? -eq 0 ]; then
echo ""
echo -e "${GREEN}✓ Pipeline completed successfully${NC}"
else
echo -e "${RED}✗ Pipeline failed${NC}"
exit 1
fi
# Verify outputs
echo ""
echo -e "${YELLOW}→ Verifying outputs...${NC}"
EXPECTED_FILES=(
"raw/sdn-latest.xml"
"inputs/names.json"
"inputs/passports.json"
"outputs/passportNoAndNationalitySMT.json"
"outputs/nameAndDobSMT.json"
"outputs/nameAndYobSMT.json"
"outputs/nameAndDobSMT_ID.json"
"outputs/nameAndYobSMT_ID.json"
"outputs/nameAndDobSMT_AADHAAR.json"
"outputs/nameAndYobSMT_AADHAAR.json"
"outputs/roots.json"
"outputs/latest-roots.json"
)
MISSING_FILES=0
for file in "${EXPECTED_FILES[@]}"; do
if [ -f "$TEST_DIR/$file" ]; then
SIZE=$(du -h "$TEST_DIR/$file" | cut -f1)
echo -e "${GREEN} ✓ $file ($SIZE)${NC}"
else
echo -e "${RED} ✗ Missing: $file${NC}"
MISSING_FILES=$((MISSING_FILES + 1))
fi
done
if [ $MISSING_FILES -eq 0 ]; then
echo ""
echo -e "${GREEN}✓ All expected files created${NC}"
else
echo ""
echo -e "${RED}✗ $MISSING_FILES files missing${NC}"
exit 1
fi
# ═══════════════════════════════════════════════════════════════
# TEST 2: Roots Validation
# ═══════════════════════════════════════════════════════════════
echo ""
echo "═══════════════════════════════════════════════════════════════"
echo " TEST 2: Roots Validation"
echo "═══════════════════════════════════════════════════════════════"
echo ""
echo -e "${YELLOW}→ Checking roots.json structure...${NC}"
# Check if roots.json is valid JSON and has expected keys
ROOTS_FILE="$TEST_DIR/outputs/roots.json"
if ! jq empty "$ROOTS_FILE" 2>/dev/null; then
echo -e "${RED}✗ roots.json is not valid JSON${NC}"
exit 1
fi
EXPECTED_ROOTS=(
"passport_no_and_nationality"
"name_and_dob"
"name_and_yob"
"name_and_dob_id_card"
"name_and_yob_id_card"
"aadhaar_name_and_dob"
"aadhaar_name_and_yob"
)
MISSING_ROOTS=0
for root in "${EXPECTED_ROOTS[@]}"; do
if jq -e ".$root" "$ROOTS_FILE" > /dev/null 2>&1; then
ROOT_VALUE=$(jq -r ".$root" "$ROOTS_FILE")
echo -e "${GREEN} ✓ $root: ${ROOT_VALUE:0:20}...${NC}"
else
echo -e "${RED} ✗ Missing root: $root${NC}"
MISSING_ROOTS=$((MISSING_ROOTS + 1))
fi
done
if [ $MISSING_ROOTS -eq 0 ]; then
echo ""
echo -e "${GREEN}✓ All roots present and valid${NC}"
else
echo ""
echo -e "${RED}✗ $MISSING_ROOTS roots missing${NC}"
exit 1
fi
# ═══════════════════════════════════════════════════════════════
# TEST 3: Tree Statistics
# ═══════════════════════════════════════════════════════════════
echo ""
echo "═══════════════════════════════════════════════════════════════"
echo " TEST 3: Tree Statistics"
echo "═══════════════════════════════════════════════════════════════"
echo ""
NAMES_COUNT=$(jq '. | length' "$TEST_DIR/inputs/names.json")
PASSPORTS_COUNT=$(jq '. | length' "$TEST_DIR/inputs/passports.json")
echo -e "${BLUE} Names entries: $NAMES_COUNT${NC}"
echo -e "${BLUE} Passport entries: $PASSPORTS_COUNT${NC}"
# Check if numbers are reasonable (should be thousands)
if [ "$NAMES_COUNT" -lt 1000 ]; then
echo -e "${RED}✗ Too few names entries (expected > 1000)${NC}"
exit 1
fi
if [ "$PASSPORTS_COUNT" -lt 100 ]; then
echo -e "${RED}✗ Too few passport entries (expected > 100)${NC}"
exit 1
fi
echo ""
echo -e "${GREEN}✓ Entry counts look reasonable${NC}"
# ═══════════════════════════════════════════════════════════════
# TEST 4: Dry-Run Full Update (if credentials available)
# ═══════════════════════════════════════════════════════════════
echo ""
echo "═══════════════════════════════════════════════════════════════"
echo " TEST 4: Dry-Run Full Update"
echo "═══════════════════════════════════════════════════════════════"
echo ""
if [ -z "$PRIVATE_KEY" ]; then
echo -e "${YELLOW}⚠ PRIVATE_KEY not set - skipping dry-run test${NC}"
echo -e "${YELLOW} To run: export PRIVATE_KEY=0x...${NC}"
else
echo -e "${YELLOW}→ Running dry-run update...${NC}"
export NETWORK="${NETWORK:-celo-sepolia}"
export DRY_RUN=true
export OFAC_DATA_DIR="$TEST_DIR"
export GCS_BUCKET_NAME="${GCS_BUCKET_NAME:-self-ofac-test}"
# Run with existing data (skip download)
yarn tsx common/scripts/ofac/runOfacAutoUpdate.ts
if [ $? -eq 0 ]; then
echo ""
echo -e "${GREEN}✓ Dry-run completed successfully${NC}"
else
echo -e "${RED}✗ Dry-run failed${NC}"
exit 1
fi
fi
# ═══════════════════════════════════════════════════════════════
# SUMMARY
# ═══════════════════════════════════════════════════════════════
echo ""
echo "╔══════════════════════════════════════════════════════════════╗"
echo "║ TEST SUMMARY ║"
echo "╚══════════════════════════════════════════════════════════════╝"
echo ""
echo -e "${GREEN}✓ All tests passed!${NC}"
echo ""
echo "Test artifacts saved to: $TEST_DIR"
echo ""
echo "Next steps:"
echo " 1. Review generated files in $TEST_DIR"
echo " 2. Test with real GCS credentials (set GOOGLE_APPLICATION_CREDENTIALS)"
echo " 3. Test on staging network (celo-sepolia)"
echo " 4. Deploy to production when ready"
echo ""
# Optional: Clean up test directory
read -p "Delete test directory? (y/N) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
rm -rf "$TEST_DIR"
echo -e "${GREEN}✓ Test directory deleted${NC}"
else
echo -e "${BLUE}ℹ Test directory preserved: $TEST_DIR${NC}"
fi

103
common/scripts/ofac/test-onchain-only.sh Executable file
View File

@@ -0,0 +1,103 @@
#!/bin/bash
# Quick test script for on-chain updates only (skips tree building)
# Uses existing tree files and tests TEE on-chain calls + GCS uploads
set -e
echo "╔══════════════════════════════════════════════════════════════╗"
echo "║ OFAC ON-CHAIN UPDATE TEST (SKIP TREE BUILDING) ║"
echo "╚══════════════════════════════════════════════════════════════╝"
echo ""
# Configuration
FORK_RPC="${FORK_RPC:-http://localhost:8545}"
DATA_DIR="${DATA_DIR:-./test-ofac-data}"
ROOTS_PATH="${ROOTS_PATH:-${DATA_DIR}/outputs/roots.json}"
echo "🔧 Configuration:"
echo " Fork RPC: $FORK_RPC"
echo " Roots file: $ROOTS_PATH"
echo ""
# Check prerequisites
if [ ! -f "$ROOTS_PATH" ]; then
echo "❌ ERROR: Roots file not found: $ROOTS_PATH"
echo ""
echo " Run the full pipeline first to generate trees:"
echo " ./common/scripts/ofac/test-with-fork.sh"
echo ""
exit 1
fi
# Check if fork is running
echo "🔍 Checking if local fork is running..."
if ! curl -s -X POST -H "Content-Type: application/json" \
--data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
"$FORK_RPC" > /dev/null 2>&1; then
echo "❌ ERROR: Local fork not running at $FORK_RPC"
echo ""
echo "Start a fork first:"
echo " anvil --fork-url https://forno.celo.org --port 8545"
echo ""
exit 1
fi
echo "✅ Fork is running at $FORK_RPC"
echo ""
# Use test private key
TEST_PRIVATE_KEY="${TEST_PRIVATE_KEY:-0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80}"
TEE_ADDRESS=$(cast wallet address --private-key "$TEST_PRIVATE_KEY" 2>/dev/null || echo "unknown")
echo "📋 Test Details:"
echo " TEE Address: $TEE_ADDRESS"
echo ""
echo "This will:"
echo " ✓ Skip pipeline (use existing trees)"
echo " ✓ Impersonate OPERATIONS_ROLE holder to call update functions"
echo " ✓ Batch all transactions in one block"
echo " ✓ Verify on-chain roots match expected values"
echo " ✓ Upload tree files to Google Cloud Storage bucket"
echo ""
read -p "Press Enter to start test, or Ctrl+C to cancel..."
echo ""
# Run the test
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
cd "$REPO_ROOT"
# Use credentials from ofac folder if not set, or allow override via env var
DEFAULT_CREDS="${SCRIPT_DIR}/gcs-credentials.json"
# Use default if env var is not set, or if it's set to a placeholder value
if [ -z "$GOOGLE_APPLICATION_CREDENTIALS" ] || [ "$GOOGLE_APPLICATION_CREDENTIALS" = "/path/to/key.json" ]; then
GOOGLE_CREDS="$DEFAULT_CREDS"
else
GOOGLE_CREDS="$GOOGLE_APPLICATION_CREDENTIALS"
fi
# Check if credentials file exists
if [ ! -f "$GOOGLE_CREDS" ]; then
echo "❌ ERROR: GCS credentials file not found: $GOOGLE_CREDS"
echo ""
echo " Expected location: $DEFAULT_CREDS"
echo " Or set GOOGLE_APPLICATION_CREDENTIALS environment variable"
exit 1
fi
PRIVATE_KEY="$TEST_PRIVATE_KEY" \
NETWORK="celo" \
RPC_URL="$FORK_RPC" \
ROOTS_PATH="$ROOTS_PATH" \
TREES_DIR="${DATA_DIR}/outputs" \
GOOGLE_APPLICATION_CREDENTIALS="$GOOGLE_CREDS" \
SKIP_PIPELINE=true \
yarn tsx common/scripts/ofac/runOfacAutoUpdate.ts
echo ""
echo "╔══════════════════════════════════════════════════════════════╗"
echo "║ ON-CHAIN TEST COMPLETED ║"
echo "╚══════════════════════════════════════════════════════════════╝"
echo ""

141
common/scripts/ofac/test-with-fork.sh Executable file
View File

@@ -0,0 +1,141 @@
#!/bin/bash
# OFAC Auto Updater - Local Fork Test
# Tests with Anvil/Hardhat fork of Celo mainnet registry contracts
# Uses REAL GCS staging bucket for uploads
set -e
echo "╔══════════════════════════════════════════════════════════════╗"
echo "║ OFAC AUTO UPDATER - CELO FORK + REAL GCS UPLOAD ║"
echo "╚══════════════════════════════════════════════════════════════╝"
echo ""
# Configuration
FORK_NETWORK="${FORK_NETWORK:-celo}" # Must be 'celo' to use real registry addresses
FORK_RPC="${FORK_RPC:-http://localhost:8545}" # Local fork RPC
DATA_DIR="${DATA_DIR:-./test-ofac-data}"
GCS_BUCKET="${GCS_BUCKET:-self-ofac-test}"
GCS_BASE_PATH="${GCS_BASE_PATH:-ofac}"
# Use credentials from ofac folder if not set, or allow override via env var
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEFAULT_CREDS="${SCRIPT_DIR}/gcs-credentials.json"
# Use default if env var is not set, or if it's set to a placeholder value
if [ -z "$GOOGLE_APPLICATION_CREDENTIALS" ] || [ "$GOOGLE_APPLICATION_CREDENTIALS" = "/path/to/key.json" ]; then
GOOGLE_CREDS="$DEFAULT_CREDS"
else
GOOGLE_CREDS="$GOOGLE_APPLICATION_CREDENTIALS"
fi
echo "🔧 Configuration:"
echo " Forking: $FORK_NETWORK (using real Celo registry addresses)"
echo " Fork RPC: $FORK_RPC"
echo " Data Dir: $DATA_DIR"
echo " GCS Bucket: $GCS_BUCKET"
echo " GCS Path: $GCS_BASE_PATH"
echo " GCS Credentials: $GOOGLE_CREDS"
echo ""
# Validate network
if [ "$FORK_NETWORK" != "celo" ]; then
echo "⚠️ WARNING: Fork network is '$FORK_NETWORK', but registry addresses are configured for 'celo'"
echo " The script will use Celo registry addresses regardless"
fi
# Check prerequisites
if [ ! -f "$GOOGLE_CREDS" ]; then
echo "❌ ERROR: GCS credentials file not found: $GOOGLE_CREDS"
echo ""
echo " Expected location: $DEFAULT_CREDS"
echo " Or set GOOGLE_APPLICATION_CREDENTIALS environment variable"
exit 1
fi
# Check if fork is running
echo "🔍 Checking if local fork is running..."
if ! curl -s -X POST -H "Content-Type: application/json" \
--data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
"$FORK_RPC" > /dev/null 2>&1; then
echo "❌ ERROR: Local fork not running at $FORK_RPC"
echo ""
echo "Start a fork first:"
echo " 1. Using Anvil (Foundry) - RECOMMENDED:"
echo " anvil --fork-url https://forno.celo.org --port 8545"
echo " (This forks at the latest block automatically)"
echo ""
echo " 2. Using Hardhat:"
echo " npx hardhat node --fork https://forno.celo.org"
echo ""
echo "⚠️ IMPORTANT: The fork must include the real Celo registry contracts:"
echo " - IdentityRegistry: 0x37F5CB8cB1f6B00aa768D8aA99F1A9289802A968"
echo " - IdentityRegistryIdCard: 0xeAD1E6Ec29c1f3D33a0662f253a3a94D189566E1"
echo " - IdentityRegistryAadhaar: 0xd603Fa8C8f4694E8DD1DcE1f27C0C3fc91e32Ac4"
echo ""
exit 1
fi
echo "💡 TIP: If roots already match on-chain, restart your fork to reset state:"
echo " 1. Stop current fork (Ctrl+C)"
echo " 2. Restart: anvil --fork-url https://forno.celo.org --port 8545"
echo ""
echo "✅ Fork is running at $FORK_RPC"
echo ""
TEST_PRIVATE_KEY="${TEST_PRIVATE_KEY:-0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80}"
TEE_ADDRESS=$(cast wallet address --private-key "$TEST_PRIVATE_KEY" 2>/dev/null || echo "unknown")
echo "📋 Test Details:"
echo " TEE Address: $TEE_ADDRESS"
echo ""
echo "This will perform:"
echo " ✓ Pipeline: Download + Parse + Build trees"
echo " ✓ Contract calls on Anvil LOCAL FORK (real Celo registry contracts)"
echo " ✓ Impersonate OPERATIONS_ROLE holder to call update functions"
echo " ✓ REAL GCS uploads (to staging bucket: $GCS_BUCKET)"
echo " ✓ Measure mismatch window"
echo ""
echo "⚠️ NOTE: This uses REAL GCS uploads. Files will be uploaded to:"
echo " gs://$GCS_BUCKET/$GCS_BASE_PATH/"
echo ""
read -p "Press Enter to start fork test, or Ctrl+C to cancel..."
echo ""
# Run the test
# Get repo root (3 levels up from script dir: scripts/ofac -> scripts -> common -> repo root)
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
cd "$REPO_ROOT"
PRIVATE_KEY="$TEST_PRIVATE_KEY" \
NETWORK="celo" \
RPC_URL="$FORK_RPC" \
OFAC_DATA_DIR="$DATA_DIR" \
GCS_BUCKET_NAME="$GCS_BUCKET" \
GCS_BASE_PATH="$GCS_BASE_PATH" \
GOOGLE_APPLICATION_CREDENTIALS="$GOOGLE_CREDS" \
yarn tsx common/scripts/ofac/runOfacAutoUpdate.ts
echo ""
echo "╔══════════════════════════════════════════════════════════════╗"
echo "║ FORK TEST COMPLETED ║"
echo "╚══════════════════════════════════════════════════════════════╝"
echo ""
echo "📊 Results:"
echo ""
echo " 📦 Check GCS Bucket:"
echo " Web UI: https://console.cloud.google.com/storage/browser/$GCS_BUCKET/$GCS_BASE_PATH?project=ofac-upload-test"
echo ""
if command -v gsutil &> /dev/null; then
echo " Command line:"
echo " gsutil ls gs://$GCS_BUCKET/$GCS_BASE_PATH/"
echo " gsutil cat gs://$GCS_BUCKET/$GCS_BASE_PATH/current.json"
else
echo " 💡 Install gsutil for command-line access:"
echo " brew install google-cloud-sdk"
echo " Or: https://cloud.google.com/sdk/docs/install"
fi
echo ""
echo " 🔗 Fork transactions logged in your terminal above"
echo ""

229
common/src/utils/trees.ts
View File

@@ -51,9 +51,6 @@ export function buildAadhaarSMT(field: any[], treetype: string): [number, number
for (let i = 0; i < field.length; i++) {
const entry = field[i];
if (i !== 0) {
console.log('Processing', treetype, 'number', i, 'out of', field.length);
}
let leaf = BigInt(0);
let reverse_leaf = BigInt(0);
@@ -66,14 +63,16 @@ export function buildAadhaarSMT(field: any[], treetype: string): [number, number
}
if (leaf == BigInt(0) || tree.createProof(leaf).membership) {
console.log('This entry already exists in the tree, skipping...');
// Suppressed verbose log - duplicates are expected
// console.log('This entry already exists in the tree, skipping...');
continue;
}
count += 1;
tree.add(leaf, BigInt(1));
if (reverse_leaf == BigInt(0) || tree.createProof(reverse_leaf).membership) {
console.log('This entry already exists in the tree, skipping...');
// Suppressed verbose log - duplicates are expected
// console.log('This entry already exists in the tree, skipping...');
continue;
}
tree.add(reverse_leaf, BigInt(1));
@@ -83,11 +82,101 @@ export function buildAadhaarSMT(field: any[], treetype: string): [number, number
return [count, performance.now() - startTime, tree];
}
// Async version of buildAadhaarSMT with progress updates
export async function buildAadhaarSMTAsync(
field: any[],
treetype: string,
monitor?: any,
treeIndex?: number
): Promise<[number, number, SMT]> {
let count = 0;
const startTime = performance.now();
const hash2 = (childNodes: ChildNodes) =>
childNodes.length === 2 ? poseidon2(childNodes) : poseidon3(childNodes);
const tree = new SMT(hash2, true);
// Yield immediately to allow other trees to start
await yieldToEventLoop();
await new Promise(resolve => setTimeout(resolve, 10));
for (let i = 0; i < field.length; i++) {
const entry = field[i];
let leaf = BigInt(0);
let reverse_leaf = BigInt(0);
if (treetype == 'name_and_dob') {
leaf = processNameAndDobAadhaar(entry, i);
reverse_leaf = processNameAndDobAadhaar(entry, i, true);
} else if (treetype == 'name_and_yob') {
leaf = processNameAndYobAadhaar(entry, i);
reverse_leaf = processNameAndYobAadhaar(entry, i, true);
}
if (leaf == BigInt(0) || tree.createProof(leaf).membership) {
// Update progress even if entry is skipped
if (monitor && treeIndex !== undefined) {
monitor.updateTree(treeIndex, {
entriesAdded: count,
entriesProcessed: i + 1,
});
}
continue;
}
count += 1;
tree.add(leaf, BigInt(1));
if (reverse_leaf == BigInt(0) || tree.createProof(reverse_leaf).membership) {
// Update progress after adding first leaf
if (monitor && treeIndex !== undefined) {
monitor.updateTree(treeIndex, {
entriesAdded: count,
entriesProcessed: i + 1,
});
await yieldToEventLoop(); // Yield to allow render
}
continue;
}
tree.add(reverse_leaf, BigInt(1));
count += 1;
// Update progress monitor EVERY entry AFTER processing
// Yield every entry when monitor is present to allow smooth rendering
if (monitor && treeIndex !== undefined) {
monitor.updateTree(treeIndex, {
entriesAdded: count,
entriesProcessed: i + 1,
});
// Yield every entry to allow monitor render interval to pick up updates
// This ensures smooth progress display (1, 2, 3...) instead of jumps
await yieldToEventLoop();
} else if (i !== 0 && i % 50 === 0) {
// If no monitor, only yield every 50 entries for parallelism
await yieldToEventLoop();
}
}
// Final progress update
if (monitor && treeIndex !== undefined) {
monitor.updateTree(treeIndex, {
entriesAdded: count,
entriesProcessed: field.length,
});
}
return [count, performance.now() - startTime, tree];
}
// SMT trees for 3 levels of matching :
// 1. Passport Number and Nationality tree : level 3 (Absolute Match)
// 2. Name and date of birth combo tree : level 2 (High Probability Match)
// 3. Name and year of birth combo tree : level 1 (Partial Match)
// NEW: ID card specific trees
// Helper to yield control to event loop for parallel execution
const yieldToEventLoop = () => new Promise(resolve => setImmediate(resolve));
export function buildSMT(field: any[], treetype: string): [number, number, SMT] {
let count = 0;
const startTime = performance.now();
@@ -99,10 +188,11 @@ export function buildSMT(field: any[], treetype: string): [number, number, SMT]
for (let i = 0; i < field.length; i++) {
const entry = field[i];
// Suppressed verbose log - dashboard shows progress instead
// Optimization: Log progress less frequently
if (i !== 0 && i % 100 === 0) {
console.log('Processing', treetype, 'number', i, 'out of', field.length);
}
// if (i !== 0 && i % 100 === 0) {
// console.log('Processing', treetype, 'number', i, 'out of', field.length);
// }
let leaf = BigInt(0);
// Determine document type based on treetype for name processing
@@ -143,8 +233,104 @@ export function buildSMT(field: any[], treetype: string): [number, number, SMT]
tree.add(leaf, BigInt(1));
}
console.log('Total', treetype, 'entries added:', count, 'out of', field.length);
console.log(treetype, 'tree built in', (performance.now() - startTime).toFixed(2), 'ms');
return [count, performance.now() - startTime, tree];
}
// Async wrapper for parallel execution - yields control every N iterations
export async function buildSMTAsync(
field: any[],
treetype: string,
monitor?: any,
treeIndex?: number
): Promise<[number, number, SMT]> {
let count = 0;
const startTime = performance.now();
const hash2 = (childNodes: ChildNodes) =>
childNodes.length === 2 ? poseidon2(childNodes) : poseidon3(childNodes);
const tree = new SMT(hash2, true);
// CRITICAL: Yield immediately to allow other promises in Promise.all to start
await yieldToEventLoop();
// Also yield after a tiny delay to ensure all promises have started
await new Promise(resolve => setTimeout(resolve, 10));
for (let i = 0; i < field.length; i++) {
const entry = field[i];
let leaf = BigInt(0);
let docType: 'passport' | 'id_card' = 'passport';
if (treetype.endsWith('_id_card')) {
docType = 'id_card';
}
if (treetype == 'passport_no_and_nationality') {
leaf = processPassportNoAndNationality(entry.Pass_No, entry.Pass_Country, i);
} else if (treetype == 'name_and_dob') {
leaf = processNameAndDob(entry, i, 'passport');
} else if (treetype == 'name_and_yob') {
leaf = processNameAndYob(entry, i, 'passport');
} else if (treetype == 'name_and_dob_id_card') {
leaf = processNameAndDob(entry, i, 'id_card');
} else if (treetype == 'name_and_yob_id_card') {
leaf = processNameAndYob(entry, i, 'id_card');
} else if (treetype == 'country') {
const keys = Object.keys(entry);
leaf = processCountry(keys[0], entry[keys[0]], i);
}
if (leaf == BigInt(0)) {
// Still update progress even if entry is skipped
if (monitor && treeIndex !== undefined) {
monitor.updateTree(treeIndex, {
entriesAdded: count,
entriesProcessed: i + 1
});
await yieldToEventLoop(); // Yield to allow render
}
continue;
}
if (tree.createProof(leaf).membership) {
// Still update progress even if entry is duplicate
if (monitor && treeIndex !== undefined) {
monitor.updateTree(treeIndex, {
entriesAdded: count,
entriesProcessed: i + 1
});
await yieldToEventLoop(); // Yield to allow render
}
continue;
}
count += 1;
tree.add(leaf, BigInt(1));
// Update progress monitor EVERY entry AFTER processing
// Yield every entry to allow monitor to render smoothly
if (monitor && treeIndex !== undefined) {
monitor.updateTree(treeIndex, {
entriesAdded: count,
entriesProcessed: i + 1
});
// Yield every entry to allow monitor render interval to pick up updates
// This ensures smooth progress display (1, 2, 3...) instead of jumps
await yieldToEventLoop();
} else if (i !== 0 && i % 50 === 0) {
// If no monitor, only yield every 50 entries for parallelism
await yieldToEventLoop();
}
}
// Final progress update
if (monitor && treeIndex !== undefined) {
monitor.updateTree(treeIndex, {
entriesAdded: count,
entriesProcessed: field.length // All entries processed
});
}
return [count, performance.now() - startTime, tree];
}
@@ -228,7 +414,8 @@ export function getCountryLeaf(
i?: number
): bigint {
if (country_by.length !== 3 || country_to.length !== 3) {
console.log('parsed passport length is not 3:', i, country_to, country_by);
// Suppressed verbose debug log
// console.log('parsed passport length is not 3:', i, country_to, country_by);
return;
}
try {
@@ -319,7 +506,8 @@ function processPassportNoAndNationality(
index: number
): bigint {
if (passno.length > 9) {
console.log('passport number length is greater than 9:', index, passno);
// Suppressed verbose debug log - invalid passport numbers are skipped
// console.log('passport number length is greater than 9:', index, passno);
} else if (passno.length < 9) {
while (passno.length != 9) {
passno += '<';
@@ -328,10 +516,12 @@ function processPassportNoAndNationality(
const countryCode = getCountryCode(nationality);
if (!countryCode) {
console.log('Error getting country code', index, nationality);
// Suppressed verbose debug log - invalid country codes are skipped
// console.log('Error getting country code', index, nationality);
return BigInt(0);
}
console.log('nationality and countryCode', nationality, countryCode);
// Suppressed verbose debug log
// console.log('nationality and countryCode', nationality, countryCode);
const leaf = getPassportNumberAndNationalityLeaf(
stringToAsciiBigIntArray(passno),
@@ -339,7 +529,8 @@ function processPassportNoAndNationality(
index
);
if (!leaf) {
console.log('Error creating leaf value', index, passno, nationality);
// Suppressed verbose debug log - invalid entries are skipped
// console.log('Error creating leaf value', index, passno, nationality);
return BigInt(0);
}
return leaf;
@@ -464,7 +655,7 @@ function processName(
arr += '<';
}
}
console.log('arr', arr, 'arr.length', arr.length);
// Debug log removed for clean parallel tree building output
const nameArr = stringToAsciiBigIntArray(arr);
// getNameLeaf will select the correct Poseidon hash based on nameArr.length
return getNameLeaf(nameArr, i);
@@ -515,7 +706,8 @@ function processCountry(country1: string, country2: string, i: number) {
const leaf = getCountryLeaf(arr, arr2, i);
if (!leaf) {
console.log('Error creating leaf value', i, country1, country2);
// Suppressed verbose debug log - invalid entries are skipped
// console.log('Error creating leaf value', i, country1, country2);
return BigInt(0);
}
return leaf;
@@ -690,7 +882,8 @@ export function getPassportNumberAndNationalityLeaf(
i?: number
): bigint {
if (passport.length !== 9) {
console.log('parsed passport length is not 9:', i, passport);
// Suppressed verbose debug log - invalid passport numbers are skipped
// console.log('parsed passport length is not 9:', i, passport);
return;
}
if (nationality.length !== 3) {