chore: Initial commit

Co-authored-by: Franklin Delehelle <franklin.delehelle@odena.eu>
Co-authored-by: Alexandre Belling <alexandrebelling8@gmail.com>
Co-authored-by: Pedro Novais <jpvnovais@gmail.com>
Co-authored-by: Roman Vaseev <4833306+Filter94@users.noreply.github.com>
Co-authored-by: Bradley Bown <bradbown@googlemail.com>
Co-authored-by: Victorien Gauch <85494462+VGau@users.noreply.github.com>
Co-authored-by: Nikolai Golub <nikolai.golub@consensys.net>
Co-authored-by: The Dark Jester <thedarkjester@users.noreply.github.com>
Co-authored-by: jonesho <81145364+jonesho@users.noreply.github.com>
Co-authored-by: Gaurav Ahuja <gauravahuja9@gmail.com>
Co-authored-by: Azam Soleimanian <49027816+Soleimani193@users.noreply.github.com>
Co-authored-by: Andrei A <andrei.alexandru@consensys.net>
Co-authored-by: Arijit Dutta <37040536+arijitdutta67@users.noreply.github.com>
Co-authored-by: Gautam Botrel <gautam.botrel@gmail.com>
Co-authored-by: Ivo Kubjas <ivo.kubjas@consensys.net>
Co-authored-by: gusiri <dreamerty@postech.ac.kr>
Co-authored-by: FlorianHuc <florian.huc@gmail.com>
Co-authored-by: Arya Tabaie <arya.pourtabatabaie@gmail.com>
Co-authored-by: Julink <julien.fontanel@consensys.net>
Co-authored-by: Bogdan Ursu <bogdanursuoffice@gmail.com>
Co-authored-by: Jakub Trąd <jakubtrad@gmail.com>
Co-authored-by: Alessandro Sforzin <alessandro.sforzin@consensys.net>
Co-authored-by: Olivier Bégassat <olivier.begassat.cours@gmail.com>
Co-authored-by: Steve Huang <97596526+stevehuangc7s@users.noreply.github.com>
Co-authored-by: bkolad <blazejkolad@gmail.com>
Co-authored-by: fadyabuhatoum1 <139905934+fadyabuhatoum1@users.noreply.github.com>
Co-authored-by: Blas Rodriguez Irizar <rodrigblas@gmail.com>
Co-authored-by: Eduardo Andrade <eduardofandrade@gmail.com>
Co-authored-by: Ivo Kubjas <tsimmm@gmail.com>
Co-authored-by: Ludcour <ludovic.courcelas@consensys.net>
Co-authored-by: m4sterbunny <harrie.bickle@consensys.net>
Co-authored-by: Alex Panayi <145478258+alexandrospanayi@users.noreply.github.com>
Co-authored-by: Diana Borbe - ConsenSys <diana.borbe@consensys.net>
Co-authored-by: ThomasPiellard <thomas.piellard@gmail.com>
This commit is contained in:
Julien Marchand
2024-07-31 18:16:31 +02:00
commit a001342170
2702 changed files with 695073 additions and 0 deletions

View File

@@ -0,0 +1,3 @@
dist
node_modules
coverage

View File

@@ -0,0 +1,15 @@
module.exports = {
extends: "../../.eslintrc.js",
env: {
commonjs: true,
es2021: true,
node: true,
jest: true,
},
parserOptions: {
sourceType: "module",
},
rules: {
"prettier/prettier": "error",
},
};

View File

@@ -0,0 +1,3 @@
dist
node_modules
coverage

View File

@@ -0,0 +1,3 @@
module.exports = {
...require("../../.prettierrc.js"),
};

View File

@@ -0,0 +1,13 @@
Copyright 2024 Consensys Software Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

View File

@@ -0,0 +1,92 @@
# @consensys/linea-native-libs
`@consensys/linea-native-libs` is a Node.js library that provides an interface to native Go libraries using the `ffi-napi` and `ref-napi` packages.
It provides the following Go libraries wrapper:
- `GoNativeCompressor`: This class allows you to initialize the transaction compressor, check for errors, and get the worst compressed transaction size for a given RLP-encoded transaction.
## Installation
Install the required npm package:
```bash
npm install @consensys/linea-native-libs
```
## Usage
### Compressor library
#### Importing the Class
```javascript
import { GoNativeCompressor } from '@consensys/linea-native-libs';
```
#### Initializing the Compressor
Create an instance of `GoNativeCompressor` by providing a data size limit:
```javascript
const dataLimit = 1024; // Example data limit
const compressor = new GoNativeCompressor(dataLimit);
```
#### Getting the Compressed Transaction Size
To get the worst compressed transaction size for a given RLP-encoded transaction:
```javascript
const rlpEncodedTransaction = new Uint8Array([...]); // Your RLP-encoded transaction
const compressedTxSize = compressor.getCompressedTxSize(rlpEncodedTransaction);
console.log(`Compressed Transaction Size: ${compressedTxSize}`);
```
#### Methods
#### `constructor(dataLimit: number)`
- **Parameters:**
- `dataLimit`: The data limit for the compressor.
- **Description:** Initializes the compressor with the given data limit and loads the native library.
#### `getCompressedTxSize(rlpEncodedTransaction: Uint8Array): number`
- **Parameters:**
- `rlpEncodedTransaction`: The RLP-encoded transaction as a `Uint8Array`.
- **Returns:** The worst compressed transaction size as a `number`.
- **Description:** Returns the worst compressed transaction size for the given RLP-encoded transaction. Throws an error if compression fails.
#### `getError(): string | null`
- **Returns:** The error message as a `string` or `null` if no error.
- **Description:** Retrieves the last error message from the native library.
#### Error Handling
If an error occurs during initialization or compression, an `Error` will be thrown with a descriptive message.
#### Example
```javascript
import { GoNativeCompressor } from '@consensys/linea-native-libs';
const dataLimit = 1024;
const compressor = new GoNativeCompressor(dataLimit);
const rlpEncodedTransaction = new Uint8Array([...]);
try {
const compressedTxSize = compressor.getCompressedTxSize(rlpEncodedTransaction);
console.log(`Compressed Transaction Size: ${compressedTxSize}`);
} catch (error) {
console.error(`Error: ${error.message}`);
}
```
## License
This project is licensed under the Apache-2.0 License.

View File

@@ -0,0 +1,21 @@
module.exports = {
collectCoverage: true,
collectCoverageFrom: ["./src/**/*.ts"],
coverageDirectory: "coverage",
coverageProvider: "babel",
coverageReporters: ["html", "json-summary", "text"],
coverageThreshold: {
global: {
branches: 85.71,
functions: 100,
lines: 95.23,
statements: 95.34,
},
},
preset: "ts-jest",
resetMocks: true,
restoreMocks: true,
testTimeout: 2500,
testPathIgnorePatterns: ["src/scripts", "src/index.ts"],
coveragePathIgnorePatterns: ["src/scripts", "src/index.ts"],
};

View File

@@ -0,0 +1,56 @@
{
"name": "@consensys/linea-native-libs",
"version": "0.1.0",
"description": "Linea native libs",
"exports": {
".": {
"import": {
"types": "./dist/index.d.mts",
"default": "./dist/index.mjs"
},
"require": {
"types": "./dist/index.d.cts",
"default": "./dist/index.cjs"
}
},
"./package.json": "./package.json"
},
"main": "./dist/index.cjs",
"module": "./dist/index.mjs",
"types": "./dist/index.d.cts",
"files": [
"dist"
],
"scripts": {
"postbuild": "bash ./src/scripts/build.sh",
"build": "ts-bridge --project tsconfig.build.json --clean",
"clean": "rimraf dist build coverage node_modules",
"test": "jest --bail --detectOpenHandles --forceExit && jest-it-up",
"lint:ts": "npx eslint '**/*.ts'",
"lint:ts:fix": "npx eslint --fix '**/*.ts'",
"prettier": "prettier -c '**/*.ts'",
"prettier:fix": "prettier -w '**/*.ts'",
"lint:fix": "pnpm run lint:ts:fix && pnpm run prettier:fix"
},
"author": "",
"license": "MIT",
"devDependencies": {
"@jest/globals": "^29.7.0",
"@ts-bridge/cli": "^0.1.4",
"@ts-bridge/shims": "^0.1.1",
"@types/ffi-napi": "^4.0.10",
"@types/jest": "^29.5.12",
"@types/ref-napi": "^3.0.12",
"@types/unzipper": "^0.10.9",
"dotenv": "^16.4.5",
"ethers": "^6.13.1",
"jest": "^29.7.0",
"jest-it-up": "^3.1.0",
"node-fetch": "^3.3.2",
"ts-jest": "^29.1.5",
"unzipper": "^0.12.1"
},
"dependencies": {
"koffi": "^2.9.0"
}
}

View File

@@ -0,0 +1,72 @@
import { KoffiFunction, load } from "koffi";
import path from "path";
import { getCompressorLibPath } from "./helpers";
const COMPRESSOR_DICT_PATH = path.join(__dirname, "./lib/compressor_dict.bin");
/**
* Class representing a Go Native Compressor.
*/
export class GoNativeCompressor {
private initFunc: KoffiFunction;
private errorFunc: KoffiFunction;
private worstCompressedTxSizeFunc: KoffiFunction;
/**
* Creates an instance of GoNativeCompressor.
* @param {number} dataLimit - The data limit for the compressor.
* @throws {Error} Throws an error if initialization fails.
*/
constructor(dataLimit: number) {
const compressorLibPath = getCompressorLibPath();
const lib = load(compressorLibPath);
this.initFunc = lib.func("Init", "bool", ["int", "char*"]);
this.errorFunc = lib.func("Error", "char*", []);
this.worstCompressedTxSizeFunc = lib.func("WorstCompressedTxSize", "int", ["char*", "int"]);
this.init(dataLimit);
}
/**
* Initializes the compressor with the given data limit.
* @param {number} dataLimit - The data limit for the compressor.
* @returns {boolean} Returns `true` if initialization is successful.
* @throws {Error} Throws an error if initialization fails.
* @private
*/
private init(dataLimit: number): boolean {
const isInitSuccess = this.initFunc(dataLimit, COMPRESSOR_DICT_PATH);
if (!isInitSuccess) {
throw new Error("Error while initialization the compressor library.");
}
return isInitSuccess;
}
/**
* Gets the worst compressed transaction size for a given RLP-encoded transaction.
* @param {Uint8Array} rlpEncodedTransaction - The RLP-encoded transaction.
* @returns {number} The worst compressed transaction size.
* @throws {Error} Throws an error if compression fails.
*/
public getCompressedTxSize(rlpEncodedTransaction: Uint8Array): number {
const compressedTxSize = this.worstCompressedTxSizeFunc(rlpEncodedTransaction, rlpEncodedTransaction.byteLength);
const error = this.getError();
if (error) {
throw new Error(`Error while compressing the transaction: ${error}`);
}
return compressedTxSize;
}
/**
* Retrieves the last error message from the native library.
* @returns {string | null} The error message or null if no error.
* @private
*/
private getError(): string | null {
try {
return this.errorFunc();
} catch (e) {
return null;
}
}
}

View File

@@ -0,0 +1,46 @@
import { describe, it, beforeEach, expect } from "@jest/globals";
import { Transaction, Wallet, ethers } from "ethers";
import { GoNativeCompressor } from "../GoNativeCompressor";
const TEST_ADDRESS = "0x0000000000000000000000000000000000000001";
const TEST_PRIVATE_KEY = "0x0000000000000000000000000000000000000000000000000000000000000001";
describe("GoNativeCompressor", () => {
const dataLimit = 800_000;
let compressor: GoNativeCompressor;
beforeEach(() => {
compressor = new GoNativeCompressor(dataLimit);
});
describe("getCompressedTxSize", () => {
it("Should throw an error if an error occured during tx compression", () => {
const transaction = Transaction.from({
to: TEST_ADDRESS,
value: ethers.parseEther("2"),
});
const rlpEncodedTransaction = ethers.encodeRlp(transaction.unsignedSerialized);
const input = ethers.getBytes(rlpEncodedTransaction);
expect(() => compressor.getCompressedTxSize(input)).toThrow(
"Error while compressing the transaction: rlp: too few elements for types.DynamicFeeTx",
);
});
it("Should return compressed tx size", async () => {
const transaction = Transaction.from({
to: TEST_ADDRESS,
value: ethers.parseEther("2"),
});
const signer = new Wallet(TEST_PRIVATE_KEY);
const encodedSignedTx = await signer.signTransaction(transaction);
const rlpEncodedTransaction = ethers.encodeRlp(encodedSignedTx);
const input = ethers.getBytes(rlpEncodedTransaction);
const compressedTxSize = compressor.getCompressedTxSize(input);
expect(compressedTxSize).toStrictEqual(43);
});
});
});

View File

@@ -0,0 +1,58 @@
import { describe, it, expect } from "@jest/globals";
import os from "os";
import fs, { Dirent } from "fs";
import path from "path";
import { getCompressorLibPath } from "..";
describe("Helpers", () => {
afterEach(() => {
jest.clearAllMocks();
});
describe("getCompressorLibPath", () => {
it("Should throw an error when the os platform is not supported", () => {
jest.mock("os");
const platform = "android";
jest.spyOn(os, "platform").mockReturnValueOnce(platform);
expect(() => getCompressorLibPath()).toThrow(`Unsupported platform: ${platform}`);
});
it("Should throw an error when the resources folder does not exist", () => {
jest.spyOn(fs, "existsSync").mockReturnValueOnce(false);
const platform = os.platform();
const arch = os.arch();
const dirPath = path.resolve("src", "compressor", "lib", `${platform}-${arch}`);
expect(() => getCompressorLibPath()).toThrow(`Directory does not exist: ${dirPath}`);
});
it("Should throw an error when the lib file does not exist", () => {
jest.spyOn(fs, "existsSync").mockReturnValueOnce(true);
jest.spyOn(fs, "readdirSync").mockReturnValueOnce([]);
const platform = os.platform();
const arch = os.arch();
const dirPath = path.resolve("src", "compressor", "lib", `${platform}-${arch}`);
expect(() => getCompressorLibPath()).toThrow(`No matching library file found in directory: ${dirPath}`);
});
it("Should return lib compressor", async () => {
jest.mock("os");
jest.spyOn(fs, "existsSync").mockReturnValueOnce(true);
const filename = "blob_compressor_v0.1.0.dylib";
jest.spyOn(fs, "readdirSync").mockReturnValueOnce([filename] as unknown as Dirent[]);
const platform = "darwin";
const arch = "arm64";
jest.spyOn(os, "platform").mockReturnValueOnce(platform);
jest.spyOn(os, "arch").mockReturnValueOnce("arm64");
const dirPath = path.resolve("src", "compressor", "lib", `${platform}-${arch}`);
const libPath = getCompressorLibPath();
expect(libPath).toStrictEqual(`${dirPath}/${filename}`);
});
});
});

View File

@@ -0,0 +1,47 @@
import { existsSync, readdirSync } from "fs";
import os from "os";
import path from "path";
/**
* Mapping of OS platforms to their respective compressor library file extensions.
* @type {Record<string, string>}
*/
const OS_COMPRESSOR_LIB_EXTENSION_MAPPING: Record<string, string> = {
darwin: ".dylib",
linux: ".so",
win32: ".exe",
};
/**
* Gets the path to the compressor library based on the current OS platform and architecture.
*
* @returns {string} The absolute path to the compressor library file.
* @throws {Error} Throws an error if the platform is unsupported, the directory does not exist, or no matching library file is found.
*/
export function getCompressorLibPath(): string {
const platform = os.platform().toString();
const arch = os.arch();
const directory = `${platform}-${arch}`;
const fileExtension = OS_COMPRESSOR_LIB_EXTENSION_MAPPING[platform];
if (!fileExtension) {
throw new Error(`Unsupported platform: ${platform}`);
}
const dirPath = path.join(__dirname, "..", "lib", directory);
if (!existsSync(dirPath)) {
throw new Error(`Directory does not exist: ${dirPath}`);
}
const files = readdirSync(dirPath);
const libFile = files.find((file) => file.startsWith("blob_compressor") && file.endsWith(fileExtension));
if (!libFile) {
throw new Error(`No matching library file found in directory: ${dirPath}`);
}
return path.resolve(dirPath, libFile);
}

View File

@@ -0,0 +1 @@
export { GoNativeCompressor } from "./compressor/GoNativeCompressor";

View File

@@ -0,0 +1,15 @@
#!/bin/bash
set -e
error_handler() {
echo "Error occurred in script at line: $1"
exit 1
}
trap 'error_handler $LINENO' ERR
node ./dist/scripts/build.mjs
cp -R src/compressor/lib/ dist/compressor/lib
rm -rf ./dist/scripts
echo "Build script executed successfully."

View File

@@ -0,0 +1,146 @@
/* eslint-disable @typescript-eslint/no-explicit-any */
import fetch from "node-fetch";
import * as fs from "fs";
import * as path from "path";
import { Open } from "unzipper";
import { exec } from "child_process";
import { getBuildConfig } from "./config";
async function downloadAndParseJson(url: string, headers: Record<string, string> = {}): Promise<any> {
const response = await fetch(url, {
method: "GET",
headers: {
Accept: "application/json",
...headers,
},
});
if (!response.ok) {
throw new Error(`Failed to load JSON from ${url}. HTTP error code: ${response.status}`);
}
return await response.json();
}
async function getReleaseAssetUrl(authToken: string, nativeLibReleaseTag: string): Promise<string> {
const urlStr = "https://api.github.com/repos/ConsenSys/zkevm-monorepo/releases";
const json = await downloadAndParseJson(urlStr, { Authorization: `token ${authToken}` });
const release = json.find((release: any) => release.tag_name === nativeLibReleaseTag);
if (!release) {
const releases = json.map((release: any) => release.tag_name);
throw new Error(`Release ${nativeLibReleaseTag} not found! releases: ${releases}`);
}
if (release.assets.length === 0) {
throw new Error(`Release ${nativeLibReleaseTag} has no assets!`);
}
const asset = release.assets.find((asset: any) => asset.name.includes(nativeLibReleaseTag));
return `https://${authToken}:@api.github.com/repos/Consensys/zkevm-monorepo/releases/assets/${asset.id}`;
}
async function downloadFileUsingCurl(authToken: string, url: string, outputFilePath: string): Promise<string> {
const outputDirectory = path.dirname(outputFilePath);
// Ensure the output directory exists
fs.mkdirSync(outputDirectory, { recursive: true });
const command = `curl -L -H 'Accept:application/octet-stream' -u ${authToken}: -o ${outputFilePath} ${url}`;
return new Promise((resolve, reject) => {
exec(command, (error: any, _: any, stderr: any) => {
if (error) {
reject(new Error(`Failed to download file using curl: ${stderr}`));
} else {
resolve(outputFilePath);
}
});
});
}
const architectureResourceDirMapping: Record<string, string> = {
darwin_arm64: "darwin-arm64",
darwin_x86_64: "darwin-x64",
linux_arm64: "linux-arm64",
linux_amd64: "linux-x64",
linux_x86_64: "linux-x64",
};
async function downloadReleaseAsset(authToken: string, nativeLibReleaseTag: string): Promise<string> {
const assetReleaseUrl = await getReleaseAssetUrl(authToken, nativeLibReleaseTag);
const fileName = `${nativeLibReleaseTag}.zip`;
const destPath = path.resolve("build", fileName);
console.log(`Downloading ${fileName} from ${assetReleaseUrl} to ${destPath}`);
return await downloadFileUsingCurl(authToken, assetReleaseUrl, destPath);
}
function getBinaryResourceFolder(libFile: string): string {
const destResource = Object.entries(architectureResourceDirMapping).find(([key]) => libFile.includes(key));
if (!destResource) {
throw new Error(`No architecture found for ${libFile}`);
}
return destResource[1];
}
function getBinaryResourceFileName(libFile: string, libName: string): string {
const versionPattern = /v\d+\.\d+\.\d+/;
const match = libFile.match(versionPattern);
const version = match ? match[0] : null;
const extension = path.extname(libFile);
return `${libName}_${version}${extension}`;
}
async function downloadReleaseAndExtractToResources(
authToken: string,
nativeLibReleaseTag: string,
libName: string,
): Promise<void> {
const outputFile = await downloadReleaseAsset(authToken, nativeLibReleaseTag);
if (!fs.existsSync(outputFile)) {
throw new Error(`Output file ${outputFile} does not exist`);
}
const extractPath = path.resolve("build", nativeLibReleaseTag);
const zipFile = await Open.file(outputFile);
await zipFile.extract({ path: extractPath, concurrency: 5 });
console.log("Extraction complete");
const files = fs.readdirSync(extractPath);
if (files.length === 0) {
throw new Error("No files found in the extracted zip file.");
}
for (const file of files) {
if (file.includes(libName) && (file.endsWith(".so") || file.endsWith(".dylib"))) {
const destResourceDir = getBinaryResourceFolder(file);
const destResourceFileName = getBinaryResourceFileName(file, libName);
const destPath = path.resolve("src", "compressor", "lib", destResourceDir);
fs.mkdirSync(destPath, { recursive: true });
fs.copyFileSync(path.join(extractPath, file), path.join(destPath, destResourceFileName));
console.log(`Copying ${file} to ${path.join(destPath, destResourceFileName)}`);
}
}
}
async function fetchLib(authToken: string, nativeLibReleaseTag: string, libName: string): Promise<void> {
await downloadReleaseAndExtractToResources(authToken, nativeLibReleaseTag, libName);
}
async function main() {
const { authToken, nativeLibReleaseTag } = getBuildConfig();
await fetchLib(authToken, nativeLibReleaseTag, "blob_compressor");
}
main()
.then(() => {
process.exit(0);
})
.catch((error) => {
console.error(error);
process.exit(1);
});

View File

@@ -0,0 +1,27 @@
import { config } from "dotenv";
config();
type BuildConfig = {
authToken: string;
nativeLibReleaseTag: string;
};
export function getBuildConfig(): BuildConfig {
const authToken = process.env.GITHUB_API_ACCESS_TOKEN;
if (!authToken) {
throw new Error("GITHUB_API_ACCESS_TOKEN environment variable is not set");
}
const nativeLibReleaseTag = process.env.NATIVE_LIBS_RELEASE_TAG;
if (!nativeLibReleaseTag) {
throw new Error("NATIVE_LIBS_RELEASE_TAG environment variable is not set");
}
return {
authToken,
nativeLibReleaseTag,
};
}

View File

@@ -0,0 +1,19 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"declaration": true,
"declarationMap": true,
"emitDeclarationOnly": true,
"inlineSources": true,
"noEmit": false,
"outDir": "dist",
"rootDir": "src",
"sourceMap": true
},
"include": ["./src/**/*.ts"],
"exclude": [
"./src/helpers/**/*",
"./src/**/__tests__/**/*",
"./src/**/*.test.ts",
]
}

View File

@@ -0,0 +1,17 @@
{
"extends": "../../tsconfig.json",
"compilerOptions": {
"esModuleInterop": true,
"exactOptionalPropertyTypes": true,
"forceConsistentCasingInFileNames": true,
"lib": ["ES2020"],
"module": "CommonJS",
"moduleResolution": "node",
"noEmit": true,
"noErrorTruncation": true,
"noUncheckedIndexedAccess": true,
"strict": true,
"target": "es2020"
},
"exclude": ["./dist", "**/node_modules"]
}