github/opencommit
1
1
Fork 0
mirror of https://github.com/di-sukharev/opencommit.git synced 2026-01-30 16:08:14 -05:00
Code Issues Packages Projects Releases Wiki Activity

fix(githook.ts): capitalize OpenCommit in console output

Browse Source 
chore(githook.ts): remove unused import
refactor(tokenCount.ts): simplify token count calculation
The console output now correctly capitalizes OpenCommit for consistency with the project name. The unused import of fileURLToPath is removed. The tokenCount function is refactored to simplify the calculation of the token count. The Tiktoken library is no longer used, and the token count is now calculated based on the length of the content divided by an average token length of 2.7 characters.
This commit is contained in:
di-sukharev
2023-03-29 11:26:05 +08:00
parent 853662acc4
commit 1b70de1d20
2 changed files with 13 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
src/utils/tokenCount.ts
View File

@@ -1,14 +1,14 @@
import { Tiktoken } from '@dqbd/tiktoken/lite';
import cl100k_base from '@dqbd/tiktoken/encoders/cl100k_base.json' assert { type: 'json' };
// import { Tiktoken } from '@dqbd/tiktoken/lite';
// import cl100k_base from '@dqbd/tiktoken/encoders/cl100k_base.json' assert { type: 'json' };
export function tokenCount(content: string): number {
const encoding = new Tiktoken(
cl100k_base.bpe_ranks,
cl100k_base.special_tokens,
cl100k_base.pat_str
);
const tokens = encoding.encode(content);
encoding.free();
// const encoding = new Tiktoken(
// cl100k_base.bpe_ranks,
// cl100k_base.special_tokens,
// cl100k_base.pat_str
// );
// const tokens = encoding.encode(content);
// encoding.free();
return tokens.length;
return content.length / 2.7;
}