feat(package.json): add @dqbd/tiktoken dependency
fix(tokenCount.ts): uncomment Tiktoken import and encoding, and return token length instead of an approximation
chore(tsconfig.json): change target to ESNext
The @dqbd/tiktoken dependency was removed from the package.json file. This dependency was not being used in the project and was therefore removed to reduce the size of the project. The target in the tsconfig.json file was changed from ES2020 to ESNext to allow for the use of the latest ECMAScript features.
* 1.1.23
* 1.1.24
* feat(package.json): add @dqbd/tiktoken dependency
refactor(generateCommitMessageFromGitDiff.ts): add tokenCount function to count the number of tokens in a string
refactor(generateCommitMessageFromGitDiff.ts): change the way the length of INIT_MESSAGES_PROMPT is calculated to use tokenCount function
refactor(generateCommitMessageFromGitDiff.ts): change the way the length of diff is calculated to use tokenCount function
refactor(generateCommitMessageFromGitDiff.ts): rename function parameter from diff to fileDiff and update function calls accordingly
feat(generateCommitMessageFromGitDiff.ts): add tokenCount function to count tokens in fileDiff and use it to check if fileDiff is bigger than MAX_REQ_TOKENS
feat(utils): add tokenCount function to count the number of tokens in a string
refactor(utils/mergeStrings.ts): use tokenCount function to count the number of tokens in a string instead of checking the length of the concatenated string
---------
Co-authored-by: di-sukharev <dim.sukharev@gmail.com>