92 Commits

Author SHA1 Message Date
FoxxMD
00680494a3 feat(filter): Add createdAt activity criteria matching
* May be a string or array of strings. Passes if any expression matches
  * Value may be a convience day-of-week value (mon, tues, wed...)
  * or a cron expression
2022-09-15 11:22:04 -04:00
FoxxMD
54755dc480 refactor(guest)!: Migrate invites to be owned by the server
It makes more sense for the CM instance that will actually have a bot added to it to own the invite for that bot.

* (BC) Move Invite into server entity mappings and rename to BotInvite
  * Add guests and initialConfig (future use)
  * BC -- Existing invites need to have instance defined to be used
  * BC -- Cache-based invite storage has been REMOVED
  * BC -- Removed inviteMaxAge config property (for now)
* Add SubredditInvite entity for future use
* Force/implement server (api) having a defined friendly name. This is used to determine which invites in a DB belong to which server instance
  * If not defined in config it is generated at random and then written to config
* Refactor how invites are retrieved and parsed client-side -- CRUD using server api
  * BC -- Invite URL structure has changed from ?invite=id to /invite/id...
* Added better UI for migration redirect
  * Show all reachable instances in redirect page header
  * Error page also shows reachable instances in page header
2022-08-11 14:47:45 -04:00
FoxxMD
73302b718e chore: Bump version for 0.11.4 release 2022-07-15 09:27:04 -04:00
FoxxMD
16cb28cb72 feat(report): Implement report tracking #78
* Store reports with likely creation time to DB
* Use some trickery and caching in Manager to get likely report creation time
2022-07-12 17:29:28 -04:00
FoxxMD
0ffddaac9e Revert "chore: Bump got and sharp versions for dependabot fixes"
This reverts commit fbf328d90f.
2022-07-11 08:56:44 -04:00
FoxxMD
4ddf6ddf26 Revert "chore: Bump passport version for dependabot fixes"
This reverts commit 7f61c190ea.
2022-07-11 08:56:41 -04:00
FoxxMD
7f61c190ea chore: Bump passport version for dependabot fixes 2022-07-08 12:34:54 -04:00
FoxxMD
fbf328d90f chore: Bump got and sharp versions for dependabot fixes 2022-07-08 12:33:03 -04:00
FoxxMD
192c1659a0 feat(database): Initial influx db implementation #84 2022-07-06 15:14:58 -04:00
FoxxMD
4bfb57a6cf chore: Update connect-typeorm dependency
Moved to 2.0.0 b/c it now supports typeorm >= 0.3.0
2022-07-05 09:43:25 -04:00
FoxxMD
8c0755c8c2 working when no includes in config 2022-06-30 12:39:57 -04:00
FoxxMD
7542947029 feat(sentiment): More improvements to sentiment and language processing
* Implement separate language detection functionality
* Clearer/simpler sentiment processing
* Add languageHints to help coerce low confidence language detection
* Add test cases for lang detection, sentiment detection, and sentiment tests
* Fix neutral range -- was not using normalized score range
2022-05-25 12:59:37 -04:00
FoxxMD
2d02434e7e feat(sentiment): Better use of NLPjs and more robust language detection
* Build NLP container ad-hoc so only supported languages are included from npm
* Use vader/wink as heuristics for detecting language when content is very short
* Add languageHint option for sentiment config to make coercing a confident sentiment easier
* Refactor lang processing to fail the sentiment test rather than throwing an error when language is not support/not confident -- provides more insight into outcome
2022-05-24 16:34:37 -04:00
FoxxMD
1c94548947 chore: Update TS version and move into dev dependencies
Not needed to actually run the app after it has been built so moving to dev-dependencies
2022-05-24 10:26:46 -04:00
FoxxMD
90b8f481ec chore: Remove unused dependencies #88 2022-05-24 09:33:29 -04:00
FoxxMD
af3b917b57 feat(logging): Move log formatting for ui into client #91
* Use browserify to include logform functions and triple-beam symbols in client js
* Implement default log transform function in client js
* Remove formatted message and transport data from non-streaming log data sent to client
2022-05-19 12:11:33 -04:00
FoxxMD
4411d1a413 feat(sentiment): Improve scoring and analysis
* Implement english-only scoring with wink https://github.com/winkjs/wink-sentiment (AFINN, emojis)
* Implement english-only scoring with NLP.js https://github.com/axa-group/nlp.js/blob/master/docs/v3/sentiment-analysis.md (AFINN, Senticon, Pattern)
* Refactor language processing into standalone functions for future use
* Add limited multi-langauge support
  * Can run sentiment with NLP.js on english, german, spanish, and french
* Normalize all scores to range between -1 and +1
* Improve score accuracy by averaging all scores
2022-05-11 16:09:29 -04:00
FoxxMD
522ba33377 feat(rule): Implement sentiment rule using VADER #83 2022-05-11 12:39:23 -04:00
FoxxMD
9cb5c4fe38 feat: More improvements for window functionality and dependency cleanup
* Add deprecation warnings to rules when building if properties should be migrated to window
* Add `debug` option to window to increase verbosity of filter logging. Default to false.
* Fix object assigment when building hash key for window filters
* Further cleanup for circular dependencies by moving some filter and logging functions into respective files
2022-05-10 12:49:56 -04:00
FoxxMD
951fab1070 docs: Add deprecation notices to window-related filters on rules 2022-05-09 16:23:01 -04:00
FoxxMD
d250c30f14 refactor(database): Improve event query performance
* Query for non-hydrated events to get ids then get fully hydrated objects using only ids -- dramatically improves performance
* Remove typeorm-pagination due to non-optimized count/select approach (should use typeorm getManyAndCount)
  * Also removes dup typeorm dependency
2022-04-19 12:54:18 -04:00
FoxxMD
c81703720b refactor(session): Migrate session storage into typeorm database 2022-04-05 10:56:47 -04:00
FoxxMD
f9727f58a9 Bump minimum node version to LTS 2022-03-29 14:46:40 -04:00
FoxxMD
c292da47df feat(database): Support better-sqlite3 as a database 2022-03-29 14:17:33 -04:00
FoxxMD
5a7738b5b2 feat(database): Implement event pagination 2022-03-24 16:08:45 -04:00
FoxxMD
fc3479fb8a refactor(database): Update typeorm/dependencies and fix naming strategy
* Use updated logger adapter so that typeorm can be updated
* Update to typeorm 0.3.3
* Refactor naming strategy to be what i want it to be
2022-03-24 12:44:49 -04:00
FoxxMD
6c8bbdff1f feat(database): Make all non-updateable entities have random, friendly ids
* Replace any incremented, generated number id with a random id from nanoid
* Replace UUID with random id from nanoid
2022-03-23 14:48:17 -04:00
FoxxMD
5971ef7565 refactor(database): Partially working implementation for actioned event using database
* Reworked many mappings to fix cascades and relationships
* Add constructors to entities where useful
* Renamed many db entities classes to be different than interface names
* Re-generated initial migration and added convenience npm script for running it
* Added ormconfig for typeorm 3 cli
* Create all run/check/rule/action premises (static info) on manager creation
2022-03-22 15:07:38 -04:00
FoxxMD
bced882c75 refactor(db): Update typeorm to 0.3.0 2022-03-18 15:34:55 -04:00
FoxxMD
b561fa6b3c refactor(database): Partial conflict resolution with edge
Still need to refactor interfaces/code related to storing actioned events
2022-03-17 10:00:21 -04:00
FoxxMD
d82032073b Merge branch 'edge' into databaseSupport
# Conflicts:
#	package-lock.json
#	package.json
#	src/App.ts
#	src/Common/interfaces.ts
#	src/ConfigBuilder.ts
#	src/Subreddit/Manager.ts
#	src/Subreddit/SubredditResources.ts
#	src/Utils/loggerFactory.ts
#	src/Web/Client/index.ts
#	src/Web/Server/server.ts
#	src/Web/assets/views/events.ejs
#	src/index.ts
#	src/util.ts
2022-03-17 09:41:54 -04:00
FoxxMD
935d983626 feat(client): Implement snoowrap retry on timeout errors
Makes CM less sensitive to random blips in reddit API by enabling snoowrap to retry some network-issue related error codes alongside status codes

* Add timeoutCodes as an operator configurable array of error codes that is passed to extended snoowrap
* Patch snoowrap to check for timeout codes on request error and use retry logic if found
* Additionally, add retryErrorCodes (status codes) to operator configuration
2022-03-16 12:31:04 -04:00
FoxxMD
8991797d35 feat(testing): WIP added initial testing framework and some util tests
Using mocha, chai, and nyc

* tests for parsing string for numeric value comparison
* tests for parsing string for durations and duration comparisons
* tests for parsing reddit entity (subreddit/user) from string
* tests for parsing submission/comment id from reddit permalink string
* tests for initial config parsing/merging

Still can't get nyc to get coverage for everything in src using "all" -- causes reporting to show 0 for everything??
2022-03-09 10:59:43 -05:00
FoxxMD
f7a7e817f9 feat(cache): Implement cache migrations
* Store migration state in cache instance
* Migrate on default cache init or private cache init
* Implement first migration to deal with run structure in actioned events
2022-02-08 17:00:18 -05:00
FoxxMD
0293928a99 feat(cache): Implement cache key manipulation based on key pattern
* Implement glob pattern or regex as argument
* Implement scan search for redis for efficiency otherwise iterate keys using generic function
* Implement cache reset based on passed item from action -- reset item crit for activities, author crit for users, and overwrite any cached activity
2022-02-08 13:01:09 -05:00
FoxxMD
41038b9bcd feat(logging): Implement richer errors everywhere
* Use ErrorWithCause so we can get and print a chain of error causes
* Make reddit error response in stack trace more readable by replacing them with a "translated" parent response and add them as the cause
* Properly handle error formatting for winston by looking at shape of log object for error rather than testing instanceof (see comments in errorAwareFormat)
* Fix formatting in web interface for log lines with white-space pre css and properly splitting timestamp from rest of the message
2022-01-27 16:27:03 -05:00
FoxxMD
d2e1b5019f chore: Update packages 2022-01-21 13:02:31 -05:00
FoxxMD
780e5c185e refactor(author filter): Strongly structure comparison/matching data for more consistent manipulation and output
* Use interface for comparison results at both criteria property level and criteria level
* Implement summary functions to build string results of comparisons
* Output all comparisons to debug and provide summaries to verbose (when applicable)
2022-01-20 14:08:54 -05:00
FoxxMD
5811af0342 feat(config): Refactor config parsing to preserve comments and enable writing
* use node-comment and yaml@next to keep comment information intact
* store ast/source version of parsed config for operator
* implement generic yaml/json operator config classes to keep everything organized and simplify marshalling source to js/string
* refactor file parsing and json/yaml parsing to have better single responsibility
2022-01-17 15:51:43 -05:00
FoxxMD
3bb9c2cee3 feat(databse): Add missing typeorm requirements and logger adapter
* ormconfig using default sqlite connection to generate migrations
* add winston adapter to provide logger to typeorm
2022-01-14 15:05:00 -05:00
FoxxMD
bee3935270 feat(database): Add database backend capability
* Add typeorm dependency with backend drivers for sqljs and, optionally, postgres/mongodb/mysql/mariadb
* Add operator configuration structure for global database connection
* Implement config parsing and defaults for sqljs db location or in-memory fallback

#66
2022-01-04 12:12:24 -05:00
FoxxMD
f081598da6 feat(repost): Improve text matching and default values
* Use 3 different matching algorithms using the highest score out of the three
* Weight score based on length of the sentence
* Increase minimum number of words to 3
* Enforce min word count on external (youtube) comments
2021-12-18 17:42:43 -05:00
FoxxMD
0e395792db refactor: Remove unusused ts-auto-guard package
Has @ts-morph as a dependency which is a huge package
2021-12-17 14:36:03 -05:00
FoxxMD
dcbeb784e8 refactor: Remove set-random-interval
* depended on and always downloaded an entire, older typescript version (even with production install) -- was not necessary for one function
* refactor project to use newer TS version (specify any type for catch blocks to fix compiler errors)
2021-12-17 14:25:48 -05:00
FoxxMD
aeaeb6ce27 refactor(docker): Remove monaco editor local dependency to reduce image size
Wasn't necessary to have it installed locally (can use CDN) and is not a part of the core experience
2021-12-17 14:00:54 -05:00
FoxxMD
a48a850c98 Update winston repository package format/information 2021-12-16 14:58:43 -05:00
FoxxMD
b8369a9e9f Package bump for security
https://github.com/advisories/GHSA-93q8-gq69-wqmw
2021-12-14 12:06:48 -05:00
FoxxMD
a8c7b1dac9 Interim implementation of repost rule 2021-11-30 20:42:40 -05:00
FoxxMD
00b9d87cdc Remove unused types package 2021-10-12 11:56:29 -04:00
FoxxMD
463a4dc0eb feat(image): Implement perceptual hashing image comparison
* Generate perceptual hashes using blockhash-js of images that can be cache/stored
* Take advantage of reddit thumbnail code ImageData to hash lower-res to begin with (but represent full url)
* Refactor imageDetection config so hash and pixel approaches have different configs
* Cache phash results to reduce reddit traffic and speed up performance

Addresses cpu/memory issues with pixel comparison. Allow pixel for finer comparisons if needed using tiered thresholds. Closes #26
2021-10-11 15:28:48 -04:00