1117 Commits

Author SHA1 Message Date
FoxxMD
d418bf50eb Some cleanup on activity window docs 2022-05-12 12:22:48 -04:00
FoxxMD
5c8e73fee0 docs(activity window): Overhaul activity window docs
* Improve existing documentation with simpler language
* Organize/add table of contents
* Add window lifecycle diagram
* Add documentation for filter/fetch properties #55
2022-05-12 12:07:26 -04:00
FoxxMD
4411d1a413 feat(sentiment): Improve scoring and analysis
* Implement english-only scoring with wink https://github.com/winkjs/wink-sentiment (AFINN, emojis)
* Implement english-only scoring with NLP.js https://github.com/axa-group/nlp.js/blob/master/docs/v3/sentiment-analysis.md (AFINN, Senticon, Pattern)
* Refactor language processing into standalone functions for future use
* Add limited multi-langauge support
  * Can run sentiment with NLP.js on english, german, spanish, and french
* Normalize all scores to range between -1 and +1
* Improve score accuracy by averaging all scores
2022-05-11 16:09:29 -04:00
FoxxMD
c919532aac Merge branch 'edge' into sentimentRule 2022-05-11 12:42:15 -04:00
FoxxMD
676a3917e7 feat(database): Add planned rule/action types to database types
Getting ahead of the curve to avoid requiring migrations when these are released
2022-05-11 12:42:02 -04:00
FoxxMD
522ba33377 feat(rule): Implement sentiment rule using VADER #83 2022-05-11 12:39:23 -04:00
FoxxMD
3a18cc219f refactor(file organization): Move some comparison functionality into own file 2022-05-11 09:36:11 -04:00
FoxxMD
9cb5c4fe38 feat: More improvements for window functionality and dependency cleanup
* Add deprecation warnings to rules when building if properties should be migrated to window
* Add `debug` option to window to increase verbosity of filter logging. Default to false.
* Fix object assigment when building hash key for window filters
* Further cleanup for circular dependencies by moving some filter and logging functions into respective files
2022-05-10 12:49:56 -04:00
FoxxMD
57700b5e76 chore: Add nvmrc 2022-05-10 12:46:08 -04:00
FoxxMD
52ae16be1c refactor: Reduce circular dependencies 2022-05-09 20:12:08 -04:00
FoxxMD
951fab1070 docs: Add deprecation notices to window-related filters on rules 2022-05-09 16:23:01 -04:00
FoxxMD
0fd57af67e feat(window): More improvements for new window usage and caching
* Fix list function passed from author activities convenience method
* Move author history caching into main activity fetching function
* Do a better job at rehydrating snoowrap objects from cache data -- set as fetched, substitute relationships for non-fetching objects, and remove listing related objects
* Cache key for results based on window and pre-filter only -- post filter can be done after fetching cached results (Should save api calls!)
2022-05-09 15:34:19 -04:00
FoxxMD
3ad5b26be6 fix(ts): Fix TS issue with typings in same folder as regular files preventing schema generation
* Rename interface/types folder
* Move typigns back into folder with same name
2022-05-09 11:09:33 -04:00
FoxxMD
d20c2becc8 feat(window): Refactor window to support subreddit and item filtering, everywhere
* Move some interfaces and types into own files to breakup huge interfaces file
* Refactor window shape for config and "full" usage in app to support subreddit/item filtering
* Refactor author activities into resources class so we can take advantage of caching on subreddit/item filter results
2022-05-09 11:04:08 -04:00
FoxxMD
002bedf4b8 feat: Rebase in old subreddit window filter improvement
* subreddit filtering on window can now use include or exclude
* subreddit resources uses batch/cache retrieval
* temp fix to keep string subreddit name parity check in recent activity self inclusion logic
2022-05-05 23:03:18 -04:00
FoxxMD
11d32de9c5 chore: Bump dependencies for security fixes 2022-05-05 12:42:13 -04:00
FoxxMD
50303dfeb0 fix: Fix dryRun state handling for dispatched actions
Fixes scenario where a dispatched activity does not inherit DR state from currently processing activity

* Add dryrun state to dispatch activity data in app and database
  * Use general DR state for dispatched activity rather than DR for dispatched action
* Defer to explicitly defined DR in Task data when manager/queue handles task
2022-05-05 10:13:21 -04:00
FoxxMD
9b15c88547 fix: Handle more use cases when user is suspended, deleted, or shadowbanned
* Implement a "not serious" property for these errors so we don't report as an error to manager since they not an actual problem with the api or CM
* Check for [deleted] user name before trying to fetch an author
2022-05-04 12:57:16 -04:00
FoxxMD
4042305f49 fix: Fix rule result rendering when rule name is null 2022-05-04 12:02:48 -04:00
FoxxMD
ea9a867b3a fix: Handle scenario where author is suspended
* Throw friendly error when fetching activities
* Fail author criteria that are not present if author is suspended
2022-05-04 11:56:50 -04:00
FoxxMD
ff2415b024 docs: Add more links to main readme 2022-04-27 10:15:40 -04:00
FoxxMD
cf553cad59 docs: Re-organize docs structure and cleanup main readme 2022-04-27 10:09:13 -04:00
FoxxMD
1df9cd2b2a docs: Finish subreddit configuration docs refactor 2022-04-27 09:44:17 -04:00
FoxxMD
344dcc9879 refactor: New docs for subreddit configurations
More features documented with better examples. Not done yet...
2022-04-26 15:54:20 -04:00
FoxxMD
9ddebeb7fd fix: Don't clear queue meta on queue start
Can accidentally clear delayed items added to meta before queue is started
2022-04-26 09:12:51 -04:00
FoxxMD
3451cd5d3a fix(filter): Fix source assignment overwriting itemIs criteria
Need to deep clone itemIs criteria to prevent accidental assignment of new source
2022-04-25 12:37:15 -04:00
FoxxMD
2fa2606950 feat: Add option to disable delays when running activities as user
* Convenience for testing dispatched actions with delays
* Fix nullable hydrated dispatched action properties
2022-04-25 12:36:13 -04:00
FoxxMD
fc24b05eb3 fix: Fix links in documentation 2022-04-22 15:30:52 -04:00
FoxxMD
bfc7f8a508 docs: Rewrite and improve operator-related documentation
* Break out documentation into more standalone docs and reorganize into an operator folder
* Remove outdated information on adding bot
* Add additional information on docker install
* Make configuration more opinionated for "recommend" approach
* Add docs on database and caching
* Rewrite operator getting started guide to be more concise
2022-04-22 15:24:56 -04:00
FoxxMD
5a6f6b2680 Merge branch 'edge' into databaseSupport 2022-04-21 17:45:56 -04:00
FoxxMD
b7c314e9f5 fix(regex): Remove undefined matches from regex parsing
Could happen if there is a capture group that does not capture but entire expression does match
2022-04-21 17:43:55 -04:00
FoxxMD
24e9295aca feat(database): Implement event retention policy per subreddit
* user-configurable retention period (number of events OR duration) at operator, bot, subreddit override, and subreddit config level
* run database cleanup using retention policy on startup and every 30 minutes
* show retention policy in UI on manager overview
2022-04-21 15:03:02 -04:00
FoxxMD
0a1306130d feat(check): Implement post behavior to determine if event should be stored
* Extends postBehavior interface to allow specifying different record output options (database, influx)
* Can specify for *either* post behavior which enables storing events that were not triggered
2022-04-21 12:08:10 -04:00
FoxxMD
942d4fb2f2 feat(filter)!: Change behavior for rules that fail filter criteria tests
Instead of "skipping" the rule will now fail. This aligns Rule behavior with how filters work through the rest of CM which should reduce cognitive load and development effort.

If the skipping behavior is still desired a user can use a RuleSet with OR condition to achieve the same effect.
2022-04-21 10:19:54 -04:00
FoxxMD
fd173b6a2f feat(database): Implement user-configurable time series statistics
* Collect same stats as all time but on at a specified frequency
* Frequency is configurable at operator, bot, and subreddit level
* Operator and bot level frequency can have an enforced minimum
2022-04-20 13:44:35 -04:00
FoxxMD
c039a13fe6 fix(check): Fix how cached rules are counted for stats 2022-04-20 13:35:28 -04:00
FoxxMD
6d208b3daf fix(manager): Fix manager state parsing on init
* Add types for managing running state
* Only use invokee if state is stopped/user, otherwise always set to stopped/system
2022-04-20 11:46:23 -04:00
FoxxMD
65c159081e feat(database): Add related and author search params to event query 2022-04-19 14:44:09 -04:00
FoxxMD
d250c30f14 refactor(database): Improve event query performance
* Query for non-hydrated events to get ids then get fully hydrated objects using only ids -- dramatically improves performance
* Remove typeorm-pagination due to non-optimized count/select approach (should use typeorm getManyAndCount)
  * Also removes dup typeorm dependency
2022-04-19 12:54:18 -04:00
FoxxMD
c701e741af fix(database): Fix join/select for event querying 2022-04-19 10:02:48 -04:00
FoxxMD
3051b4af4e feat(ui): Improve events ux
* Make title a link to "default" events view
* Fix event link
* Always return first page when fetching events by permalink to reset any existing pagination state
2022-04-19 10:02:32 -04:00
FoxxMD
8085e8ced6 Merge branch 'edge' into databaseSupport 2022-04-19 09:30:29 -04:00
FoxxMD
749980abbe feat(repeat): Make identifying repeat sets from reference submission more fuzzy
When reference submission is a self post identifier may vary slightly since it is considering both title and body. Use string matching on identifies to find "close" matches for reference submission
2022-04-18 11:40:10 -04:00
FoxxMD
db83f1d0e7 Update schema after merge 2022-04-14 14:24:10 -04:00
FoxxMD
77960ba186 Merge branch 'edge' into databaseSupport
# Conflicts:
#	src/Schema/Action.json
#	src/Schema/App.json
#	src/Schema/OperatorConfig.json
#	src/Schema/Rule.json
#	src/Schema/RuleSet.json
#	src/Subreddit/SubredditResources.ts
2022-04-14 14:23:28 -04:00
FoxxMD
03fc562643 fix(filter): Remove/implement debug placeholder for bot self account substitution
oops
2022-04-14 13:44:07 -04:00
FoxxMD
9ef58dfd27 feat(filter): Implement testing for which moderator approved or removed an activity
Extends removed/approved criteria to accept a string, strings, or object with testing behavior and names
2022-04-14 13:31:56 -04:00
FoxxMD
1ce415bcd5 Merge branch 'edge' into databaseSupport
# Conflicts:
#	src/util.ts
2022-04-14 12:11:28 -04:00
FoxxMD
e4a9e47d6a fix(filter): Handle scenario where submissionState is incorrectly used on submission itemIs
* Check for found criteria results when building summary
* change log level from debug to warn when submissionState is detected on submission itemIs
2022-04-14 12:10:41 -04:00
FoxxMD
dc1a999376 Merge branch 'edge' into databaseSupport
# Conflicts:
#	src/Rule/RepeatActivityRule.ts
2022-04-14 10:07:35 -04:00