Compare commits

..

654 Commits

Author SHA1 Message Date
FoxxMD
9e364c3e28 Refactor cache instantiation to use async method 2022-10-17 11:42:58 -04:00
FoxxMD
acd4b54d8e Implement typings for cache-manager-redis-store@3^ 2022-10-17 11:42:39 -04:00
FoxxMD
19f4ff0b53 feat: Update to cache-manager-redis-store ^3.0.0
* Also bump node minor version to be in line with cache-manager-redis-store
2022-10-17 11:42:04 -04:00
FoxxMD
98a8568eb6 fix: Add missing else condition 2022-10-13 16:17:43 -04:00
FoxxMD
457f947603 fix: Further improvements for influxdb logging
* Suppress write failure warnings to reduce noise in log
  * Can be toggled using debug flag in config
2022-10-13 12:34:11 -04:00
FoxxMD
7fb69ae67a fix: Attempt to decrease frequency of influxdb timeout errors
* Use keep-alive http agent to reuse open connections
* Decrease max batch size and flush interval to payload sent is smaller
* Add debug logging for fail/success/retry events on flush
2022-10-13 11:39:45 -04:00
FoxxMD
2241d40e49 fix: Fix custom footer never loading
Custom footer needs to be loaded AFTER resources are set
2022-10-13 08:58:12 -04:00
FoxxMD
a3ca3f17ec refactor(recent): Log image parsing error with cause 2022-10-12 14:39:21 -04:00
FoxxMD
f527a17fa2 feat(modnote): Implement existing note check before adding note to replace allowDuplicates
* Use "modActions" authorIs filtering to check note prior to adding note using "existingNoteCheck" property
  * Refactors concept of "allowDuplicates" to allow any arbitrary modActions test to be used
* Provide convenience ModLogCriteria generation for "existingNoteCheck" based on boolean (emulates allowDuplicates functionality)
2022-10-12 13:00:47 -04:00
FoxxMD
e98364eae9 feat(filter): Improve modAction filtering functionality
* Implement filtering activityType by "false" in order to return actions/notes not added to a specific activity
* Implement "referencesCurrentActivity" property to allow filtering by actions/notes that are associated with the activity being processed
* Implement using "count" for "current" search to enable criteria condition based on presence or non-presence of current action/note passing
2022-10-12 12:58:49 -04:00
FoxxMD
8b125d7433 feat(ui): Improve visibility and resilience for live log stream
* Tie loading indicator to live stream status and display error if one occurs
  * Add manual restart action to end of error
* Restart stream automatically if reader ends, up to 3 retries
2022-10-11 12:19:55 -04:00
FoxxMD
6ee060c5ce fix(logs): Remove listeners from log stream event emitter before end of response to prevent write-after-end errors 2022-10-11 12:18:00 -04:00
FoxxMD
9b12d0b2b3 feat(bot): Improve log wording for manager loading phase 2022-10-11 10:51:00 -04:00
FoxxMD
b174c7928a feat(ui): Add favicon files 2022-10-11 10:49:57 -04:00
FoxxMD
74dfe9258a fix(filter): Fix mod action note filtering assignment 2022-10-10 15:00:39 -04:00
FoxxMD
1cf8855a24 feat(testing): Implement initial author filter tests 2022-10-10 12:06:08 -04:00
FoxxMD
adc69894fc fix(filter): Fix detecting empty filter when using 'replace' filter default behavior 2022-10-10 11:03:57 -04:00
FoxxMD
3435c683c8 feat(filter): Add author flair item criteria 2022-10-05 11:24:44 -04:00
FoxxMD
f0032cd433 Bump version 2022-10-05 08:56:53 -04:00
FoxxMD
ade0b7948e Add patreon funding link 2022-10-04 09:40:03 -04:00
FoxxMD
542aa26c62 feat(history): Implement ratio comparison #112 2022-10-03 13:25:05 -04:00
FoxxMD
3faf4ca3dc fix: Fix wiki location usage when getting from subreddit resources 2022-09-28 10:34:48 -04:00
FoxxMD
2f35b82d5e feat(history): Log failure results for easier rule tuning 2022-09-28 10:25:26 -04:00
FoxxMD
c9d8bf637b chore: Add tests for item criteria 2022-09-26 16:59:46 -04:00
FoxxMD
027f4087e3 refactor: Improve activity source parse and comparison
* Implement a DTO class for activity source to make parts usage (type, identifier) and matching easier
* Implement regex to parse type and identifier from activity source string
* Refactor activity source interface/types to better distinguish as string, data, and class
2022-09-26 12:07:24 -04:00
FoxxMD
1b20122ffc docs: Fix some typos 2022-09-23 10:35:13 -04:00
FoxxMD
5d53571ec0 fix: Fix typo in default footer
Thanks u/SampleOfNone for catching that
2022-09-22 10:59:57 -04:00
FoxxMD
b3df1b4d41 Update default confidence for MHS based on feedback from Welton 2022-09-20 10:56:39 -04:00
FoxxMD
4abe8e07f3 docs: Add docs for MHS toxic content prediction rule 2022-09-20 10:44:08 -04:00
FoxxMD
9bb95106ba feat(mhs): Add default confidence threshold 2022-09-20 09:35:38 -04:00
FoxxMD
02414478bf feat: Implement rule that tests activity content against moderatehatespeech.com ML model using api #110
* Add mhs rule type and MHS credentials interface
* Implement MHS rule with similar criteria options to sentiment
* Allow testing against author history content
2022-09-19 16:20:35 -04:00
FoxxMD
8b0a582464 fix(recent): Empty viable activity list if processing activity is required to be reference but is not a valid type 2022-09-19 11:46:20 -04:00
FoxxMD
d1db5f4688 fix: Fix subreddit filtering result when no subreddit criteria are present
Should return all items instead of none. Thanks u/SampleOfNone for helping track this down with examples.
2022-09-19 10:01:39 -04:00
FoxxMD
44f9389b69 docs: Add general subreddit/moderator docs readme
* Also add Guest Access docs
2022-09-16 16:40:28 -04:00
FoxxMD
71b2d0597d docs: Update screenshots 2022-09-16 15:38:52 -04:00
FoxxMD
57cfcebe9f fix: Set default invites to empty array on heartbeat 2022-09-16 14:34:28 -04:00
FoxxMD
07ecc505ff fix: Check bot entity exists before getting invites 2022-09-16 14:29:34 -04:00
FoxxMD
81213686ce Merge branch 'subredditInviteUI' into edge
# Conflicts:
#	src/Subreddit/SubredditResources.ts
2022-09-16 13:31:37 -04:00
FoxxMD
08735d505a refactor(manager): Improve wiki CRUD and onboarding initial config
* Refactor write/read into separate functions
* Improve error hinting for wiki read/write/permissions WRT mod/oauth permissions
* Remove superfluous error wrapping to reduce logging length for wiki errors
* More debug logging for onboarding process
* Don't return error if manager fails to parse after all onboarding complete (not critical)
2022-09-16 13:27:41 -04:00
FoxxMD
1a62c752c1 feat(ui): Improve finished onboarding response
* don't redirect on error
* increase delay to redirect on success
2022-09-16 13:24:09 -04:00
FoxxMD
6aa7367297 fix(client): Improve responses dependent on server information
* run initHeartbeat on any GET route that render a page so that user doesn't get access denied on initial app load
* Force client refresh if no invite found on initial check for onboarding landing
2022-09-16 13:22:44 -04:00
FoxxMD
cf9583227c feat: Add subredditExists function to snoowrap client
Reddit returns 403 if the subreddit exists but is private. Using this function wraps the error so we can just get boolean back along with subreddit object, if successful
2022-09-16 13:20:07 -04:00
FoxxMD
aa505ba3f2 feat: Improve external resource fetching in subreddit resources
* Add "default" hint to force val to a url or wiki key if neither is detected but know it should be one of them
* Refactor wiki/url fetching into own functions for better reuseability
* Implement mod permission getter function to check for valid permissions on wiki page error
* Improve error hints on wiki page read failure
2022-09-16 13:19:23 -04:00
FoxxMD
a0182d89ca feat: util function for generating full wiki urls 2022-09-16 13:16:26 -04:00
FoxxMD
d46f0a5be8 fix: Improve error detection for log transform
Check for error name OR stack existence
2022-09-16 13:16:06 -04:00
FoxxMD
4a55d35e14 feat(filter): Rename createdAt to createdOn
Better naming
2022-09-15 13:27:41 -04:00
FoxxMD
1284051fe8 fix(filter): Fix copy-past typo for createdAt 2022-09-15 11:24:53 -04:00
FoxxMD
00680494a3 feat(filter): Add createdAt activity criteria matching
* May be a string or array of strings. Passes if any expression matches
  * Value may be a convience day-of-week value (mon, tues, wed...)
  * or a cron expression
2022-09-15 11:22:04 -04:00
FoxxMD
77856a6d97 chore: Bump version 2022-09-14 15:30:12 -04:00
FoxxMD
b216cd08e1 Merge branch 'edge' into subredditInviteUI 2022-09-14 15:27:14 -04:00
FoxxMD
052c1218c6 feat(ui): Add home logo 2022-09-14 15:26:59 -04:00
FoxxMD
c2343683bb Finish implementation accept flow 2022-09-14 15:12:33 -04:00
FoxxMD
fcf718f1d0 fix(ui): Fix typo in reddit status indicator switch 2022-09-14 13:21:48 -04:00
FoxxMD
77f848007a Implement subreddit onboarding accept process
* Refactor some manager/bot methods to be more accessible for invites/wiki
* Add routes and authentication for getting invite information and checking user is moderator of subreddit
* Add route for accept invite and completing onboarding guest/config
2022-09-14 13:20:53 -04:00
FoxxMD
95216b3950 docs: Add GHCR image location to install docs
Closes #109
2022-09-13 15:06:13 -04:00
FoxxMD
58a21e8d05 Scope required permissions for github token 2022-09-13 14:35:11 -04:00
FoxxMD
49ac8cda19 feat: Update github actions to push to multiple registries
And add documentation for GH actions local dev/testing
2022-09-13 14:23:22 -04:00
FoxxMD
bc8be3608b Use methods for bot entity subreddit invite crud 2022-09-13 10:33:55 -04:00
FoxxMD
1b69cd78bb Refactor subreddit invites to use db and add interface
* Refactor to use db instead of cache for persisting invites
* Implement subreddit invite helper page
* Add initial config and guests as optional data for invite
* Refactor bot to use db subreddit invite and auto-accept when no config/guests

TODO: subreddit accept page, mod authorization, initial config usage, and documentation
2022-09-12 16:29:41 -04:00
FoxxMD
e736379f85 feat(ui): Improve guest expiration interface
* Set default time to 24 hours
* Add label and tooltip for expiration datetime picker
2022-09-07 11:57:29 -04:00
FoxxMD
c0e1a93fb4 fix(config): Use correct filter defaults data structure for runs
Should use "json" data structure type so named filters can be used in defaults on runs
2022-09-06 11:18:24 -04:00
FoxxMD
bd35b06ebf fix(filter): Fix missed toLowerCase for named filter getter 2022-09-02 17:41:28 -04:00
FoxxMD
f852e85234 feat(author): authorIs 'name' criteria may be regular expression 2022-09-02 16:21:53 -04:00
FoxxMD
661ae11e18 refactor(ui): Replace websockets opStats with client delta polling #106
* Refactor/remove websockets functionality for relaying opstats from server with direct polling by client
* Implement delta responses initially introduced in #91 to reduce bandwidth
2022-09-01 16:01:37 -04:00
FoxxMD
18c1ac0fd7 Chore: Bump version 2022-09-01 09:03:12 -04:00
FoxxMD
2fb503f09f feat(message): Implement templating for to field so subreddit can be used generically
since 'to' be now be templated a user can configure a message to send to the subreddit the action is being processed from using `to: 'r/{{item.subreddit}}'`
2022-08-31 09:58:24 -04:00
FoxxMD
7e5eeb71da fix(submission): Fix targets data type to accept array 2022-08-31 09:31:25 -04:00
FoxxMD
84f2da8b6d feat(action): Implement Action templating data #104
* Add top-level 'actionSummary' template variable that renders a markdown list of action results
* Add individual action result/data, in the same structure as rules, under the top-level 'actions' template variable
2022-08-30 17:12:51 -04:00
FoxxMD
be51f8ae43 fix(image): Fix hash result test assertion 2022-08-30 16:13:35 -04:00
FoxxMD
f82d985eab feat(template): Refactor templating to be more extensible and add some requested data
* Implement interfaces for template parts
* Refactor Action constructor to take an object for runtime options (cleaner, more extensible)
* Refactor subreddit resource and snoowraputils content rendering and organization to use objects of optional data rather than required arguments
  * Make almost all data optional and only parse/render if included
  * Move rule results parsing/formatting into own function
  * Refactor footer render to use renderContent (DRY)
* Add some requested template data #104
  * {{item.subreddit}} #87
  * {{check}} #87
* Updated templating documentation
2022-08-26 13:42:02 -04:00
FoxxMD
aab014650a fix(template): Fix rule data spread to template 2022-08-26 11:22:59 -04:00
FoxxMD
113ac3e10e fix(template): Fix subreddit breakdown reference override 2022-08-25 17:14:28 -04:00
FoxxMD
afb6aad26d fix(template): Fix subreddit breakdown when empty activities array 2022-08-25 16:30:15 -04:00
FoxxMD
33b60825d9 fix(template): Return empty string if no subreddits in breakdown 2022-08-25 16:16:15 -04:00
FoxxMD
b5202a33ac feat(template): Add subreddit breakdown to template data for Recent and History rule 2022-08-25 10:39:03 -04:00
FoxxMD
c02a2ad622 chore: Bump version constant due to BC 2022-08-23 09:49:12 -04:00
FoxxMD
e9f08915a4 chore: Bump version constant 2022-08-23 09:47:35 -04:00
FoxxMD
4b95ccd0ba feat(docs): Simplify getting started operator instructions
Take advantage of First Time Setup page when no configuration is present to help generate a minimum configuration.
2022-08-23 09:46:54 -04:00
FoxxMD
567a2f0720 feat(action): Implement creating submissions #93
* Create self/link submissions, no reddit media (upload) yet
* Submission can use user modifiers (nsfw, spoiler) and mod modifiers (sticky, distinguish, lock)
* Can create submission in current subreddit (default) or arbitrary subreddit defined in configuration (targets)
  * Can create multiple submissions by defining multiple targets
2022-08-22 14:00:27 -04:00
FoxxMD
61ff402256 feat(config): Allow wikiConfig override at subreddit-level 2022-08-22 10:36:18 -04:00
FoxxMD
9158bda992 feat(comment): Implement arbitrary comment targets #93 2022-08-19 11:01:54 -04:00
FoxxMD
873f9d3c91 refactor(image): Normalize image hash usage a bit more to reduce complexity 2022-08-18 13:35:37 -04:00
FoxxMD
783ef5db53 fix(image): url variable name usage 2022-08-16 13:36:36 -04:00
FoxxMD
f0cb5c1315 fix(ui): Fix constant reassignment 2022-08-16 13:36:20 -04:00
FoxxMD
8e1b916ea4 fix(database): Remove table data check due to incompatibility with postgres
Not sure what the issue is but a generic message should be enough for now
2022-08-16 10:28:10 -04:00
FoxxMD
4acf87eacd feat(database): Implement automated backup for better-sqlite3 connection 2022-08-16 10:05:20 -04:00
FoxxMD
5dd5a32c51 fix(database): Fix how new columns are added to existing table on invite migration 2022-08-16 09:47:53 -04:00
FoxxMD
207907881f feat: Add initial setup wizard
* If web client does not have credentials (or operator) then redirect login to form to fill them in
  * Write to config location on form completion
2022-08-15 17:17:23 -04:00
FoxxMD
44da276d41 fix: Fix missing column type specifying for db session entity 2022-08-15 16:05:00 -04:00
FoxxMD
6c98b6f995 fix: cleanup guests and no-config scenario
* Check for no/null guest data during invite creation/usage for both client and server side
* Add instances tab on invite page
* Replace instance select on invite page with query string usage
* Redirect from status page to auth helper when instance has no bots
2022-08-15 15:59:51 -04:00
FoxxMD
cc0c3dfe61 docs: Fix incorrect url for minimum config 2022-08-15 15:57:45 -04:00
FoxxMD
b48d75fda3 fix(config): Fix setting api friendly name
addIn creates missing collections automatically
2022-08-15 15:57:26 -04:00
FoxxMD
2adf2d258d fix(database): Handle invite migration when db is sqljs
* Create table if it does not exist (sqljs has two database)
* Drop Invite table from sqljs db if it has no rows
2022-08-15 15:56:43 -04:00
FoxxMD
c55a1c6502 feat(ui): Add initial guests to bot invite and rename Guest Mod to Guest
* Rename to Guest since this is more accurate of a description -- "mod" is confusing since user doesn't have any actual mod power
* Add guests as an option to invite creation and display on invite page
2022-08-11 15:23:09 -04:00
FoxxMD
0011ff8853 chore: Re-add clean username for guest due to merge 2022-08-11 14:53:48 -04:00
FoxxMD
4bbf871051 Merge branch 'edge' into tempAccess
# Conflicts:
#	src/Web/Server/routes/authenticated/user/index.ts
#	src/Web/Server/server.ts
2022-08-11 14:50:45 -04:00
FoxxMD
54755dc480 refactor(guest)!: Migrate invites to be owned by the server
It makes more sense for the CM instance that will actually have a bot added to it to own the invite for that bot.

* (BC) Move Invite into server entity mappings and rename to BotInvite
  * Add guests and initialConfig (future use)
  * BC -- Existing invites need to have instance defined to be used
  * BC -- Cache-based invite storage has been REMOVED
  * BC -- Removed inviteMaxAge config property (for now)
* Add SubredditInvite entity for future use
* Force/implement server (api) having a defined friendly name. This is used to determine which invites in a DB belong to which server instance
  * If not defined in config it is generated at random and then written to config
* Refactor how invites are retrieved and parsed client-side -- CRUD using server api
  * BC -- Invite URL structure has changed from ?invite=id to /invite/id...
* Added better UI for migration redirect
  * Show all reachable instances in redirect page header
  * Error page also shows reachable instances in page header
2022-08-11 14:47:45 -04:00
FoxxMD
01f95a37e7 feat(ui): Add guided tour for main dashboard 2022-08-03 14:37:56 -04:00
FoxxMD
5ddad418b0 fix(manager): Fix missing valid config check before starting queue 2022-08-02 16:05:48 -04:00
FoxxMD
b5b2e88c1f fix(guest mod): Parse/clean name from user input before adding
So that we don't care if user sends FoxxMD or u/FoxxMD
2022-08-02 15:46:56 -04:00
FoxxMD
194ded7be6 fix(auth): Fix subreddit normalization on client instance 2022-08-02 15:31:00 -04:00
FoxxMD
7ba375d702 fix(auth): Fix subreddit normalization 2022-08-02 14:15:01 -04:00
FoxxMD
9a4c38151f fix(auth): Fix default instance middleware accessibility check 2022-08-02 14:06:49 -04:00
FoxxMD
f8df6fc93f docs: Remove removal reason footnote 2022-08-02 13:02:28 -04:00
FoxxMD
86a3b229cb feat(remove): Implement removal reason id and note
* Cache subreddit removal reasons and display in popup helper in authenticated config editor
* Add fields for removal reason note/reason in remove action
* Update remove action documentation to include new fields
2022-08-02 12:34:40 -04:00
FoxxMD
ca3e8d7d80 feat: Add removal reason related endpoints to snoowrap client
Using undocumented endpoints pulled from praw documentation/code. Snoowrap can now:

* add removal reason/mod note on a removed activity
* get subreddit removal reasons (used for getting ids for use with removal reason endpoint)
2022-08-02 11:09:05 -04:00
FoxxMD
5af4384871 Merge branch 'tempAccess' into edge 2022-08-01 12:08:10 -04:00
FoxxMD
47957e6ab9 fix: Remove debug statements for image processing 2022-07-29 15:05:08 -04:00
FoxxMD
7f1a404b4e feat(image): Implement image flip detection in recent rule 2022-07-29 14:42:49 -04:00
FoxxMD
1f64a56260 feat(image): Improve image normalization to make duplicate detection more powerful
* Trim image to remove arbitrary borders
* Convert image to greyscale in order to reduce effect of saturation differences
* Implement "mirrored" (y-axis flip) hash calculations
* Implement local file fetch and image processing test suite
2022-07-29 14:19:32 -04:00
FoxxMD
366cb2b629 More guest mod cleanup and fixes
* Default value on guest mod CRUD
* Validate expire time client-side
2022-07-28 14:28:43 -04:00
FoxxMD
b9b442ad1e feat(ui): Implement saving config as guest mod 2022-07-28 14:14:55 -04:00
FoxxMD
81a1bdb446 refactor: Simplify ACL for server/client users and introduce guest context
* Use BotInstance class on client side for easier state tracking and simplifying ACL functions (moved from user to instance)
* Implement and use same interface for client/server Bot representations
* Implement mod/guest context for status and live stats data
* Restrict guest mod CRUD to mods only
2022-07-28 13:39:46 -04:00
FoxxMD
1e4b369b1e Fix CRUD for guest mods 2022-07-28 09:45:30 -04:00
FoxxMD
7a34a7b531 Almost working correctly guest mod add
Need to fix something wrong with time parsing server-side
2022-07-27 17:06:55 -04:00
FoxxMD
b83bb6f998 Refactor guest relationship
* Use eager relation
* Fix orphan row usage
2022-07-27 16:18:34 -04:00
FoxxMD
3348af2780 Implement guest mod removal via UI and refactor guest rendering into live stats 2022-07-27 15:32:00 -04:00
FoxxMD
04896a7363 Implement guest mod entities and read-only ui 2022-07-27 13:07:56 -04:00
FoxxMD
d8003e049c chore(schema): Update schema to reflect ban property max length changes 2022-07-25 16:36:24 -04:00
FoxxMD
b67a933084 feat(ui): Real-time data usage improvements based on visibility
Stop or restart real-time data (logs, stats) based on page visibility API so that client does continue to consume data when page is in the background/in a non-visible tab
2022-07-25 16:35:32 -04:00
FoxxMD
d684ecc0ff docs: Add memory management notes
* Document memory management approaches
* Change node args for docker to a better name (NODE_ARGS)
* Implement default node arg for docker `--max_old_space_size=512`
2022-07-25 12:20:02 -04:00
FoxxMD
9efd4751d8 feat(templating): Render content for more actions and properties
* Template render Message action 'title'
* Template render Ban action 'reason' and 'note'
2022-07-25 11:56:22 -04:00
FoxxMD
9331c2a3c8 feat(templating): Include activity id and title for all activities
* Include reddit thing id as 'id'
* Include 'title' -- for submission this is submission title. For comment this is the first 50 characters of the comment truncated with '...'
* Include 'shortTitle' -- same as above but truncated to 15 characters
2022-07-25 11:41:43 -04:00
FoxxMD
d6f7ce2441 feat(server): Better handling for subreddit invite CRUD
* Return 400 with descriptive error when invalid value or duplicate invite requested
* Better value comparison for subreddit invite deletion
2022-07-25 11:24:19 -04:00
FoxxMD
ffd7033faf feat(server): Add server child logger to requests for easier logging 2022-07-25 11:22:44 -04:00
FoxxMD
df5825d8df fix(ui): Fix setting style for non-existing element on subreddit invite 2022-07-25 11:13:32 -04:00
FoxxMD
42c6ca7af5 feat(ui): More live stat delta improvements and better stream handling on the browser
* More granular delta for nested objects
* Custom delta structure for delayedItems
* Fix abort controller overwrite
* Enforce maximum of two unfocused log streams before cancelling immediately on visible change to reduce concurrent number of requests from browser
2022-07-22 13:50:53 -04:00
FoxxMD
1e94835f97 feat(ui): Reduce data usage for live stats using response deltas 2022-07-22 09:46:01 -04:00
FoxxMD
6230ef707d feat(ui): Improve delayed activity cancel handling
* Fix missing bot param in DELETE call
* Fix missing database removal of canceled activity
* Implement ability to cancel all accessible
2022-07-21 10:11:19 -04:00
FoxxMD
b290a4696d fix(ui): Use correct auth middleware for api proxy endpoint 2022-07-20 13:04:46 -04:00
FoxxMD
4c965f7215 feat(docker): Expand NODE_FLAGS (ENV) variable in node run command to allow arbitrary flags be passed through docker ENVs 2022-07-20 11:18:13 -04:00
FoxxMD
ce990094a1 feat(config): Gate memory monitoring behind operator config option monitorMemory 2022-07-20 10:20:52 -04:00
FoxxMD
4196d2acb0 feat(influx): Add server memory metrics 2022-07-19 16:28:58 -04:00
FoxxMD
3150da8b4a feat(influx): Add manager health metrics 2022-07-19 14:26:14 -04:00
FoxxMD
655c82d5e1 fix(filter): Make source filter stricter by requiring exact value rather than "contains"
Fixes issue where source that is a shorter version of a longer identifier does not accidentally get run first
2022-07-18 15:55:41 -04:00
FoxxMD
73302b718e chore: Bump version for 0.11.4 release 2022-07-15 09:27:04 -04:00
FoxxMD
bbf91ceac0 feat(filter): Implement filtering by upvoteRatio for authorIs 2022-07-14 13:56:12 -04:00
FoxxMD
67b793c2aa feat: Support parsing filename from non-raw GIST url in editor
* Support filename for non-raw gist
* Fix raw gist URL usage by fetching from CDN
2022-07-14 12:48:04 -04:00
FoxxMD
d635e5a65d feat: Support parsing filename from non-raw GIST url 2022-07-14 11:12:04 -04:00
FoxxMD
8dc140a953 feat(report): Implement filtering reports in itemIs by time
* Refactor time parsing and comparison utils to consolidate logic and be more flexible
* Tests for the above
* Fix indexes for ActivityReport (should be unique)
* Fix missing eager load for Activity-Report relationship
* Implement time filtering for reports in itemIs
2022-07-13 13:00:49 -04:00
FoxxMD
eaa9f627e2 refactor(report): Cleanup monitoring in manager 2022-07-13 09:36:33 -04:00
FoxxMD
16cb28cb72 feat(report): Implement report tracking #78
* Store reports with likely creation time to DB
* Use some trickery and caching in Manager to get likely report creation time
2022-07-12 17:29:28 -04:00
FoxxMD
fa450f9f8f docs(database): Add influx/grafana documentation 2022-07-11 13:46:27 -04:00
FoxxMD
1e76ca6c0e docs(config): Add partial configuration documentation 2022-07-11 13:01:40 -04:00
FoxxMD
25b7ea497f feat(filter): Implement specifying report reason in report comparison 2022-07-11 12:19:28 -04:00
FoxxMD
2f93774346 fix: Fix unnecessary subreddit fetch call on init
Snoowrap doesn't add fetch property to objects from listings so even though subreddit may already be fetched it always re-fetches, using an api call. Do a dirty fetch check here to prevent wasting calls when we are sure the subreddit is fetched.
2022-07-11 12:18:30 -04:00
FoxxMD
e2f6a92a90 fix(filter): Fix logic for assigning number of reports to compare 2022-07-11 10:01:58 -04:00
FoxxMD
0ffddaac9e Revert "chore: Bump got and sharp versions for dependabot fixes"
This reverts commit fbf328d90f.
2022-07-11 08:56:44 -04:00
FoxxMD
4ddf6ddf26 Revert "chore: Bump passport version for dependabot fixes"
This reverts commit 7f61c190ea.
2022-07-11 08:56:41 -04:00
FoxxMD
7f61c190ea chore: Bump passport version for dependabot fixes 2022-07-08 12:34:54 -04:00
FoxxMD
fbf328d90f chore: Bump got and sharp versions for dependabot fixes 2022-07-08 12:33:03 -04:00
FoxxMD
36c6f7f1b8 feat(ban): Implement templating for ban note/reason 2022-07-08 10:51:47 -04:00
FoxxMD
0379ad17b9 feat: Add more Activity data to templating data #94
* Add votes, reports, upvote ratio, nsfw, op, and spoiler
* Update templating documentation
2022-07-08 10:24:10 -04:00
FoxxMD
02ed9f91f9 feat: Improve api health monitoring
* Potentially fix #30
* Break out api sampling and output into own function
* Add influx metrics for used calls and sample faster
* Fix default tags inheritance
2022-07-07 13:05:14 -04:00
FoxxMD
9fcc3db7b2 fix(modnote): Use correct activity identifier for reddit_id 2022-07-07 09:43:25 -04:00
FoxxMD
65ea84a69d refactor(database): Move Event creation to earlier in activity handling function for a more accurate processing time 2022-07-06 17:41:14 -04:00
FoxxMD
a4b8d3a8ef feat(database): Add api health measurement to influxdb 2022-07-06 17:11:38 -04:00
FoxxMD
2692a5fecb fix(database): Fix processing/queued time value 2022-07-06 16:48:39 -04:00
FoxxMD
192c1659a0 feat(database): Initial influx db implementation #84 2022-07-06 15:14:58 -04:00
FoxxMD
cc241e41f4 refactor(logging): Improve log retention performance 2022-07-05 12:30:51 -04:00
FoxxMD
4bfb57a6cf chore: Update connect-typeorm dependency
Moved to 2.0.0 b/c it now supports typeorm >= 0.3.0
2022-07-05 09:43:25 -04:00
FoxxMD
b6dedae7a1 fix(usernote): Fix regression with note types
Can use a custom note type so don't restrict to only the documented ones
2022-07-01 12:06:48 -04:00
FoxxMD
f2783bd7a4 feat(config): More error handling and internal improvements to config fragment parsing
* Refactor config fragment parsing function to always return an array for simpler usage in hydrate function
* Add error handling to all fragment calls so user can get a clear description of which fragment failed, contextually
2022-07-01 11:46:54 -04:00
FoxxMD
2efe41eadd fix(cache): Use correct cache key and improve external url cache re-use
* Fix incorrect variable used for cache key that prevented cache from working at all (oops)
* Don't add subreddit context to ext url cache key so it can be re-used from any subreddit
2022-07-01 11:45:11 -04:00
FoxxMD
ebd60b9abe feat(config): Support config fragments as array of fragments #76 2022-07-01 10:20:53 -04:00
FoxxMD
2a16df49a4 feat(config): Implement cache control for config fragments #76 2022-06-30 14:44:04 -04:00
FoxxMD
0d6841259b feat(config): Initial implementation for partial configs #76 2022-06-30 13:04:02 -04:00
FoxxMD
8c0755c8c2 working when no includes in config 2022-06-30 12:39:57 -04:00
FoxxMD
5e1da5bc5d feat(bot): Implement better in-situ subreddit add/removal
* Refactor manager build/init from bot to be independent of bot init process
  * Remove/destroy managers for subreddits no longer moderated by bot account or not in operator list
  * Add/create managers for subreddits that should be moderated
* Run manager build function during heartbeat to sync managers
2022-06-28 13:24:56 -04:00
FoxxMD
242c6a49b5 docs(usernote): Document usernote types in schema and in documentation 2022-06-27 09:33:57 -04:00
FoxxMD
aa399c160e docs(sentiment): Add sentiment documentation 2022-06-21 16:13:38 -04:00
FoxxMD
d7cab4092d fix(ui): Hide save reason input when user cannot save 2022-06-16 14:25:06 -04:00
FoxxMD
0370e592f9 feat(ui): Add text input to specify reason for wiki edit on config page 2022-06-16 09:43:27 -04:00
FoxxMD
116d06733a fix(ui): Stop trying live stats if an error is encountered 2022-06-14 22:33:12 -04:00
FoxxMD
22a8a694a7 fix(modnote): Fix initial implementation bugs
* Improve action criteria to FullCriteria function
  * Don't include undefined properties
  * Iterate entries with switch to simplify property transformations
* Fix mod action test switch case matching to be case-insensitive (same as key)
* Fix missing/bad assignments for filtering mod actions
* Fix typo usage of foundNoteResult in modActions case block
* Throw error if mod action criteria isn't recognized as note/log instead of silently falling back to log
2022-06-14 12:55:35 -04:00
FoxxMD
2ed24eee11 fix(logging): Fix missing modAction property transformation for filter summary 2022-06-14 12:51:47 -04:00
FoxxMD
8822d8520a fix(check): Fix check data used to determine trigger text hint 2022-06-14 12:51:02 -04:00
FoxxMD
9832292a5b docs(mod actions): Add docs for mod log/note actions and filtering 2022-06-13 15:19:34 -04:00
FoxxMD
7a86c722fa feat(author criteria): Add temporal filtering to usernotes and mod notes 2022-06-13 13:42:57 -04:00
FoxxMD
2ca4043c02 chore(organization): Move comparison and comparison parsing functions into Comparisons file 2022-06-13 12:13:20 -04:00
FoxxMD
4da8a0b353 feat(author criteria): Refactor criteria test order to optimize api usage #92
* Test props in order of least likely to use an API call
  * Enables simplifying shadowbanned test and allows testing for more properties on shadowbanned user
2022-06-13 11:50:12 -04:00
FoxxMD
492ff78b13 chore(modnotes): Cleanup interfaces and add missing activityType to modnote criteria 2022-06-10 13:34:04 -04:00
FoxxMD
64a0b0890d feat: Improve/fix bot addition
* Fix existing bot removal
* Return response to client after testing client rather than after managers build to avoid long response time if bot has many subreddits
2022-06-10 12:31:56 -04:00
FoxxMD
546daddd49 feat(config): granularly merge bot data into yaml operator config 2022-06-10 12:15:46 -04:00
FoxxMD
f91d81029f feat(modnote): Implement modnotes as part of author criteria 2022-06-09 15:28:29 -04:00
FoxxMD
68ee1718e0 feat(modnote): Prevent getting modnotes for subreddit not currently moderated 2022-06-09 12:58:29 -04:00
FoxxMD
c0d19ede39 feat(modnote): Improve handling of modnote data and implement caching
* Normalize (depopulate from snoowrap) mod note raw data so it can be constructed agnostic of source (cache or api)
* Implement cache GET for modnotes with default TTL of 60 seconds
* Refactor mod note action and implement cache PUT when new notes are added
2022-06-09 12:54:23 -04:00
FoxxMD
bb05d64428 feat: Improve reddit entity type guards
* Use 'in' operator to avoid triggering snoowrap proxy
* Add subreddit type guard
* Fix missing undefined check for names
2022-06-09 12:17:51 -04:00
FoxxMD
1977c7317f fix(modnote): Fix modnote mapping from created response 2022-06-07 14:36:36 -04:00
FoxxMD
6f784d5aa2 feat(modnote): Implement fetching modnotes and refactor to get in sync with edge 2022-06-07 14:29:33 -04:00
FoxxMD
4b5c9b82e4 Merge branch 'edge' into modnote
# Conflicts:
#	src/Action/ActionFactory.ts
#	src/Action/index.ts
#	src/Common/types.ts
#	src/Schema/Action.json
#	src/Schema/App.json
#	src/util.ts
2022-06-07 12:52:58 -04:00
FoxxMD
0315ad23ae feat(ui): Prevent indexing by robots 2022-06-02 13:45:39 -04:00
FoxxMD
da70753f42 feat(database): Add missing indexes to improve event retrieval performance
I was under the impression primary keys were always indexed but that is not the case, at least for postgres. This migration explicitly creates uniques indexes for all tables that use random ids and adds other indexes to filter/premise/result tables on other FK strings. Improves event retrieval timing dramatically.
2022-05-26 14:36:15 -04:00
FoxxMD
d59f1b63d1 chore: Bump version 2022-05-26 09:59:17 -04:00
FoxxMD
7542947029 feat(sentiment): More improvements to sentiment and language processing
* Implement separate language detection functionality
* Clearer/simpler sentiment processing
* Add languageHints to help coerce low confidence language detection
* Add test cases for lang detection, sentiment detection, and sentiment tests
* Fix neutral range -- was not using normalized score range
2022-05-25 12:59:37 -04:00
FoxxMD
2d02434e7e feat(sentiment): Better use of NLPjs and more robust language detection
* Build NLP container ad-hoc so only supported languages are included from npm
* Use vader/wink as heuristics for detecting language when content is very short
* Add languageHint option for sentiment config to make coercing a confident sentiment easier
* Refactor lang processing to fail the sentiment test rather than throwing an error when language is not support/not confident -- provides more insight into outcome
2022-05-24 16:34:37 -04:00
FoxxMD
e2824ea94c chore(docker): Remove superfluous dev packages from final docker layer #88
typeorm depends on ts-node as an *optional peer* dependency -- ts-node can be used in the typeorm cli to parse entities and run migrations.

However CM doesn't use typeorm CLI for running production so its not needed. And ts-node depends on typescript so npm install --production always installs both at about ~30MB.

I couldn't find a good way to remove peer deps ONLY for typeorm so instead just manually remove the folders from the prod install in the final layer of the docker.
2022-05-24 10:31:15 -04:00
FoxxMD
1c94548947 chore: Update TS version and move into dev dependencies
Not needed to actually run the app after it has been built so moving to dev-dependencies
2022-05-24 10:26:46 -04:00
FoxxMD
2073e3f650 chore: Update dependencies 2022-05-24 09:37:59 -04:00
FoxxMD
90b8f481ec chore: Remove unused dependencies #88 2022-05-24 09:33:29 -04:00
FoxxMD
9ad9092e9e feat(dispatch): Fix regression introduced when removing processing state from dispatch action lifecycle
Should have removed entire check for processing but only removed part of if condition, resulting in matched activities never being returned
2022-05-23 14:23:16 -04:00
FoxxMD
12adfe9975 fix(approve): Improve activity type detection and add more result output 2022-05-23 14:22:01 -04:00
FoxxMD
83dceddae8 feat(ui): Simplify live stats by moving into browser logic
* Refactor to use interval in browser to call to api proxy endpoint and get live stats directly instead of using websockets. Generally simplifies things.
* Remove empty/superfluous data from cache stats returned for live data
2022-05-23 14:13:28 -04:00
FoxxMD
99b46cb97f fix(polling): Improve logic for returning only 'new' Activities from polling sources
Prevents CM from iterating through n+1 pages of polling sources (mostly unmoderated) due to a source-of-truth change

See comments for scenario this helps avoid
2022-05-23 12:35:42 -04:00
FoxxMD
3ac07cb3e2 fix(cache): Fix cache init bug caused by passing additional (unnecessary) args 2022-05-23 11:12:51 -04:00
FoxxMD
d7f08d4e27 chore: Remove superagent browser dep 2022-05-19 15:49:32 -04:00
FoxxMD
338f393969 feat(logging): Simplify log streaming logic #91
* Get streamed logs directly from api (through proxy) in browser using streaming apis instead of through client websockets
* Use observer visibility to determine which logs to stream
* Timeout and abort any streaming logs if tab hasn't been visible for more than 15 seconds

TODO streaming system logs
2022-05-19 15:21:41 -04:00
FoxxMD
57e930ca8a feat(logging): Only load logs for initially viewed manager #91
* Don't include any logs in 'status' page response data
* Make log api call from browser to retrieve initial logs based on observer
2022-05-19 13:07:18 -04:00
FoxxMD
af3b917b57 feat(logging): Move log formatting for ui into client #91
* Use browserify to include logform functions and triple-beam symbols in client js
* Implement default log transform function in client js
* Remove formatted message and transport data from non-streaming log data sent to client
2022-05-19 12:11:33 -04:00
FoxxMD
d01bcc53fe fix(resource): Fix throwing the wrong type of error 2022-05-17 15:07:49 -04:00
FoxxMD
e2fe2b4745 feat(resources): Add more try-catch blocks for activity caching to make error tracing easier and reduce change of unhandled promise rejection 2022-05-17 15:01:18 -04:00
FoxxMD
785099b20c fix(database): Add missing managerId to premise entities?
Thought this was added automatically by typeorm by a user's logs seem to think otherwise
2022-05-17 15:00:39 -04:00
FoxxMD
726ceb03d2 fix(delay): Improve delayed activities interaction in manager and fix usage in resources
* Fix missing assignment for filtered activities after removing activity
* Remove 'processing' state from delayed activity lifecycle
  * Allows delayed queue to immediately remove activity after pushing to firehose -- simplifies lifecycle since another function (queue) doesn't have to handle this
    * Removing delayed activity function call from queue logic reduces calls to database
  * Likelihood an activity is cancelled while also processing is small (i hope...)
* Re-order item refresh logic in activity handling so delayed items are fetched before any proxy properties are accessed
* Catch errors on adding delayed activities from DB and just log -- not essential function
2022-05-17 14:18:55 -04:00
FoxxMD
1c37771591 chore: Update version constant 2022-05-17 14:12:17 -04:00
FoxxMD
67aeaea5f1 chore(dispatch): Add id manually to activity proxy when hydrated from database 2022-05-17 13:34:36 -04:00
FoxxMD
a8ac4b8497 fix(manager): Change activity property used to track queue meta
Use 'name' since this should *always* be present on a submission/comment -- when snoowrap creates an empty proxy it only includes 'name'.

* Potentially fixes facet of #64
* Fixes shouldRefresh undefined when activity is from delayed activities in database (empty proxy)
2022-05-17 13:33:37 -04:00
FoxxMD
71571d3672 fix(delay): Reduce delayed activities lifecycle complexity
* Refactor from async while depending on queue state into interval that always runs and just checks paused status of queue -- eliminates need to restart while loop if queue state is not running
* Add canary debug output for delayed activities to be able to know if it is actually running
2022-05-17 12:56:29 -04:00
FoxxMD
2799b6caeb fix(delay): Clean up dispatched action fields and parsing in ui
* Use duration field as SECONDS and remove additional time-based field (not necessary)
* Refactor dayjs usage in UI to parse action delay duration and correctly display "time until dispatch" to show if duration is negative
2022-05-17 12:10:23 -04:00
FoxxMD
e8f94ad1be Merge branch 'sentimentRule' into edge 2022-05-13 10:41:40 -04:00
FoxxMD
9080c7bdf4 Some more docs additions for subreddit state 2022-05-12 12:27:18 -04:00
FoxxMD
d418bf50eb Some cleanup on activity window docs 2022-05-12 12:22:48 -04:00
FoxxMD
5c8e73fee0 docs(activity window): Overhaul activity window docs
* Improve existing documentation with simpler language
* Organize/add table of contents
* Add window lifecycle diagram
* Add documentation for filter/fetch properties #55
2022-05-12 12:07:26 -04:00
FoxxMD
4411d1a413 feat(sentiment): Improve scoring and analysis
* Implement english-only scoring with wink https://github.com/winkjs/wink-sentiment (AFINN, emojis)
* Implement english-only scoring with NLP.js https://github.com/axa-group/nlp.js/blob/master/docs/v3/sentiment-analysis.md (AFINN, Senticon, Pattern)
* Refactor language processing into standalone functions for future use
* Add limited multi-langauge support
  * Can run sentiment with NLP.js on english, german, spanish, and french
* Normalize all scores to range between -1 and +1
* Improve score accuracy by averaging all scores
2022-05-11 16:09:29 -04:00
FoxxMD
c919532aac Merge branch 'edge' into sentimentRule 2022-05-11 12:42:15 -04:00
FoxxMD
676a3917e7 feat(database): Add planned rule/action types to database types
Getting ahead of the curve to avoid requiring migrations when these are released
2022-05-11 12:42:02 -04:00
FoxxMD
522ba33377 feat(rule): Implement sentiment rule using VADER #83 2022-05-11 12:39:23 -04:00
FoxxMD
3a18cc219f refactor(file organization): Move some comparison functionality into own file 2022-05-11 09:36:11 -04:00
FoxxMD
9cb5c4fe38 feat: More improvements for window functionality and dependency cleanup
* Add deprecation warnings to rules when building if properties should be migrated to window
* Add `debug` option to window to increase verbosity of filter logging. Default to false.
* Fix object assigment when building hash key for window filters
* Further cleanup for circular dependencies by moving some filter and logging functions into respective files
2022-05-10 12:49:56 -04:00
FoxxMD
57700b5e76 chore: Add nvmrc 2022-05-10 12:46:08 -04:00
FoxxMD
52ae16be1c refactor: Reduce circular dependencies 2022-05-09 20:12:08 -04:00
FoxxMD
951fab1070 docs: Add deprecation notices to window-related filters on rules 2022-05-09 16:23:01 -04:00
FoxxMD
0fd57af67e feat(window): More improvements for new window usage and caching
* Fix list function passed from author activities convenience method
* Move author history caching into main activity fetching function
* Do a better job at rehydrating snoowrap objects from cache data -- set as fetched, substitute relationships for non-fetching objects, and remove listing related objects
* Cache key for results based on window and pre-filter only -- post filter can be done after fetching cached results (Should save api calls!)
2022-05-09 15:34:19 -04:00
FoxxMD
3ad5b26be6 fix(ts): Fix TS issue with typings in same folder as regular files preventing schema generation
* Rename interface/types folder
* Move typigns back into folder with same name
2022-05-09 11:09:33 -04:00
FoxxMD
d20c2becc8 feat(window): Refactor window to support subreddit and item filtering, everywhere
* Move some interfaces and types into own files to breakup huge interfaces file
* Refactor window shape for config and "full" usage in app to support subreddit/item filtering
* Refactor author activities into resources class so we can take advantage of caching on subreddit/item filter results
2022-05-09 11:04:08 -04:00
FoxxMD
002bedf4b8 feat: Rebase in old subreddit window filter improvement
* subreddit filtering on window can now use include or exclude
* subreddit resources uses batch/cache retrieval
* temp fix to keep string subreddit name parity check in recent activity self inclusion logic
2022-05-05 23:03:18 -04:00
FoxxMD
11d32de9c5 chore: Bump dependencies for security fixes 2022-05-05 12:42:13 -04:00
FoxxMD
50303dfeb0 fix: Fix dryRun state handling for dispatched actions
Fixes scenario where a dispatched activity does not inherit DR state from currently processing activity

* Add dryrun state to dispatch activity data in app and database
  * Use general DR state for dispatched activity rather than DR for dispatched action
* Defer to explicitly defined DR in Task data when manager/queue handles task
2022-05-05 10:13:21 -04:00
FoxxMD
9b15c88547 fix: Handle more use cases when user is suspended, deleted, or shadowbanned
* Implement a "not serious" property for these errors so we don't report as an error to manager since they not an actual problem with the api or CM
* Check for [deleted] user name before trying to fetch an author
2022-05-04 12:57:16 -04:00
FoxxMD
4042305f49 fix: Fix rule result rendering when rule name is null 2022-05-04 12:02:48 -04:00
FoxxMD
ea9a867b3a fix: Handle scenario where author is suspended
* Throw friendly error when fetching activities
* Fail author criteria that are not present if author is suspended
2022-05-04 11:56:50 -04:00
FoxxMD
ff2415b024 docs: Add more links to main readme 2022-04-27 10:15:40 -04:00
FoxxMD
cf553cad59 docs: Re-organize docs structure and cleanup main readme 2022-04-27 10:09:13 -04:00
FoxxMD
1df9cd2b2a docs: Finish subreddit configuration docs refactor 2022-04-27 09:44:17 -04:00
FoxxMD
344dcc9879 refactor: New docs for subreddit configurations
More features documented with better examples. Not done yet...
2022-04-26 15:54:20 -04:00
FoxxMD
9ddebeb7fd fix: Don't clear queue meta on queue start
Can accidentally clear delayed items added to meta before queue is started
2022-04-26 09:12:51 -04:00
FoxxMD
3451cd5d3a fix(filter): Fix source assignment overwriting itemIs criteria
Need to deep clone itemIs criteria to prevent accidental assignment of new source
2022-04-25 12:37:15 -04:00
FoxxMD
2fa2606950 feat: Add option to disable delays when running activities as user
* Convenience for testing dispatched actions with delays
* Fix nullable hydrated dispatched action properties
2022-04-25 12:36:13 -04:00
FoxxMD
fc24b05eb3 fix: Fix links in documentation 2022-04-22 15:30:52 -04:00
FoxxMD
bfc7f8a508 docs: Rewrite and improve operator-related documentation
* Break out documentation into more standalone docs and reorganize into an operator folder
* Remove outdated information on adding bot
* Add additional information on docker install
* Make configuration more opinionated for "recommend" approach
* Add docs on database and caching
* Rewrite operator getting started guide to be more concise
2022-04-22 15:24:56 -04:00
FoxxMD
5a6f6b2680 Merge branch 'edge' into databaseSupport 2022-04-21 17:45:56 -04:00
FoxxMD
b7c314e9f5 fix(regex): Remove undefined matches from regex parsing
Could happen if there is a capture group that does not capture but entire expression does match
2022-04-21 17:43:55 -04:00
FoxxMD
24e9295aca feat(database): Implement event retention policy per subreddit
* user-configurable retention period (number of events OR duration) at operator, bot, subreddit override, and subreddit config level
* run database cleanup using retention policy on startup and every 30 minutes
* show retention policy in UI on manager overview
2022-04-21 15:03:02 -04:00
FoxxMD
0a1306130d feat(check): Implement post behavior to determine if event should be stored
* Extends postBehavior interface to allow specifying different record output options (database, influx)
* Can specify for *either* post behavior which enables storing events that were not triggered
2022-04-21 12:08:10 -04:00
FoxxMD
942d4fb2f2 feat(filter)!: Change behavior for rules that fail filter criteria tests
Instead of "skipping" the rule will now fail. This aligns Rule behavior with how filters work through the rest of CM which should reduce cognitive load and development effort.

If the skipping behavior is still desired a user can use a RuleSet with OR condition to achieve the same effect.
2022-04-21 10:19:54 -04:00
FoxxMD
fd173b6a2f feat(database): Implement user-configurable time series statistics
* Collect same stats as all time but on at a specified frequency
* Frequency is configurable at operator, bot, and subreddit level
* Operator and bot level frequency can have an enforced minimum
2022-04-20 13:44:35 -04:00
FoxxMD
c039a13fe6 fix(check): Fix how cached rules are counted for stats 2022-04-20 13:35:28 -04:00
FoxxMD
6d208b3daf fix(manager): Fix manager state parsing on init
* Add types for managing running state
* Only use invokee if state is stopped/user, otherwise always set to stopped/system
2022-04-20 11:46:23 -04:00
FoxxMD
65c159081e feat(database): Add related and author search params to event query 2022-04-19 14:44:09 -04:00
FoxxMD
d250c30f14 refactor(database): Improve event query performance
* Query for non-hydrated events to get ids then get fully hydrated objects using only ids -- dramatically improves performance
* Remove typeorm-pagination due to non-optimized count/select approach (should use typeorm getManyAndCount)
  * Also removes dup typeorm dependency
2022-04-19 12:54:18 -04:00
FoxxMD
c701e741af fix(database): Fix join/select for event querying 2022-04-19 10:02:48 -04:00
FoxxMD
3051b4af4e feat(ui): Improve events ux
* Make title a link to "default" events view
* Fix event link
* Always return first page when fetching events by permalink to reset any existing pagination state
2022-04-19 10:02:32 -04:00
FoxxMD
8085e8ced6 Merge branch 'edge' into databaseSupport 2022-04-19 09:30:29 -04:00
FoxxMD
749980abbe feat(repeat): Make identifying repeat sets from reference submission more fuzzy
When reference submission is a self post identifier may vary slightly since it is considering both title and body. Use string matching on identifies to find "close" matches for reference submission
2022-04-18 11:40:10 -04:00
FoxxMD
db83f1d0e7 Update schema after merge 2022-04-14 14:24:10 -04:00
FoxxMD
77960ba186 Merge branch 'edge' into databaseSupport
# Conflicts:
#	src/Schema/Action.json
#	src/Schema/App.json
#	src/Schema/OperatorConfig.json
#	src/Schema/Rule.json
#	src/Schema/RuleSet.json
#	src/Subreddit/SubredditResources.ts
2022-04-14 14:23:28 -04:00
FoxxMD
03fc562643 fix(filter): Remove/implement debug placeholder for bot self account substitution
oops
2022-04-14 13:44:07 -04:00
FoxxMD
9ef58dfd27 feat(filter): Implement testing for which moderator approved or removed an activity
Extends removed/approved criteria to accept a string, strings, or object with testing behavior and names
2022-04-14 13:31:56 -04:00
FoxxMD
1ce415bcd5 Merge branch 'edge' into databaseSupport
# Conflicts:
#	src/util.ts
2022-04-14 12:11:28 -04:00
FoxxMD
e4a9e47d6a fix(filter): Handle scenario where submissionState is incorrectly used on submission itemIs
* Check for found criteria results when building summary
* change log level from debug to warn when submissionState is detected on submission itemIs
2022-04-14 12:10:41 -04:00
FoxxMD
dc1a999376 Merge branch 'edge' into databaseSupport
# Conflicts:
#	src/Rule/RepeatActivityRule.ts
2022-04-14 10:07:35 -04:00
FoxxMD
61109fa4c0 feat(filter): Implement named filter reuse in configuration 2022-04-13 15:21:03 -04:00
FoxxMD
95eaa66080 feat(filter): Implement named filter criteria (author, item) infrastructure
Refactor internal code to handle anonymous or named criteria for filters
2022-04-13 11:28:32 -04:00
FoxxMD
bda4d82191 feat: Implement filtering subreddits by own profile
isOwnProfile in SubredditState allows checking if any Activity belongs to the user profile of the Author of the Activity being checked
2022-04-12 12:13:34 -04:00
FoxxMD
69cb823203 refactor(cache): Implement check caching correctly based on new check result data structure
* Use check result from DB if found
* Otherwise build simple rule results and partial check properties from cache data
2022-04-12 10:44:04 -04:00
FoxxMD
ec41c2f8ef feat: Show filter outcome as check summary when any failed 2022-04-12 10:43:02 -04:00
FoxxMD
964f9f9225 fix: show check name in events ui 2022-04-12 10:42:27 -04:00
FoxxMD
870b0068e9 refactor(ui): Refactor client and ui rendering to match data structures (rulesets) and render correctly 2022-04-12 09:44:10 -04:00
FoxxMD
26a563def3 fix: Correct display text for generic value comparison
Check if percent capture group is undefined OR empty
2022-04-12 08:53:55 -04:00
FoxxMD
a3685b8c81 refactor(database): Improve rule result associations
* Separate rule results from check result ownership since results can be re-used
* Implement ruleset data structure for better representation of results
* Add ruleset assoication to check result
* Partially refactor check caching (TODO)
2022-04-12 08:42:49 -04:00
FoxxMD
5710147291 fix(database): Use repository to save changes to manager state 2022-04-11 10:03:20 -04:00
FoxxMD
e7732c5a55 feat(filter): Implement exclude functionality for itemIs filter
* Same behavior/structure as authorIs
* Refactor interfaces to use a generic FilterOption type
* Allow simplified data for item/author filters -- when prop data is an array of criteria treat as "include OR"
2022-04-08 14:20:27 -04:00
FoxxMD
ccee0ef6c0 refactor: Move author interfaces and utility functions into common files 2022-04-08 10:42:16 -04:00
FoxxMD
2893f40ec5 fix(database): Check for undefined runs on CMEvent
Ensure that we only try to sort runs on an Event if they exist
2022-04-07 15:25:46 -04:00
FoxxMD
b670ad73c4 refactor(docker): Improve data directory flexibility and build order
* Create data directory if not /config
* Copy s6 files in base to make incremental builds faster
2022-04-07 15:23:14 -04:00
FoxxMD
6764995dfe refactor(docker)!: Switch to linuxserverio.io alpine as base and re-work init to make permissions less rigid
* Switching to lsio as base means inclusion of s6-overlay for improved init, handling of default permissions, and a maintained base for security/bug fixes
* Also means we can use PUID and PGID as variables instead of requiring --user in run command (more friendly to unraid, portainer, etc...)
* BREAKING: mount directory for configuration,etc. has been changed to /config by default to align with lsio
2022-04-07 14:58:10 -04:00
FoxxMD
11be4085be refactor(docker): Reclaim some space
* Clear npm cache after final production install to reclaim 100MB on image size
* More comprehensive dockerignore to prevent accidentally including development files
2022-04-07 13:10:33 -04:00
FoxxMD
31a3ae9a6b fix(database): Fix existing tables detection
* Use correct property from entity metadata which has all real db prefixes added (tablePath)
* Look for specified metadata table name instead of generic 'migrations'
* Simplify conditions (DRY)
2022-04-07 11:47:30 -04:00
FoxxMD
339b85e30a Merge branch 'edge' into databaseSupport
# Conflicts:
#	src/Action/UserNoteAction.ts
2022-04-07 10:54:30 -04:00
FoxxMD
f0eb1f040e feat(usernote): Make user note wiki edit reason actually useful
Include the note type, user affected, submission/comment id, and preview of text in edit reason
2022-04-07 09:53:33 -04:00
FoxxMD
5896623822 fix(ui): Improve action data parsing and fix undefined type 2022-04-07 09:26:49 -04:00
FoxxMD
3c6bd9f1e4 fix(ui): Fix passed value for instance id when no friendly name is set
Fixes #85
2022-04-07 09:17:11 -04:00
FoxxMD
af28ab5291 fix: Explicitly set sorting method for author activities
Don't rely on default sort chosen by reddit as comment/overview are sorted by new but submitted is sorted by hot (why??). Make sure requests always sort by new to prevent inconsistencies.
2022-04-06 19:03:45 -04:00
FoxxMD
27786d9e6f refactor(usernotes): Ensure notes subreddit is entirely independent of any parent object
To prevent any accidental referenced object property re-assignment only use subreddit name to construct subreddit for usernotes. Want to make sure notes don't get read/written to different subs by mistake...this shouldn't be happening but just to be double extra sure.
2022-04-06 13:26:40 -04:00
FoxxMD
dc25078b01 fix(cache): Fix missing re-hydration of cached subreddits 2022-04-06 13:03:16 -04:00
FoxxMD
565c0ea34f refactor(database): Move session secret logic into storage provider 2022-04-06 11:51:13 -04:00
FoxxMD
8a6e67aaae feat(database): Add more helpful logging for database init
* Normalize logging prefixes for database
* Make intent of sqljs location name change clearer
* Add warning for using sqljs in production
2022-04-06 11:26:45 -04:00
FoxxMD
5cbc669dfd refactor(database): Revert f4e5f1a2 and allow specifying cache/database for web independently
Since web DB needs to be built separately from server to allow migrations anyway might as well provide *all* the flexibility

* Reverts breaking change removing caching from web
* Add database config override in web config
2022-04-06 11:00:17 -04:00
FoxxMD
2b39644202 fix(database): Fix some lingering issue for web migrations
* Remove user name from log (there is no user)
* Fix endpoint verb for getting database logs
2022-04-06 09:39:56 -04:00
FoxxMD
3f1aeeb421 refactor(database): Separate web and server persistence
To make persistence align with client-server architecture

* Create separate datasource instances for client and server
* Refactor migration logic into service so code can be reused between client/server
* Implement endpoints for backup/migrate on client
* Move migrations into separate folders and move client sessions, invites, and web settings into own migration
* Defer session secret generation until client instantiation and save to database when possible
2022-04-05 16:32:52 -04:00
FoxxMD
f4e5f1a22f refactor(web)!: Migrate all web storage to use either database or session
* BREAKING: Replace web.caching with web.storage and restrict options to specifying either top-level database or cache
  * Simplifies init but provides options for different use cases
  * Default to database
* Implement generic storage provider for web persistence. Uses either cache or database
* Refactor session/invites to use storage provider
* Implement invite entity for database
* Allow specifying different storage provider for sessions
2022-04-05 13:18:18 -04:00
FoxxMD
c81703720b refactor(session): Migrate session storage into typeorm database 2022-04-05 10:56:47 -04:00
FoxxMD
ed06213c62 refactor(database): Simplify rule/action premise data structures
* Remove action/rule as standalone entities because they depend too much on premise data to make sense (not normalized). Instead, consolidate into premise entities.
* Breakout premise config into config/itemIs/authorIs data to make future querying easier
2022-04-04 13:37:51 -04:00
FoxxMD
402119d718 fix(app): Clearer startup logging 2022-03-31 13:13:07 -04:00
FoxxMD
2e8f90c1a8 feat(database): More database and config improvements for startup
* Improve errors when checking if file is readable or directory/file is writable
* Implement custom winston-typeorm logging mappings so migrations are output at INFO level
* Add typeorm logging options to database config and default to displaying error, warning, and migrations
* Ensure directory is writeable before using log rotation transport for winston
* Move init logger into index so its always available
* Simplify printing stack for SimpleError when it's a cause
* Better handling for access permissions when reading operator config file
* Add hint to errors relating to permissions access if app detects its running in docker
2022-03-31 13:05:07 -04:00
FoxxMD
408c495d63 refactor(dispatch): Move manager state peristence out of config-dependent resources for earlier access
So that syncing state doesn't require config to be valid
2022-03-31 09:51:50 -04:00
FoxxMD
f469dad106 refactor(dispatch): Add author name to database object to avoid have to fetch activities on startup
Storing author name means we can show delayed summary without having to fetch every delayed activity
2022-03-31 09:24:22 -04:00
FoxxMD
e9b76bc503 refactor(docker): Improve permissions and set default db driver
* Change project location to /home/node to avoid root permissions when creating WORKDIR
* Create default config directory with correct permissions in the event no volume is mounted
* Set default db driver to better-sqlite3 since we control environment -- it has better performance and we know we can use a pre-built binary b/c of base docker image
2022-03-30 16:20:07 -04:00
FoxxMD
db2c7d40da feat(config): Add DB_DRIVER env as convenience
* Makes it easier to switch between sqljs and sqlite
* Useful for docker
2022-03-30 16:17:35 -04:00
FoxxMD
cf909d49cc refactor(docker): Provide a default DATA_DIR to simplify mount binding 2022-03-30 12:56:41 -04:00
FoxxMD
4fdbb354ec feat(config): Improve/simplify configuration paths
* Add DATA_DIR env to allow specifying a "base directory" for all other config paths. Defaults to project root (same as existing behavior)
* OPERATOR_ENV OPERATOR_CONFIG LOG_DIR and database locations can now be specified as absolute OR relative (from DATA_DIR)
* Look for config.yaml and config.json if no OPERATOR_CONFIG is present
2022-03-30 10:55:55 -04:00
FoxxMD
dfbea31120 fix(manager): Fix missing dryrun value in runtime options when handling an activity 2022-03-29 17:02:26 -04:00
FoxxMD
1a4606ea74 refactor(dispatch): Simplify source data and dispatch/delay data
* Use typeorm to transform delay from int to duration
* Also fix run state init for managers
2022-03-29 16:49:04 -04:00
FoxxMD
44accf6043 fix(database): Handle decimal transform 2022-03-29 16:47:45 -04:00
FoxxMD
ad0e3fee26 fix(database): Fix interface for database connection options so schema can generate 2022-03-29 15:11:21 -04:00
FoxxMD
f9727f58a9 Bump minimum node version to LTS 2022-03-29 14:46:40 -04:00
FoxxMD
3feb1a3e73 refactor(database): Remove raw query migration 2022-03-29 14:44:18 -04:00
FoxxMD
dc1ef28653 fix(database): Force mysql/mariadb tables to a charset that supports emojis 2022-03-29 14:40:37 -04:00
FoxxMD
ba32853264 fix(database): Fix derp on manager name length 2022-03-29 14:40:21 -04:00
FoxxMD
c292da47df feat(database): Support better-sqlite3 as a database 2022-03-29 14:17:33 -04:00
FoxxMD
1436ae7452 refactor(database): Refactor datetimes to be support across all databases
FINALLY got a solution for supporting millisecond resolution on datetime data for sqlite, mariadb, and postgres
2022-03-29 13:32:55 -04:00
FoxxMD
c09a07a3db refactor(database): Use datetime type for time aware properties instead of forced timestamp
* millisecond timestamp using bigint doesn't translate well between databases typeorm/typeorm#2400 so avoid this
* regular int/unix epoch doesn't include milliseconds which we need for ordering results and required resolution
* sqlite and mysql/mariadb support datetime with millisecond precision so settle for that, for now
2022-03-29 11:33:52 -04:00
FoxxMD
4f9c382da7 feat(database): Ensure compatibility with mariadb
* Fix some columns after testing
2022-03-29 09:33:00 -04:00
FoxxMD
84ac0190c8 fix(database): Check for resource availability before syncing state 2022-03-28 20:23:49 -04:00
FoxxMD
50dad431cf refactor(database): Refactor migrations to use API so migrations are database-agnostic 2022-03-28 20:18:02 -04:00
FoxxMD
b91f287df0 refactor(database): Remove remaining create/update decorators and use time base class 2022-03-28 20:00:41 -04:00
FoxxMD
78be8c6a54 refactor(database): Add constructors for support data to make migrations easier 2022-03-28 20:00:14 -04:00
FoxxMD
70db28d81e fix(database): Fix check for migration database 2022-03-28 19:59:42 -04:00
FoxxMD
4696f53a89 feat(database): Implement manager run state persistence
* Persist events, queue, and manager states to database on change
* Use invokee/state from database on startup to determine if manager should be run after building
2022-03-28 15:04:00 -04:00
FoxxMD
630801a5e8 feat(database): Finish implementation of db migrations for all use-cases
* Refactor database init from backup and migration attempt so they can be invoked separately
* Implement more generic db backup function to make additional backup strategies possible in the future
* Add operator config option to try migrations if database backup succeeds
* Implement client web migration runner and database backup flows
2022-03-28 13:31:10 -04:00
FoxxMD
27cb5cdb83 fix(database): Fix support for in-memory database options when using sqljs 2022-03-28 13:28:17 -04:00
FoxxMD
5a4faf686a refactor(database): Use database label for logger instead of leaf
To support backup leaf
2022-03-28 13:27:36 -04:00
FoxxMD
e7fbd9baaf fix: Fix how general errors are transformed into errors with cause 2022-03-28 12:48:45 -04:00
FoxxMD
065014dc10 feat(ui): Implement finding actioned events by permalink 2022-03-25 15:07:40 -04:00
FoxxMD
ed13840881 feat(ui): Implement pagination for actioned events 2022-03-25 14:04:47 -04:00
FoxxMD
5c83d457f5 fix(database): Fix getting author name from snoowrap object for storing activity 2022-03-25 14:01:12 -04:00
FoxxMD
12daba0161 fix(dispatch): Fix including too many properties in dispatch data
Using spread was including client/logger/etc. which can't be serialized
2022-03-25 14:00:48 -04:00
FoxxMD
ebf7c00849 fix(filter): Fix queuedAt timestamp for re-hydrated delay activities 2022-03-25 13:59:47 -04:00
FoxxMD
1dd9090088 fix(filter): Include source in criteria for stored result, if source is used
* Only for *stored* result so that cached results don't include source
2022-03-25 13:59:20 -04:00
FoxxMD
dc2c35f822 refactor(ui): Finish fixing formatting for actioned events 2022-03-25 12:27:59 -04:00
FoxxMD
d916ccfb6e fix: Fix formatting for filter summaries
* Determine what actual crit object is based on object shape (could be from app or api)
* Initialize filters are undefined so they overwrite null values from api when transforming for ui
2022-03-25 11:56:02 -04:00
FoxxMD
5a10cf0f4d fix(database): Revert filter criteria to using hash-based id instead of random
* Accidentally changed this to random. Oops.
* Needed to move author crit result into same file as base class to fix a circular dependency issue
2022-03-25 11:54:39 -04:00
FoxxMD
323833ebb6 refactor(ui): Transform db response data into old actioned event interface for ui consumption
For now just structure it the same so the ui doesn't need to be refactored as well
2022-03-25 11:42:29 -04:00
FoxxMD
ed234baf76 refactor(events): Replace boilerplate filter joins with function builder 2022-03-25 11:41:28 -04:00
FoxxMD
5a7738b5b2 feat(database): Implement event pagination 2022-03-24 16:08:45 -04:00
FoxxMD
66a63b1a74 feat(database): Implement basic, all-time historical stat tracking in database
* Simplify/refactor stats to only track all-time in database...
  * remove "last reload" stats from app and ui
  * simplify tracked stats to counts only (no maps-of-types counts)
* Add migration function in stat init to convert any all-time stats in cache to database, when database is empty
2022-03-24 14:39:50 -04:00
FoxxMD
fc3479fb8a refactor(database): Update typeorm/dependencies and fix naming strategy
* Use updated logger adapter so that typeorm can be updated
* Update to typeorm 0.3.3
* Refactor naming strategy to be what i want it to be
2022-03-24 12:44:49 -04:00
FoxxMD
b5557335dc feat(database): Add indexes for unique data 2022-03-24 11:42:33 -04:00
FoxxMD
9b5111aabf feat(database): Implement activity source data structure
* Implement a generic activity source to track where an event was retrieved from and some initial parameters
* Consolidate dispatch action data into generic source
* Refactor delayUtil logic for event handling to prevent blocking worker for a non-trivial amount of time by dispatching event
2022-03-24 11:20:52 -04:00
FoxxMD
37f6878d54 feat(database): Implement dispatched action persistence
Add 'tardyTolerant' option to dispatch action config so user can determine if an action should still run if its pasts its dispatch date
2022-03-23 17:04:48 -04:00
FoxxMD
5edc116b13 refactor(database): Refactor createdAt implementation for more accuracy
* Implement own createdAt functionality using hooks and dayjs so we can use unix timestamp to eliminate TZ ambiguity and increase granularity to milliseconds
* hook insert/load to convert timestamp between dayjs and int
* Set value on class instantiation rather than object save to preserve actual creation time
2022-03-23 15:22:59 -04:00
FoxxMD
6c8bbdff1f feat(database): Make all non-updateable entities have random, friendly ids
* Replace any incremented, generated number id with a random id from nanoid
* Replace UUID with random id from nanoid
2022-03-23 14:48:17 -04:00
FoxxMD
1dde56c208 refactor(database): Add relations map for actioned event function
* And add event order by createdAt
2022-03-23 14:19:45 -04:00
FoxxMD
aba3989063 refactor(database): Drop support for mongodb
* CM data is highly relational so it doesn't make sense to support both rdbms and document dbs
* repository usage is different for mongo, not supporting both
2022-03-23 13:54:31 -04:00
FoxxMD
5c6a797703 refactor(database): Include reddit fullname and short identifier in activity entity for database
* Use setter/getter to parse id/name to make it easier to use fullname or short id interchangeably
* rename title to content to reflect property usage better
2022-03-23 13:32:06 -04:00
FoxxMD
5a236400f0 feat(database): Implement action results persisting to event 2022-03-23 13:00:21 -04:00
FoxxMD
21efc3024d fix(run): Fix triggered result for run to use actual run result entity 2022-03-23 12:59:54 -04:00
FoxxMD
b2d0403150 refactor(action): Move logic for touched entities for recent self into action to simplify 2022-03-23 12:59:12 -04:00
FoxxMD
298239039d refactor: Move common properties for runnable components (action,rule,check,run) into a base class
* DRY
* Add runFilters function to class
2022-03-23 12:10:09 -04:00
FoxxMD
8c33e44e3c fix(database): Fix duplicated filter criteria
* Refactor/remove Author class and replace with function (was unnecessary class)
* Workaround fix for composite key on filter criteria entity
2022-03-23 11:20:25 -04:00
FoxxMD
a16d0ad0f0 refactor(database): Remove unnecessary insert cascades
Relations for entities that are already persisted and should not be duplicated
2022-03-23 09:42:32 -04:00
FoxxMD
5971ef7565 refactor(database): Partially working implementation for actioned event using database
* Reworked many mappings to fix cascades and relationships
* Add constructors to entities where useful
* Renamed many db entities classes to be different than interface names
* Re-generated initial migration and added convenience npm script for running it
* Added ormconfig for typeorm 3 cli
* Create all run/check/rule/action premises (static info) on manager creation
2022-03-22 15:07:38 -04:00
FoxxMD
b62f7f7bb5 refactor: Move RuleResult interface and associated structures into interfaces common file 2022-03-22 09:01:06 -04:00
FoxxMD
b137e8682b feat(database): Implement action/action premise database creation
* Refactor init migration to fix mappings
* Implement initialization method for actions to hydrate their action/premise entities
2022-03-21 16:19:08 -04:00
FoxxMD
f19ac19b27 feat(database): Implement rule/rule premise database creation
* Add second migration script for adding initial rule types
* Implement initialization method for rules to hydrate their rule/premise entities
2022-03-21 15:59:52 -04:00
FoxxMD
e667c29da2 feat(database): Begin database entity creation and association on startup
* Create/find bot entity
* Create/find subreddit entities for each manager
* Create/find manager entities
* Pass bot/manager entities to Manager and SubredditResources
2022-03-18 15:37:38 -04:00
FoxxMD
bced882c75 refactor(db): Update typeorm to 0.3.0 2022-03-18 15:34:55 -04:00
FoxxMD
8c738ecd49 refactor(database): More entity mapping updates and migration refresh 2022-03-18 15:34:34 -04:00
FoxxMD
828f4cf676 fix(database): Fix db paths and logging options 2022-03-18 15:33:02 -04:00
FoxxMD
c70254061a fix(contributor): Update db-branch specific changes to support new contributor action 2022-03-18 15:31:36 -04:00
FoxxMD
8ce19a61f5 Merge branch 'edge' into databaseSupport
# Conflicts:
#	src/Action/index.ts
2022-03-18 12:14:12 -04:00
FoxxMD
b01e2a98f9 feat(contributor): Implement subreddit contributor functionality
* Add action to add/remove subreddit contributor
* Add 'isContributor' author criteria
* Use cache to store subreddit contributors
2022-03-18 11:38:04 -04:00
FoxxMD
b2a6ae440e fix(schema): Update schema to include new snoowrap options 2022-03-18 10:19:38 -04:00
FoxxMD
554d7dd86e feat(modnote): Action implementation and api shape updates
* Add comments to ongoing api development from reddit
* Implement mod note add action
2022-03-18 10:17:09 -04:00
FoxxMD
29c3924ab7 Merge branch 'edge' into modnote 2022-03-18 09:25:46 -04:00
FoxxMD
b61f99355f refactor(action): Add premise to action results 2022-03-17 14:18:47 -04:00
FoxxMD
b7f78f5eb1 refactor(database): New entities and init migration for database 2022-03-17 14:04:47 -04:00
FoxxMD
b561fa6b3c refactor(database): Partial conflict resolution with edge
Still need to refactor interfaces/code related to storing actioned events
2022-03-17 10:00:21 -04:00
FoxxMD
d82032073b Merge branch 'edge' into databaseSupport
# Conflicts:
#	package-lock.json
#	package.json
#	src/App.ts
#	src/Common/interfaces.ts
#	src/ConfigBuilder.ts
#	src/Subreddit/Manager.ts
#	src/Subreddit/SubredditResources.ts
#	src/Utils/loggerFactory.ts
#	src/Web/Client/index.ts
#	src/Web/Server/server.ts
#	src/Web/assets/views/events.ejs
#	src/index.ts
#	src/util.ts
2022-03-17 09:41:54 -04:00
FoxxMD
935d983626 feat(client): Implement snoowrap retry on timeout errors
Makes CM less sensitive to random blips in reddit API by enabling snoowrap to retry some network-issue related error codes alongside status codes

* Add timeoutCodes as an operator configurable array of error codes that is passed to extended snoowrap
* Patch snoowrap to check for timeout codes on request error and use retry logic if found
* Additionally, add retryErrorCodes (status codes) to operator configuration
2022-03-16 12:31:04 -04:00
FoxxMD
f6fb0dc877 docs: Add development instructions for mocking reddit api network issues 2022-03-16 12:02:30 -04:00
FoxxMD
3dd16bba79 fix(config): Fix snoowrap options not being passed to bot 2022-03-16 12:01:03 -04:00
FoxxMD
c46fe6f128 feat(dispatch): Add cancel action in ui for delayed actions 2022-03-14 12:06:43 -04:00
FoxxMD
074c3c7340 fix(migrations): Escape prefix value before parsing to regex 2022-03-11 15:25:12 -05:00
FoxxMD
8cd8374bbe feat(filters): Add more options for filtering by flair
* Add boolean options to test for "any" or "no" flair on author or submission
* Cleanup flair testing logic
2022-03-11 15:08:37 -05:00
FoxxMD
aa0541f09b refactor(author filter): Improve caching for author filter
* Refactor main author filter logic into subredditresources to take advantage of cache provider
* Implement methods to retrieve and cache subreddit moderators and author information
2022-03-11 14:21:16 -05:00
FoxxMD
eee166467d feat(ui): Improve live stats utilization
* Refactor live stats to work for "All Subreddits" as well as individual subs
* Refactor live stats to take place of opStats and update almost all bot stats live now (only cache breakdown TODO)
* Refactor opstats to return status of bots/subreddits only for ui indicators in tabs
2022-03-11 12:41:59 -05:00
FoxxMD
95b0e529e2 fix(schema): Relax schema for source type due to issues with generation 2022-03-10 14:58:08 -05:00
FoxxMD
45be87a72a fix(item filter): Fix unknown value results
* For unknown value on activity use correct reason and add to property map
* Test for undefined property in property map (shouldn't happen)
2022-03-10 13:04:16 -05:00
FoxxMD
d632364c7d fix(item filter): Fix multiple and unknown criteria
* Log different statements based on whether value or key was undefined
* Fix for-loop break when testing multiple criteria
2022-03-10 12:49:35 -05:00
FoxxMD
9e660214eb feat(schema): Add annotations for dispatched/source 2022-03-10 12:48:27 -05:00
FoxxMD
14340b3a65 fix(ui): Include dayjs plugin for sameOrAfter comparison 2022-03-10 12:47:50 -05:00
FoxxMD
b07402628e fix(dispatch): Push activity from user through firehose so delayed activities get same behavior as from polling 2022-03-10 11:42:30 -05:00
FoxxMD
035283a596 refactor(item filter): Improve source comparison
* Move activity source normalization and verification into own function (thrown on invalid source string)
* Correct source-filter comparison by comparing source to filter rather than other way around to make sure inclusive filter is passed
2022-03-10 11:41:36 -05:00
FoxxMD
cc46f00a22 refactor(dispatch): Rename everything associated with rerun to dispatch
* Also rename item filter from 'dispatch' to 'dispatched' to match verb tense of other state properties
* Simplify identifier property name in config to just 'identifier' -- there's enough context for what it is already
2022-03-10 11:05:21 -05:00
FoxxMD
27263928cd Merge branch 'edge' into rerun 2022-03-10 10:14:58 -05:00
FoxxMD
0f122466ad feat(editor): Retrieving config schema from local URL instead of github
So that our schemas finally match whats in our code!
2022-03-10 10:12:31 -05:00
FoxxMD
32cdb29515 fix(ui): Only fetch reddit status if status element is present on page 2022-03-10 10:11:51 -05:00
FoxxMD
fe311ced32 feat(testing): Add suites for testing activity state 2022-03-09 17:15:40 -05:00
FoxxMD
e41bea7e6b fix(util): Always return false for filtered activity check if user is not a moderator 2022-03-09 17:15:21 -05:00
FoxxMD
9d169cebf3 Add test js/maps to gitignore 2022-03-09 17:14:53 -05:00
FoxxMD
5551f2c63f feat(modnote): Initial modnote creation implementation 2022-03-09 15:36:46 -05:00
FoxxMD
ff3e704cdf fix(client): Simplify client logging to ui and fix instance name
* Correctly render system logs to html
* Simplify websocket logging so it matches how logs are received o browser from server
* Fix instance redirect name when no friendly is set for api config
2022-03-09 12:11:14 -05:00
FoxxMD
caaeb2eefb fix(ui): Set system logs as seen in special case 2022-03-09 12:09:05 -05:00
FoxxMD
8991797d35 feat(testing): WIP added initial testing framework and some util tests
Using mocha, chai, and nyc

* tests for parsing string for numeric value comparison
* tests for parsing string for durations and duration comparisons
* tests for parsing reddit entity (subreddit/user) from string
* tests for parsing submission/comment id from reddit permalink string
* tests for initial config parsing/merging

Still can't get nyc to get coverage for everything in src using "all" -- causes reporting to show 0 for everything??
2022-03-09 10:59:43 -05:00
FoxxMD
aa95c26b2a Merge branch 'edge' into rerun 2022-03-08 21:30:53 -05:00
FoxxMD
11cc90e2d5 feat: Add modnote oauth permission for auth helper 2022-03-08 21:30:37 -05:00
FoxxMD
d11e511f67 Merge branch 'edge' into rerun 2022-03-08 13:02:25 -05:00
FoxxMD
a3708ca279 refactor(ui): Improve live stats usage and add delayed item info
* Update activities and bot usage stats (just overview) with live data from websocket
* Add Delayed count with tooltip to show delayed items overview
2022-03-08 13:00:40 -05:00
FoxxMD
14d0417a25 refactor(dispatch): Rename rerun to dispatch action 2022-03-07 17:04:55 -05:00
FoxxMD
f4103206db docs: some clarifications in flow control wording 2022-03-07 13:17:17 -05:00
FoxxMD
c9b1bfed40 fix(docs): mermaid issue was being inside closed detail block
For some reason. Removing blocks fixed rendering error
2022-03-07 13:11:45 -05:00
FoxxMD
7f764b4d99 fix(docs): Replace colon causing issues with mermaid diagram rendering 2022-03-07 13:06:12 -05:00
FoxxMD
fb7ddbba70 docs: Add overview for runs and flow control #73 2022-03-07 13:02:17 -05:00
FoxxMD
85b1d13718 docs: Refactor docs to use runs syntax
* Add Runs to main docs readme and concepts
* Add high level diagram in main docs readme to show CM lifecycle
* Refactor subreddit/rule examples to use runs syntax
2022-03-07 10:28:48 -05:00
FoxxMD
7f2191a11a fix: Fix bug where submission identifier was accessed via 'id' instead of 'name' property
* Was causing uncaught promise rejection in userflairaction because it should have been accessing name instead of id
* Wrap all as/is utility functions where value may be from cache (plain object) or proxy (from snoowrap) with try-catch to prevent any more uncaught promise rejections -- would rather swallow silently (for now) than crash the entire application
2022-03-06 19:30:25 -05:00
FoxxMD
c4adf4f495 feat(ui): More identifier readability and succinctness improvements 2022-03-04 21:50:48 -05:00
FoxxMD
95d146a504 feat(ui): Make dispatch action context in actioned event more succinct
Move summary into tooltip and add relevant details to header
2022-03-04 21:39:41 -05:00
FoxxMD
ccc8a0dab5 Merge branch 'edge' into rerun
# Conflicts:
#	src/Web/assets/views/events.ejs
2022-03-04 21:37:39 -05:00
FoxxMD
9b79bdbdd5 feat(ui): Make submission context in actioned event more succinct 2022-03-04 21:08:43 -05:00
FoxxMD
1f3d0b50a7 feat: Implement re-run
* Implement rerun configuration that satisfies requirements from #72
  * rerun as action
  * optional, user-defined identifier
  * cancel rerun as action
  * cancel based on re-queued sources
  * on existing behavior
  * can specify initial goto
* filter item by source (where item was retrieved from for non-cached items)
* filter item by rerun state/identifier
* Add rerun label to event logging
* Add rerun data to actioned event data
2022-03-04 15:52:05 -05:00
FoxxMD
d8d409ae6b Some rerun basics 2022-03-03 16:34:39 -05:00
FoxxMD
6b9852cc14 feat(recent activity): Implement ability to compare distinct subreddits found
Closes #77
2022-03-03 13:16:39 -05:00
FoxxMD
fbf627c971 feat(actioned event): Add more context for ui and search
* Add id and activity type to event activity data
* Include parent submission activity data if activity is a comment
* Refactor event page ui to simplify headers and move content into collapsible
  * Add context to content by including submission context for comments
2022-03-02 15:10:27 -05:00
FoxxMD
b2077132cf fix(ui): Don't strip urls from body when previewing comment in events 2022-03-02 13:31:18 -05:00
FoxxMD
f622c3ee03 fix(errors): try-catch on transform for good measure
Would rather just get top-level error than crash the app and log a bad transformation (which tells us nothing about the original error)
2022-03-02 11:22:59 -05:00
FoxxMD
ab83f3ed0c fix(errors): Another better check for respsone headers 2022-03-02 11:16:48 -05:00
FoxxMD
a021b503a0 fix(errors): Better check for response headers on error transform 2022-03-02 11:13:32 -05:00
FoxxMD
d28714aacc fix(usernote): Fix user note instantiation and link handling
* Fix argument order for new note object
* Better handling for notes without a reference link
2022-03-02 09:53:48 -05:00
FoxxMD
7632a66250 fix(user flair): Fix dry check 2022-03-01 13:45:27 -05:00
FoxxMD
bb6936d657 fix(check): Correctly get run actions for notification payload 2022-02-28 14:42:44 -05:00
FoxxMD
d4062b679a refactor(filter): Better handling for submissionState item criteria
* Allow FilterResult as a property result
* Remove pre-item testing cache optimization for submissionState to simplify flow
  * Helped reduce key count but not worth the cost of overly complex code for returning filter results
* Remove expected prop from results and instead use criteria in filter results to generate this for logs/events
* Refactor log/event generation to handle FilterResult in filter property result
2022-02-28 13:04:21 -05:00
FoxxMD
313ee0a9a3 fix(usernotes): Better handle missing moderators (no more errors thrown)
Fixes edge case where a usernote was created by a moderator that no longer mods the sub

* Store mod index so we can recreate note even if moderator is missing
* Refactor moderator hydration on usernote from raw data to just warn if moderator cannot be found
2022-02-28 12:58:24 -05:00
FoxxMD
7afc384d17 fix(ui): Fix filter numbering in events page 2022-02-28 12:56:26 -05:00
FoxxMD
fea1f240dd Merge branch 'edge' into flowControl 2022-02-28 10:01:52 -05:00
FoxxMD
1dba0e857f fix(logging): Better data shape checking for error transformer 2022-02-28 10:01:30 -05:00
FoxxMD
0966aa689f Merge branch 'edge' into flowControl 2022-02-22 15:55:00 -05:00
FoxxMD
138e237fbc fix(comment): Fix the target of the lock intention
Should be locking the created comment instead of the activity being checked
2022-02-22 13:29:58 -05:00
FoxxMD
6b38ec1669 Merge branch 'edge' into flowControl 2022-02-22 11:21:14 -05:00
FoxxMD
5cd415e300 Bump version 2022-02-22 11:11:29 -05:00
FoxxMD
7cdaa4bf25 fix(migrations): Remove unnecessary log warning for all logs on live stats 2022-02-22 11:10:51 -05:00
FoxxMD
280ddf583b fix(migrations): Fix missing TTL for cache migration 2022-02-22 10:50:44 -05:00
FoxxMD
4969cafc97 fix(ui): Add missing dayjs plugins for timestamp formatting 2022-02-22 10:43:17 -05:00
FoxxMD
5f6e63542b feat(flow): Implement max goto depth configuration 2022-02-22 10:42:52 -05:00
FoxxMD
bca9c96468 fix(ui): Fix run summary typo 2022-02-22 09:44:49 -05:00
FoxxMD
7569c06a36 Merge branch 'edge' into flowControl
# Conflicts:
#	src/Bot/index.ts
#	src/Subreddit/Manager.ts
#	src/Web/Client/index.ts
#	src/Web/Server/routes/authenticated/user/index.ts
#	src/Web/assets/public/app.css
#	src/util.ts
2022-02-22 09:27:36 -05:00
FoxxMD
88bafbc1ac fix(ui): Fix not clearing intervals on client disconnect 2022-02-21 16:47:17 -05:00
FoxxMD
a5acd6ec83 feat: Refactor client/secret api interaction to improve fetching data and enable live stats
* Only return logs for "default viewed" subreddit/bot when fetching instance status, when specified from QS
  * Greatly reduces amount of data fetched and response time
* Return logs with formatted property for non-streaming response
* Implement server live stats endpoint to return subreddit/all stats based on QS
* Use client websocket connection to return stats for currently viewed subreddit
2022-02-21 16:14:41 -05:00
FoxxMD
d93c8bdef2 Merge branch 'docUpdates' into edge 2022-02-21 12:00:37 -05:00
FoxxMD
8a32bd6485 Merge branch 'edge' into logRefactor
# Conflicts:
#	src/Web/Client/index.ts
2022-02-18 15:45:28 -05:00
FoxxMD
425cbc4826 feat: Improve user agent reporting and version display in ui 2022-02-18 15:16:37 -05:00
FoxxMD
3a2d3f5047 refactor(logging): Use logging from CMInstance instead of client
Reduces logging complexity and has better single responsibility
2022-02-18 13:37:07 -05:00
FoxxMD
ae20b85400 refactor(client): Refactor server instance into own class
* Move from plain data with interface to class and refactor heartbeat logic into class
* Makes logging easier and cleans up client code
2022-02-18 13:09:33 -05:00
FoxxMD
e993c5d376 refactor(logging): Move log collection into bot/manager for better single responsibility
* Move "sorting" log objects into lists for retrieval from server and into bot/managers for each log object type
* Refactor log filtering and aggregration under status/log endpoints to use logs from each entity rather than pulling from server

Reduces complexity in historical log data structures at the expense of slightly more runtime data crunching. The trade-off is well worth it and paves the way for easier retrieval of single/subsets of logs
2022-02-18 11:58:13 -05:00
FoxxMD
80fabeac54 fix(usernote): Fix adding new note to user note cache AFTER clearing cache
* Fixes an issue where the cached notes for a user only contain the last added note instead of all notes + new
* Also reduced api calls by caching moderator adding new note instead of calling each time
2022-02-18 09:54:18 -05:00
FoxxMD
c001be9abf feat(ui): Add reddit status indicator with link 2022-02-17 16:14:36 -05:00
FoxxMD
639a542fb2 fix(ui): Fix default values for scopes and permissions when not available 2022-02-17 13:53:06 -05:00
FoxxMD
9299258de0 feat(ui): Add moderator permissions list to subreddit overview 2022-02-17 13:37:42 -05:00
FoxxMD
59f8ac6dd4 feat(ui): Add oauth scopes list to bot overview
Visible when user is an operator
2022-02-17 13:29:37 -05:00
FoxxMD
f16155bb1f fix(flair): Fix snoowrap function used for assigning flair template id 2022-02-17 13:17:17 -05:00
FoxxMD
e2d2f73bb3 feat: Add log warning when user has no access 2022-02-15 11:00:09 -05:00
FoxxMD
9ca5d6c8c2 fix: Fix config builder to supply more defaults for a minimal configuration
* Provide a default redirect uri
* Don't add default bot instance if no credentials were specified
2022-02-14 12:07:54 -05:00
FoxxMD
4f9d1c1ca1 docs: Some clarifications for install/run directions 2022-02-14 10:54:25 -05:00
FoxxMD
d8f673bd26 fix(remove): Only warn if item looks removed, on remove action
If the item is not actually removed (it's hard to tell from reddit api) we don't want to prematurely end remove action. Just warn and try to remove anyway
2022-02-14 09:31:50 -05:00
FoxxMD
7e2068d82a fix(author): Ensure automoderator is always detected as a moderator for author isMod test 2022-02-14 09:30:54 -05:00
FoxxMD
176611dbf3 docs: Add web interface and config onboarding 2022-02-11 23:40:28 -05:00
FoxxMD
372bae0e03 feat(ui): Implement displaying author/item filters for actioned events 2022-02-10 15:15:21 -05:00
FoxxMD
6f35ec3705 fix: Set process result error property with easier-to-read error message and include "not enabled" 2022-02-10 15:14:55 -05:00
FoxxMD
a542d80c1d feat(migration): Update cache migration for new actioned event structure 2022-02-10 13:54:49 -05:00
FoxxMD
9dcf256aa1 feat(filter): Add filters to all result data 2022-02-10 13:43:33 -05:00
FoxxMD
da206f41ad feat(filter): Refactor itemIs to use FilterCriteriaResult
Normalizes item is filter so it behaves and produces the same type of results as author filter
2022-02-10 13:37:04 -05:00
FoxxMD
550beb9baf feat: Add restricts on flow control to prevent endless loops
* Restrict checks/gotos to being run once
* Throw flow control errors instead of catching to ensure we exit flow early
2022-02-10 11:21:12 -05:00
FoxxMD
3d99406f33 Merge branch 'persistActions' into edge 2022-02-09 17:09:24 -05:00
FoxxMD
7f9adcef36 refactor: Refactor manager/run/check handling for better single responsibility 2022-02-09 16:47:57 -05:00
FoxxMD
ab355977ba fix(approve): Fix approval check target 2022-02-09 16:41:39 -05:00
FoxxMD
f24eb52697 Merge branch 'edge' into flowControl 2022-02-09 13:15:01 -05:00
FoxxMD
60dbc42148 Merge remote-tracking branch 'origin/flowControl' into flowControl
# Conflicts:
#	src/Check/index.ts
#	src/Subreddit/Manager.ts
#	src/util.ts
2022-02-09 13:12:52 -05:00
FoxxMD
8667fcdef3 fix(stats): Correctly initialize all time historical stats from cache when stat is empty 2022-02-09 13:10:05 -05:00
FoxxMD
ec20445772 refactor(ui): Use checkmark symbol that matches x symbol (no emojis) 2022-02-09 13:09:39 -05:00
FoxxMD
8d9fb29848 feat(ui): Implement run context for actioned events
* Refactor events view to show checks within runs
* Build cohesive runs server-side before rendering so user can see all checks in a run together
* Add collapse/expand behavior for activity/run/check with ability to toggle based on triggered state
* Default to collapsing all non-triggered states
* Build check summary on-the-fly instead of storing in event result data
2022-02-09 13:08:48 -05:00
FoxxMD
f7a7e817f9 feat(cache): Implement cache migrations
* Store migration state in cache instance
* Migrate on default cache init or private cache init
* Implement first migration to deal with run structure in actioned events
2022-02-08 17:00:18 -05:00
FoxxMD
e09cab6872 refactor(check): Refactor structure for storing check/action result for runs
* Use array of check result objects that contain rule/action results
* Include post behavior on check summary
2022-02-08 14:35:26 -05:00
FoxxMD
f1797f29fd Merge branch 'persistActions' into flowControl
# Conflicts:
#	src/util.ts
2022-02-08 13:39:36 -05:00
FoxxMD
4eae07f831 feat(flow): Implement basic flow control structures #73
* Add Run and postCheckBehavior config structures to schema and interfaces
* Implement parsing from config and initial flow logic for running on activities in manager
2022-02-08 13:39:02 -05:00
FoxxMD
0293928a99 feat(cache): Implement cache key manipulation based on key pattern
* Implement glob pattern or regex as argument
* Implement scan search for redis for efficiency otherwise iterate keys using generic function
* Implement cache reset based on passed item from action -- reset item crit for activities, author crit for users, and overwrite any cached activity
2022-02-08 13:01:09 -05:00
FoxxMD
b56d6dbe7c fix(actions): Only include successfully run actions in notification summary 2022-02-07 22:21:22 -05:00
FoxxMD
42d269e28d feat(actions): Mutate activities during actions for immediate use and ensure cache is synced 2022-02-07 16:21:43 -05:00
FoxxMD
8f60a1da53 feat(regex): Add option to stop rule early if current activity does not match
In order to prevent history from being pulled (and using api) if user indicates current activity must also match
2022-02-07 15:15:50 -05:00
FoxxMD
f511be7c33 fix(usernote): Throw error with cause when usernote fails instead of logging quietly
* Makes error cause easier to see in stack and fixes error now logging during action failure
* Use error with cause for logging action error for clearer stack
2022-02-07 12:41:10 -05:00
FoxxMD
ebb426e696 feat(filter): Add isRedditMediaDomain submission state criteria 2022-02-07 10:36:56 -05:00
FoxxMD
63696b746e feat(flow): Implement basic flow control structures #73
* Add Run and postCheckBehavior config structures to schema and interfaces
* Implement parsing from config and initial flow logic for running on activities in manager
2022-02-04 15:06:21 -05:00
FoxxMD
c07276a3be fix(logging): Fix typo in error transform 2022-02-01 13:13:27 -05:00
FoxxMD
4a2297f5cd docs: Add github sponsor link 2022-02-01 12:01:34 -05:00
FoxxMD
f8967d55c4 feat(repeat): Use newer text comparison technique to improve repeat detection
* Use same technique as repost rule which has high accuracy and let false-positives
* Implement ability to see similarity score, case sensitivity, and text transformations
2022-01-31 14:08:21 -05:00
FoxxMD
7e8745d226 fix(polling): Fix shared polling behavior for nanny mode changes
* On hard limit stop shared streams
* On nanny mode turned off restart any stopped shared streams
2022-01-27 16:49:03 -05:00
FoxxMD
e2efc85833 fix(polling): Fix running state not changed on error
* Set running to false when error is caught. Was not caught on last stream refactor which changed polling behavior to end if any error is caught rather than waiting for external source to clear interval
* Add debugging/error messages on polling start/stop
2022-01-27 16:47:43 -05:00
FoxxMD
41038b9bcd feat(logging): Implement richer errors everywhere
* Use ErrorWithCause so we can get and print a chain of error causes
* Make reddit error response in stack trace more readable by replacing them with a "translated" parent response and add them as the cause
* Properly handle error formatting for winston by looking at shape of log object for error rather than testing instanceof (see comments in errorAwareFormat)
* Fix formatting in web interface for log lines with white-space pre css and properly splitting timestamp from rest of the message
2022-01-27 16:27:03 -05:00
FoxxMD
9fe8c9568c refactor: Move SimpleError into main Errors module 2022-01-27 11:48:23 -05:00
FoxxMD
9614f7a209 refactor(logging): Implement snoowrap errors "the right way" and implement consolidated logging function
* Implement declaration file for snoowrap errors so they can be imported directly
* Implement logging function to handle boilerplate for known error responses (reddit HTTP response, rate limit, etc.)
2022-01-27 11:43:39 -05:00
FoxxMD
8dbaaf6798 fix(logging): Defaults for log file dir 2022-01-26 12:28:56 -05:00
FoxxMD
c14ad6cb76 feat(logging): Implement separate logging options for each transport type
* Add properties for file, console, and stream in logging object of operator config
* Each property inherits a (useful) subset of winston transport options
2022-01-26 12:09:03 -05:00
FoxxMD
adda280dd3 fix(logging): Fix parsing log dir
* Correct else condition to use log dir when value is not true
* Set level to 'debug' on init logger if no value is provided to help with debugging
2022-01-26 10:27:01 -05:00
FoxxMD
15fd47bdb4 fix(polling): Correct typings for stream getter and check isFinished for Listing 2022-01-26 10:11:06 -05:00
FoxxMD
78b6d8b7b6 feat(polling): Add debug messages when streams are stopped 2022-01-26 10:00:09 -05:00
FoxxMD
61bc63ccc5 fix(polling): Emit config change event to bot only after manager has rebuilt polling 2022-01-26 09:50:30 -05:00
FoxxMD
05df8b7fe2 fix(polling): Use manager eventState to control shared stream callback rather than removing callback when events are stopped
Should prevent edge cases where shared streams are re-parsed while managers are stopped (hard limit) and then removed due to there being no callbacks
2022-01-25 18:07:15 -05:00
FoxxMD
3cb7dffb90 fix(polling): Prevent endless loop when trying to enforce continuity on a stream with no items returned 2022-01-25 09:25:59 -05:00
FoxxMD
d0aafc34b9 feat(remove): Add option to mark activity as spam 2022-01-21 13:03:05 -05:00
FoxxMD
d2e1b5019f chore: Update packages 2022-01-21 13:02:31 -05:00
FoxxMD
2a77c71645 fix(usernotes): Fix wiki entity handling to avoid unhandled rejection
Since snoowrap's WikiPage isn't a "real" object setting it as a property on the class means if it rejects the whole application crashes. Fix this by building wiki proxy every time we need it before awaiting promise for edit/retrieval so that promise scope is bound to the function we are in (that has try-catch)
2022-01-20 14:10:39 -05:00
FoxxMD
780e5c185e refactor(author filter): Strongly structure comparison/matching data for more consistent manipulation and output
* Use interface for comparison results at both criteria property level and criteria level
* Implement summary functions to build string results of comparisons
* Output all comparisons to debug and provide summaries to verbose (when applicable)
2022-01-20 14:08:54 -05:00
FoxxMD
38e2a4e69a fix(filter): Missing return on flair failure comparison 2022-01-19 15:49:44 -05:00
FoxxMD
7e0c34b6a3 fix(userflair): Fix wrong assignment for css 2022-01-19 13:10:11 -05:00
FoxxMD
e3ceb90d6f fix(filter): Fix default excludeCondition type
* Expected (prior) behavior is that all exclude criteria must pass, not just one
* Fix missing AND condition logic when all conditions pass
2022-01-19 13:09:45 -05:00
FoxxMD
6977e3bcdf feat(author): Add flair template criteria for author/submission
* Add filtering by flairTemplate id for author/submission
* Refactor flair properties for author/submission to accept string or array of strings
2022-01-19 12:48:58 -05:00
FoxxMD
f382cddc2a fix(filter): Change array merging behavior for authorIs defaults to be more sane
* Don't just overwrite (duh)
* Drop any default filters that include object keys that are also present in user-defined filters -- this way user-defined always takes precedence on merge
2022-01-19 11:52:18 -05:00
FoxxMD
99a5642bdf fix(ui): Change time formatting from 12 to 24 hour 2022-01-18 16:49:07 -05:00
FoxxMD
174d832ab0 docs: Pretty up readme header 2022-01-18 16:08:40 -05:00
FoxxMD
3ee7586fe2 fix(approve): Fix touched entity 2022-01-18 13:37:56 -05:00
FoxxMD
e2c724b4ae feat(approve): Implement approving parent submission of comment 2022-01-18 13:37:22 -05:00
FoxxMD
d581f19a36 feat(logs): Use log objects in api to improve parsing client-side
* Add options for /logs endpoint to stream objects instead of strings
* Always return log objects from /status endpoint -- fixes bug where all bots/subreddits got lines from logs that had newlines
* Return context-aware, formatted log lines to client to reduce line length IE if returning to botA -> subA then do not need to include labels for botA,subA #40
* Shorten timestamp to just time and wrap full timestamp in tooltip #40
* Emit log objects to client to reduce parsing complexity (don't have to regex for bot/subreddit name)
2022-01-18 12:59:59 -05:00
FoxxMD
48dea24bea feat: Improve first-run display in ui and add system view
* Fix bugs in UI when bot does not have a name (configured incorrectly)
* Implement instance system log view for operators
2022-01-18 10:38:39 -05:00
FoxxMD
5fc2a693a0 fix(config): Fix empty yaml config document initialization 2022-01-18 00:06:52 -05:00
FoxxMD
7be0722140 fix(bot): Fix limit rate expiration getter when there is no client initialized 2022-01-18 00:06:24 -05:00
FoxxMD
6ab9fe4bf4 feat(config): Implement persisting bots from invite process to application and config
* write to config when bot is added
* replace/add based on existing bot
* implement specify instance from instances user is operator of
* implement specify subreddits to run on using comma-separated list
* rewrite invite flow ending to be more clear on results and next steps
2022-01-17 17:47:27 -05:00
FoxxMD
5811af0342 feat(config): Refactor config parsing to preserve comments and enable writing
* use node-comment and yaml@next to keep comment information intact
* store ast/source version of parsed config for operator
* implement generic yaml/json operator config classes to keep everything organized and simplify marshalling source to js/string
* refactor file parsing and json/yaml parsing to have better single responsibility
2022-01-17 15:51:43 -05:00
FoxxMD
3cc6b30e8b feat(database): database file permissions check 2022-01-17 11:19:22 -05:00
FoxxMD
856112c2f6 Merge branch 'edge' into databaseSupport
# Conflicts:
#	.gitignore
2022-01-17 11:18:55 -05:00
FoxxMD
ed2924264a feat(util): Better check for file/dir permissions 2022-01-17 11:18:23 -05:00
FoxxMD
e9394ccf2e refactor(tooling): Ignore sqlite files 2022-01-17 09:52:18 -05:00
FoxxMD
dec72f95c6 docs: Add discord invite link 2022-01-14 16:42:01 -05:00
FoxxMD
d6bfd63deb feat(database): Implement database migration flow for App
* refactor bot init to include removing any old running ones (out of server and into app)
* run database prep before init bots
2022-01-14 15:13:28 -05:00
FoxxMD
d62ed3daf5 feat(database): Add init migration 2022-01-14 15:12:25 -05:00
FoxxMD
3bb9c2cee3 feat(databse): Add missing typeorm requirements and logger adapter
* ormconfig using default sqlite connection to generate migrations
* add winston adapter to provide logger to typeorm
2022-01-14 15:05:00 -05:00
FoxxMD
72f8d5d0f6 fix: database-related updates/conflict resolution
* Add require TS compiler options for typeorm
* Update package-lock
* Fix duplicate imports in interfaces
2022-01-14 11:21:29 -05:00
FoxxMD
a1c508fc2c Merge branch 'edge' into databaseSupport
# Conflicts:
#	src/Common/interfaces.ts
#	src/ConfigBuilder.ts
#	src/Schema/OperatorConfig.json
#	src/Subreddit/SubredditResources.ts
#	tsconfig.json
2022-01-14 11:19:34 -05:00
FoxxMD
80c11b2c7f refactor(filter): Consolidate authorIs logic and add additional control to exclude logic
* Add excludeCondition to control how exclude sets are tested (and/or)
* Refactor authorIs logic from check/rule/action into standalone function (DRY)
* Simplify filter defaults -- don't need to specify automoderator since it is always a mod
2022-01-14 10:51:29 -05:00
FoxxMD
e6a2a86828 feat(config): Implement default filter criteria behavior
* Add default behavior config to operator and manager config
* Implement configurable behavior when filter is present on check
* Add defaults to exclude mods and automoderator from checks
2022-01-13 16:46:32 -05:00
FoxxMD
96749be571 refactor(polling): Simplify and cleanup all polling logic
* Remove unused clearProcessing code
* Use same data structures (Map) for storing polling objects in both Manager and Bot to reduce cognitive load and re-use some logic
* Rename "mod" streams to "shared" streams
* Implement detection and updating of polling when manager config changes
* Implement detection and updating of shared streams on manager config update
* Use shared retry handler for manager polling to better handle general reddit api issues (all polling stops faster)
* Move initial polling buffer into polling object (instead of in manager) for better logic encapsulation and add debug logging for it
* Add more debug logging for manager/bot poll building
2022-01-13 11:39:16 -05:00
FoxxMD
6b7e8e7749 feat(polling): Implement shared streams for all polling sources
* Refactor polling config to use new 'shared' string list of polling sources and deprecate 'sharedMod' property
* Refactor how shared sources are built to look for shared intention in manager polling options before creating
* Implement continuity check for comment/submission polling to ensure no activities are missed
* Add debug logging to polling
2022-01-12 15:47:43 -05:00
FoxxMD
43b29432a2 refactor(auth): Refactor auth data structures to consolidate logic
* Add abstract user class with auth methods with implementations for client/server
* Refactor client/server logic to use class methods instead of inline auth checks

Closes #71
2022-01-12 09:57:38 -05:00
FoxxMD
ff84946068 feat(regex): Experimental support for parsing regex expressions from fetched URL
* Support fetching from reddit wiki
* Support fetching from raw URL
* Support parsing and fetching from gist, github blob, and regexr (very experimental)
2022-01-11 14:05:57 -05:00
FoxxMD
7cdde99864 fix(recent): Potential fix for reddit ACID issues on history retrieval 2022-01-11 13:00:51 -05:00
FoxxMD
8eee1fe2e1 fix(recent): Remove code that should have been deleted during refactor
Refactored recent to use batch subreddit testing but forgot to remove old, individual subreddit testing, code so activities were being counted twice
2022-01-11 10:15:16 -05:00
FoxxMD
6fc09864f6 fix: Don't delete property from object
Object passed by ref, duh
2022-01-11 10:13:48 -05:00
FoxxMD
1510980ce3 fix(util): Ensure provided state description is reattached to strong sub state 2022-01-11 10:13:14 -05:00
FoxxMD
56005f0f28 fix(bot): Fix own profile detection when building managers 2022-01-11 09:52:44 -05:00
FoxxMD
03b655515c fix(server): Fix logs not persisting for managers
* Change manager acquisition so all managers belong to a bot before they start logging so all logs are captured correctly
* Fix log capture logic that prevented all subreddits from being populated
2022-01-11 09:45:25 -05:00
FoxxMD
edd874f356 fix(server): Correctly filter bots and managers on auth on server 2022-01-11 09:15:52 -05:00
FoxxMD
7f13debe3b fix(client): Make sure all moderated subreddits are fetched 2022-01-10 16:17:24 -05:00
Matt Foxx
1565bdbf1a Merge pull request #67 from rysie/feature/dry-run-buttons
Run/Dry run buttons
2022-01-10 14:54:42 -05:00
FoxxMD
ec4cee8c77 refactor(ui): Fix and simplify button logic
* Fix url query selector to constrain to sub
* Use shared class between run buttons to simplify class modification and click event
2022-01-10 14:54:17 -05:00
Matt Foxx
04b8762926 Merge pull request #68 from rysie/feature/flair-docs
feat(docs): User flair and submission flair docs
2022-01-10 12:31:56 -05:00
FoxxMD
dcc5f87c30 refactor(docs): Clean up flair docs
* Fix regex escaped characters
* Use authorIs
* make flair action type usage more clear
2022-01-10 12:22:16 -05:00
FoxxMD
66d9c0b2a7 fix(auth): Fix bug allowing any valid moderator to see all instances 2022-01-10 10:27:13 -05:00
FoxxMD
00e7cad423 fix(auth): Logout bot after auth flow is complete 2022-01-10 10:26:45 -05:00
Marcin Macinski
bc541d00d4 feat(docs): User flair and submission flair docs 2022-01-08 00:02:37 +01:00
Marcin Macinski
c5b27628b0 feat(ui): Run/Dry run buttons 2022-01-07 23:32:12 +01:00
Matt Foxx
ede86d285b Merge pull request #62 from rysie/user-flair-action
UserFlairAction added
2022-01-06 14:55:27 -05:00
FoxxMD
52f6aabb69 feat: Prevent bot from running on reports/comments it just created
Cache reported items or new comments made by bot for a short time (default to twice polling interval, 1 minute) to prevent bot from running on things it did itself
2022-01-06 14:54:17 -05:00
FoxxMD
18175f3662 feat(item filter): Support checking for different report types: total, user, mod 2022-01-06 13:13:10 -05:00
FoxxMD
68a272d305 fix(ui): Fix subreddit intersection check for bot related routes
Remove any prefixed r/ from a bot's subreddits when checking intersection with user subreddits
2022-01-06 12:29:19 -05:00
FoxxMD
3dac91fafc fix(recent): Fix default behavior for submissionReference based on activity type
Eliminates noisy logging when it's not specified but activity is comment
2022-01-06 12:09:04 -05:00
FoxxMD
e5bb8c2a38 fix(bot): Reduce retries for more aggressive fallback on reddit api issues
* Reduce retry for snoowrap to 2 since we do our own error handling in-app and 2 is enough for the occasional, non-systemic blip
* Reduce manager retries
2022-01-05 20:46:54 -05:00
FoxxMD
61e0baf3fd feat(recent): Add combined karma to template variables 2022-01-05 17:08:15 -05:00
FoxxMD
37e9d1fcc2 fix(polling): Fix set timeout args 2022-01-05 14:28:19 -05:00
FoxxMD
5e70ca1cb6 fix: Fix and improve code related to stopping bots when reddit api is not OK
* Fix polling timeout to actually stop on error by simplifying timeout and waiting until response is OK to recreate next timeout call
* Use "unexpected exception" retry count for all non well-known "reddit blip" responses in retry handler rather than failing immediately AND log this distinction
* Fix managers not emitting errors from checks
* Fix bot not awaiting retry handler on manager error emit
* Increase nanny loop delay on error to reduce api pressure when there are many bots running
* (unrelated) set bot as running before starting managers so UI is available earlier
2022-01-05 12:58:17 -05:00
FoxxMD
7f7ed18927 refactor(server): return app earlier so UI is available earlier
Bot init can finish asynchronously without any negative affect to server/client. Returning earlier means we can access server info earlier in startup
2022-01-05 12:50:55 -05:00
FoxxMD
efed3381fd feat(config): Allow top-level operator snoowrap config 2022-01-05 10:39:43 -05:00
FoxxMD
79ee086efd refactor(database): Use database for actioned events write and read
* Implement all entities for actioned events
* Pass global database to subreddit resources
* Read/write actioned events using database, constraining to bot/subreddit
2022-01-04 16:22:46 -05:00
FoxxMD
b910b554e6 feat(database): Add database entities 2022-01-04 12:59:33 -05:00
FoxxMD
bee3935270 feat(database): Add database backend capability
* Add typeorm dependency with backend drivers for sqljs and, optionally, postgres/mongodb/mysql/mariadb
* Add operator configuration structure for global database connection
* Implement config parsing and defaults for sqljs db location or in-memory fallback

#66
2022-01-04 12:12:24 -05:00
FoxxMD
5ac5d65a28 refactor(userflair): Fix dryrun usage and add unflair functionality
* Can flair user on comment/submission
* fix dryrun if-else block (maybe a debugging artifact?)
* allow all properties to be undefined/null/empty and use as intention to unflair user
2022-01-03 21:02:21 -05:00
FoxxMD
0ae74fdce1 docs: Add missing config box screenshot 2022-01-03 13:42:35 -05:00
FoxxMD
845173822c docs(onboarding): Update moderator getting started to reflect new editor capabilities 2022-01-03 13:40:17 -05:00
FoxxMD
edb3036957 feat(editor): Resolve raw url from github blob url 2022-01-03 13:31:54 -05:00
FoxxMD
3790f0e061 feat(editor): Implement wiki page creation
If page is not created yet detect it and require more perms for re-authorization
2022-01-03 13:23:31 -05:00
FoxxMD
e3e4e4abff feat(editor): Resolve raw gist url from gist id 2022-01-03 11:50:23 -05:00
FoxxMD
fd9b83437b feat(editor): Use manager format to infer format for editor
* Provide wiki format in manager data to status api
* Remove additional View link and instead infer format for querystring from manager data
2022-01-03 11:17:04 -05:00
FoxxMD
05694f115c fix(editor): Fix how initial editor value is set
For some reason providing the data directly to a new model doesn't trigger validation and also had some other weird effects. Instead, using an empty string as initial value and then set model value to data afterwards -- which fixes everything b/c idk why.
2022-01-03 11:15:29 -05:00
FoxxMD
70ee157198 feat(config): Make manager aware of config format
* Default format to yaml
* Add detected config format as property to manager
* When neither format is valid use starting character to (naively) detect json or not
* Reduce config error noise by only show one format error based on likely type and print other to debug
2022-01-03 10:58:02 -05:00
FoxxMD
bbb4ec3c2d fix: Fix missing hyphen character on regex for parsing reddit entity name
Reddit users can have hyphens in their names. Slight tradeoff for allowing hyphens for subreddit names (they are non existent) to allow all valid reddit user names is worth it.

Found and corrected by @prometheus-22
2021-12-31 00:04:25 -05:00
FoxxMD
acb72551ec fix(bot): Add missing invite check in healthloop 2021-12-30 23:09:09 -05:00
FoxxMD
bf6affe592 feat(bot): Add basic programmatic moderator invitation acceptance
* Store subreddits to try to accept invites from in bot's default cache
* Handle invitation scenarios (none, modself missing, accepted) and starting manager after invitation
* Implement basic invitation acceptance list control in UI for bot operators
2021-12-30 22:52:37 -05:00
FoxxMD
8c2cb02a46 feat(bot): Add modself oauth permission
Used for accepting moderator invitations
2021-12-30 20:25:51 -05:00
FoxxMD
73e2af2100 feat(manager): Improve wiki page creation
* Only try page creation if response error is a 404
* Improve permission error descriptions
* Only create if it can also set page permissions to improve security
2021-12-30 19:08:47 -05:00
FoxxMD
ba4c4af5a7 Update invite view 2021-12-30 18:00:11 -05:00
FoxxMD
9ad21ee2dd feat(bot): Add ability to create non-existing wiki page and change visibility 2021-12-25 21:13:55 -05:00
FoxxMD
b32c4f213c docs(yaml): Finish adding yaml examples
#61
2021-12-25 16:42:44 -05:00
FoxxMD
7e01c8d1f8 docs(yaml): add yaml examples fo activities window
#61
2021-12-25 16:28:58 -05:00
FoxxMD
aee158ecc9 docs(yaml): add attribution and advanced concepts yaml examples
#61
2021-12-25 16:18:13 -05:00
FoxxMD
8cd2243c2d docs(yaml): add history and author yaml examples
#61
2021-12-25 16:07:46 -05:00
FoxxMD
4969789532 docs(yaml): add recentActivity yaml examples
#61
2021-12-25 15:53:54 -05:00
FoxxMD
1dcfdc14d1 docs(yaml): add regex yaml examples
#61
2021-12-25 15:23:34 -05:00
FoxxMD
f1c9b64f64 docs(yaml): Add more examples
#61
2021-12-23 21:35:18 -05:00
FoxxMD
2e5a61566b docs(yaml): create yaml versions of subreddit ready examples
#61
2021-12-23 12:34:50 -05:00
Matt Foxx
85761fa662 Merge pull request #65 from rysie/edge-submission-flair-fix
Flair action fix: import + assigning flair by flair_template_id
2021-12-23 09:33:34 -05:00
Marcin Macinski
0b1a6bd77b Flair action fix: import + assigning flair by flair_template_id 2021-12-23 12:34:08 +01:00
Marcin Macinski
51e299ca99 Merge branch 'edge' into user-flair-action 2021-12-22 01:13:33 +01:00
Marcin Macinski
7696f3c2ff UserFlairAction added 2021-12-22 00:45:59 +01:00
FoxxMD
1c9ed41e70 feat(ui): Implement basic wiki editing capabilities for editor
* Use stored scope to determine if user can save
* Only show save action if loaded from a subreddit
* Implement re-authorization flow through popup window and sockets to update status in editor
* Implement wiki location endpoint for server and wiki save endpoint for client
2021-12-21 16:30:57 -05:00
FoxxMD
2d67f9f57d refactor(ui): Migrate all editor usage to monaco-yaml base
* monaco-yaml can also do json validation since its just normal monaco
* simplifies config.ejs greatly not having to maintain two different monaco implementation, at the expense of a larger project
2021-12-21 14:22:26 -05:00
FoxxMD
975bcb6ad7 feat(ui): Enable additional scopes usage
* Store scopes in user auth object/sessions
* Implement requesting additional scopes through login route
2021-12-21 12:30:33 -05:00
FoxxMD
0d087521a7 feat(docs): Specify yaml can be validated in editor 2021-12-21 09:35:06 -05:00
FoxxMD
fb5fc961cc feat(docs): Add youtube credentials requirements for repost rule 2021-12-21 09:33:24 -05:00
FoxxMD
c04b305881 feat(docs): Add link-spamming regex config 2021-12-21 09:22:35 -05:00
FoxxMD
5c5e9a26aa refactor(ui): Unify editor look-and-feel
* Use named entry files for webpack so we can access them from config.ejs
* Move some of the validation logic to config.ejs so it can be used by json editor as well
* Use same controls for loading files/schemas/urls for both editors
* Add validation errors box for both editors
2021-12-20 23:30:41 -05:00
Matt Foxx
477d1a10ae Merge pull request #60 from rysie/edge
Support setting user flair by flair_template_id
2021-12-20 18:37:36 -05:00
Marcin Macinski
bbee92699c Support setting user flair by flair_template_id 2021-12-21 00:10:40 +01:00
FoxxMD
7f09043cdf fix(ui): Add missing yaml asset files
Need to unignore them...
2021-12-20 15:56:36 -05:00
FoxxMD
768a199c40 feat(ui): Add quick-n-dirty YAML editor
Using one-off project cm-yaml http://github.com/foxxmd/cm-yaml to run a static version of monaco-yaml inside CM
2021-12-20 15:42:37 -05:00
FoxxMD
6e4b0c7719 feat(regex): Add match sample to rule results so it can be used in content template 2021-12-20 12:33:40 -05:00
FoxxMD
89b21e6073 fix(repost): Use average of all algos to get high score
Ensures no anomalous scores dominate
2021-12-18 18:44:52 -05:00
FoxxMD
da611c5894 fix(repost): Fix max weighted score and formatting 2021-12-18 18:22:38 -05:00
FoxxMD
2c90a260c0 fix(stream): Remove processing clear behavior for polling to try to fix weird polling behavior 2021-12-18 18:08:00 -05:00
FoxxMD
f081598da6 feat(repost): Improve text matching and default values
* Use 3 different matching algorithms using the highest score out of the three
* Weight score based on length of the sentence
* Increase minimum number of words to 3
* Enforce min word count on external (youtube) comments
2021-12-18 17:42:43 -05:00
FoxxMD
55f45163a4 feat(repost): Fix some bugs and improve result data
* Wrong hash check
* Add closest match summary and percent for use in content
2021-12-18 14:54:01 -05:00
FoxxMD
e4dfa9dde3 refactor(docker): reduce image size with best practices
* use --no-cache so repository index isn't included in image
* use FROM instructions to build project with dev dependencies, then copy built project to new stage and install production-only libraries
* properly ignore docs and node_modules with dockerignore
2021-12-17 15:10:22 -05:00
FoxxMD
0e395792db refactor: Remove unusused ts-auto-guard package
Has @ts-morph as a dependency which is a huge package
2021-12-17 14:36:03 -05:00
FoxxMD
dcbeb784e8 refactor: Remove set-random-interval
* depended on and always downloaded an entire, older typescript version (even with production install) -- was not necessary for one function
* refactor project to use newer TS version (specify any type for catch blocks to fix compiler errors)
2021-12-17 14:25:48 -05:00
FoxxMD
aeaeb6ce27 refactor(docker): Remove monaco editor local dependency to reduce image size
Wasn't necessary to have it installed locally (can use CDN) and is not a part of the core experience
2021-12-17 14:00:54 -05:00
FoxxMD
d6a29c5914 fix(bot): Actually get all moderated subreddits
Previously was only getting first page of moderated subreddits (listing). Make sure to get all pages.
2021-12-17 13:24:09 -05:00
FoxxMD
c1224121d4 feat(subreddit): Implement user profile filtering for SubredditState
* Add convenience property for filtering subreddits by user profile prefix (u_)
* Do some smart property comparison to check if SubredditState name is regex or not -- use additional user prof check if it is so we don't clobber existing name regex

Closes #56
2021-12-17 13:01:47 -05:00
FoxxMD
9790e681ea docs(regex): Add subreddit-ready example for removing discord link spam 2021-12-16 16:27:06 -05:00
FoxxMD
a48a850c98 Update winston repository package format/information 2021-12-16 14:58:43 -05:00
FoxxMD
b8369a9e9f Package bump for security
https://github.com/advisories/GHSA-93q8-gq69-wqmw
2021-12-14 12:06:48 -05:00
FoxxMD
0c31bdf25e refactor(repost): Improve repost criteria configuration and add documentation
* Simplify usage b/w comment-submission by removing "criteria" in main criteria property -- comment checks can just use additional properties
* Consolidated occurrence count/time into one property to allow and/or operands on both conditions (more powerful!)
* Added documentation describing repost configuration
* Added repost configuration examples
2021-12-07 16:30:05 -05:00
FoxxMD
4b14e581dd fix: Add more staggering on heartbeat check
Always stagger subreddit checks
2021-12-06 17:31:00 -05:00
FoxxMD
b2846efd2b fix: Improve falloff behavior when reddit api error are encountered
* Turn off snoowrap request queuing
* Aggregate errors from all managers at bot-level and force all to stop (and clear queue/polling) if a small threshold is met
* Add activity refresh on check into try-catch so delayed activities in queue don't cause a loop if they fail due to api issues
2021-12-06 17:16:04 -05:00
FoxxMD
a787e4515b feat(repost): Implement occurrence count and time comparisons
* User-defined set of comparisons for testing how many reposts were found
* User-defined set of comparisons for testing when a slice/all of reposts were created
2021-12-06 13:28:02 -05:00
FoxxMD
f63e2a0ec4 Merge branch 'topComments' into edge 2021-12-01 17:34:51 -05:00
FoxxMD
9d0e098db1 feat(rule): Implement repost rule
* checks for both submission/comment
* search by submission title, url, dups/crossposts, and external (just youtube for now)
* user-defined string sameness for all search facets
* user-defined case-sensitivity and regex-based transformations for activity/repost item values
* cache comment checks
* implemented youtube client for retrieving video comments
2021-12-01 17:34:25 -05:00
FoxxMD
181390f0eb fix: Correct error description for reading a config file
Was using copy-pasted statement from wiki error which was confusing
2021-12-01 09:33:03 -05:00
FoxxMD
a8c7b1dac9 Interim implementation of repost rule 2021-11-30 20:42:40 -05:00
FoxxMD
027199d788 docs(heroku): Add information on dyno usage 2021-11-15 17:27:42 -05:00
FoxxMD
2a9f01b928 fix(heroku): Add OPERATOR to heroku deploy template 2021-11-15 17:26:34 -05:00
FoxxMD
cf54502f0d fix(heroku): Add redirect uri env to heroku deploy template 2021-11-15 17:26:34 -05:00
FoxxMD
2a3663ccc9 refactor(heroku): heroku-specific docker change
* Add image for web and worker
* Refactor port usage to conform to heroku's usage
2021-11-15 17:26:34 -05:00
FoxxMD
dc2eeffcb5 fix(bot): Remove own profile "sub" from default moderated subreddits to run on
The user profile for the bot account is returning as a moderated "subreddit" EX (/u/ContextMotBot) but its not useable by CM because it does not have a wiki and users don't use this as a sub. Filter it out of moderated subreddits when using the default behavior.
2021-11-15 13:19:21 -05:00
FoxxMD
93de38a845 fix(filter): Missing 'op' property test on itemIs for comment 2021-11-15 12:47:34 -05:00
FoxxMD
43caaca1f2 fix(config): Fix wrong provider data used when bot has no caching config
Use previously built defaultProvider, instead of hardcoded memory config, so any cache config from operator waterfalls to bot/subreddit

Closes #46
2021-11-02 15:55:03 -04:00
FoxxMD
7bcc0195fe fix(author): change include/exclude on RULE to be optional
One of these properties must be present for the rule to be valid. This is enforced at runtime. Change the schema so that both are optional from the config-side though.
2021-11-02 09:41:55 -04:00
FoxxMD
2504a34a34 refactor(docker): Remove old steps from node-canvas dep
Not using node-canvas anymore so dockerfile can be simplified #48
2021-11-01 15:55:12 -04:00
FoxxMD
e19639ad0d fix(ui): Forgot to add tailwin css to git
Oops!
2021-11-01 15:06:50 -04:00
FoxxMD
b8084e02b5 Merge branch 'colorImprovements' into edge 2021-11-01 14:58:01 -04:00
FoxxMD
2cea119657 feat(ui): Add tooltip descriptions for depleted/limit reset stats
Closes #36
2021-11-01 14:52:35 -04:00
FoxxMD
6f16d289dd feat(ui): Add lang attribute for html
Closes #35
2021-11-01 14:22:20 -04:00
FoxxMD
a96575c6b3 refactor(ui): Use local tailwind css asset
Closes #34
2021-11-01 14:19:47 -04:00
FoxxMD
0a82e83352 refactor(ui)!: Implement colorbind color changes for rest of UI and remove dark/light mode
* Apply changes made in ac409dce to the rest of the application
* Remove dark/light mode -- now always dark mode (easier to maintain this way)
* Remove dependency on tailwind dark css
2021-11-01 14:14:49 -04:00
FoxxMD
d5e1cdec61 fix(criteria): Improve criteria filtering for removed/deleted activities and logging
* Use different logging messages when criteria is not available due to mod permissions (property not available to non-mods)
* Change logging level for missing/unavailable criteria to reduce logging noise. On unavailable use debug, on missing use warn
* Improve activity removed/deleted detection based on whether activity is moddable by current user
2021-11-01 13:25:54 -04:00
FoxxMD
ef40c25b09 feat(attribution): Add additional subreddit and activity filtering functionality to criteria
* Refactor subreddit filtering with include/exclude to use subreddit state
* Add submissionState and commentState filters
2021-11-01 11:26:55 -04:00
FoxxMD
6370a2976a Merge branch 'edge' into colorImprovements 2021-11-01 10:23:40 -04:00
FoxxMD
d8180299ea fix(author): Fix missing true return statement for author flair check 2021-11-01 10:23:22 -04:00
FoxxMD
ac409dce3d POC color improvements for color blindness
Needs another pass and consolidate/clean up for pages other than status
2021-10-20 21:12:31 -04:00
FoxxMD
56c007c20d feat(author): Implement author profile description regex/string testing
May test "description" in authorIs as a regular express of string literal -- or as an array of the aforementioned values
2021-10-20 19:52:46 -04:00
517 changed files with 86194 additions and 12007 deletions

View File

@@ -1,8 +1,24 @@
node_modules
Dockerfile
.dockerignore
.gitignore
.git
src/logs
/docs
logs
.github
docs
node_modules
coverage
.nyc_output
.idea
*.bak
*.sqlite
*.sqlite*
*.json
*.json5
*.yaml
*.yml
*.env
# exceptions
!heroku.yml
!app.json
!.nycrc.json
!.mocharc.json
!tsconfig.json
!package*.json

3
.github/FUNDING.yml vendored Normal file
View File

@@ -0,0 +1,3 @@
github: [FoxxMD]
patreon: FoxxMD
custom: ["bitcoincash:qqmpsh365r8n9jhp4p8ks7f7qdr7203cws4kmkmr8q"]

3
.github/push-hook-sample.json vendored Normal file
View File

@@ -0,0 +1,3 @@
{
"ref": "refs/heads/edge"
}

View File

@@ -1,4 +1,14 @@
name: Publish Docker image to Dockerhub
name: Publish Docker image to registries
# Builds image and tags based on the type of push event:
# * branch push -> tag is branch name IE context-mod:edge
# * release (tag) -> tag is release name IE context-mod:0.13.0
#
# Then pushes tagged images to multiple registries
#
# Based on
# https://github.com/docker/build-push-action/blob/master/docs/advanced/push-multi-registries.md
# https://github.com/docker/metadata-action
on:
push:
@@ -13,8 +23,12 @@ on:
jobs:
push_to_registry:
name: Push Docker image to Docker Hub
name: Build and Push Docker image to registries
runs-on: ubuntu-latest
# https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token
permissions:
packages: write
contents: read
steps:
- name: Check out the repo
uses: actions/checkout@v2
@@ -25,12 +39,22 @@ jobs:
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GitHub Container Registry
if: github.event_name != 'pull_request'
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v3
with:
images: foxxmd/context-mod
images: |
foxxmd/context-mod
ghcr.io/foxxmd/context-mod
# generate Docker tags based on the following events/attributes
tags: |
type=raw,value=latest,enable=${{ endsWith(github.ref, 'master') }}
@@ -40,7 +64,8 @@ jobs:
latest=false
- name: Build and push Docker image
uses: docker/build-push-action@v2
if: ${{ !env.ACT }}
uses: docker/build-push-action@v3
with:
context: .
push: ${{ github.event_name != 'pull_request' }}

14
.gitignore vendored
View File

@@ -336,6 +336,7 @@ web_modules/
# dotenv environment variables file
.env
.env.test
*.env
# parcel-bundler cache (https://parceljs.org/)
.cache
@@ -381,4 +382,17 @@ dist
.pnp.*
**/src/**/*.js
**/tests/**/*.js
**/tests/**/*.map
!src/Web/assets/**
**/src/**/*.map
/**/*.sqlite
/**/*.bak
*.yaml
*.json5
!src/Schema/*.json
!.github/push-hook-sample.json
!docs/**/*.json5
!docs/**/*.yaml
!docs/**/*.json

View File

@@ -2,10 +2,13 @@
<module type="WEB_MODULE" version="4">
<component name="NewModuleRootManager">
<content url="file://$MODULE_DIR$">
<sourceFolder url="file://$MODULE_DIR$/tests" isTestSource="true" />
<excludeFolder url="file://$MODULE_DIR$/temp" />
<excludeFolder url="file://$MODULE_DIR$/.tmp" />
<excludeFolder url="file://$MODULE_DIR$/tmp" />
<excludeFolder url="file://$MODULE_DIR$/src/logs" />
<excludeFolder url="file://$MODULE_DIR$/coverage" />
<excludeFolder url="file://$MODULE_DIR$/.nyc_output" />
</content>
<content url="file://$MODULE_DIR$/node_modules" />
<orderEntry type="inheritedJdk" />

4
.mocharc.json Normal file
View File

@@ -0,0 +1,4 @@
{
"require": ["./register.js", "source-map-support/register"],
"reporter": "dot"
}

1
.nvmrc Normal file
View File

@@ -0,0 +1 @@
16.18.0

24
.nycrc.json Normal file
View File

@@ -0,0 +1,24 @@
{
"extends": "@istanbuljs/nyc-config-typescript",
"exclude": [
"node_modules/",
"**/src/Schema/**",
"**/src/Web/assets/**",
"**/tests/**",
"register.js",
"**/src/**/*.d.ts"
],
"include": [
"**/src/**/*.ts",
"**/src/**/*.js",
"**/src/**/*.js.map"
],
"extension": [
".ts"
],
"reporter": [
"text-summary",
"html"
],
"report-dir": "./coverage"
}

View File

@@ -1,39 +1,133 @@
FROM node:16-alpine3.12
FROM lsiobase/alpine:3.15 as base
ENV TZ=Etc/GMT
RUN apk update
# borrowed from node/alpine:3.15
# https://github.com/nodejs/docker-node/blob/main/16/alpine3.15/Dockerfile
#
# Start of node docker stuff
#
ENV NODE_VERSION 16.14.2
# required dependencies in order to compile linux-musl (node-canvas) on alpine
# https://github.com/node-gfx/node-canvas-prebuilt/issues/77#issuecomment-884365161
RUN apk add --no-cache build-base g++ cairo-dev jpeg-dev pango-dev giflib-dev
# required dependencies in order to compile linux-musl (node-canvas) on alpine
RUN apk add --update --repository http://dl-3.alpinelinux.org/alpine/edge/testing libmount ttf-dejavu ttf-droid ttf-freefont ttf-liberation ttf-ubuntu-font-family fontconfig vips
RUN apk add --no-cache \
libstdc++ \
&& apk add --no-cache --virtual .build-deps \
curl \
&& ARCH= && alpineArch="$(apk --print-arch)" \
&& case "${alpineArch##*-}" in \
x86_64) \
ARCH='x64' \
CHECKSUM="a6dc255e1ef1f20372306eec932b4a3648575c6d3024bcd685b8efc93dc95569" \
;; \
*) ;; \
esac \
&& if [ -n "${CHECKSUM}" ]; then \
set -eu; \
curl -fsSLO --compressed "https://unofficial-builds.nodejs.org/download/release/v$NODE_VERSION/node-v$NODE_VERSION-linux-$ARCH-musl.tar.xz"; \
echo "$CHECKSUM node-v$NODE_VERSION-linux-$ARCH-musl.tar.xz" | sha256sum -c - \
&& tar -xJf "node-v$NODE_VERSION-linux-$ARCH-musl.tar.xz" -C /usr/local --strip-components=1 --no-same-owner \
&& ln -s /usr/local/bin/node /usr/local/bin/nodejs; \
else \
echo "Building from source" \
# backup build
&& apk add --no-cache --virtual .build-deps-full \
binutils-gold \
g++ \
gcc \
gnupg \
libgcc \
linux-headers \
make \
python3 \
# gpg keys listed at https://github.com/nodejs/node#release-keys
&& for key in \
4ED778F539E3634C779C87C6D7062848A1AB005C \
141F07595B7B3FFE74309A937405533BE57C7D57 \
94AE36675C464D64BAFA68DD7434390BDBE9B9C5 \
74F12602B6F1C4E913FAA37AD3A89613643B6201 \
71DCFD284A79C3B38668286BC97EC7A07EDE3FC1 \
8FCCA13FEF1D0C2E91008E09770F7A9A5AE15600 \
C4F0DFFF4E8C1A8236409D08E73BC641CC11F4C8 \
C82FA3AE1CBEDC6BE46B9360C43CEC45C17AB93C \
DD8F2338BAE7501E3DD5AC78C273792F7D83545D \
A48C2BEE680E841632CD4E44F07496B3EB3C1762 \
108F52B48DB57BB0CC439B2997B01419BD92F80A \
B9E2F5981AA6E0CD28160D9FF13993A75599653C \
; do \
gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$key" || \
gpg --batch --keyserver keyserver.ubuntu.com --recv-keys "$key" ; \
done \
&& curl -fsSLO --compressed "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION.tar.xz" \
&& curl -fsSLO --compressed "https://nodejs.org/dist/v$NODE_VERSION/SHASUMS256.txt.asc" \
&& gpg --batch --decrypt --output SHASUMS256.txt SHASUMS256.txt.asc \
&& grep " node-v$NODE_VERSION.tar.xz\$" SHASUMS256.txt | sha256sum -c - \
&& tar -xf "node-v$NODE_VERSION.tar.xz" \
&& cd "node-v$NODE_VERSION" \
&& ./configure \
&& make -j$(getconf _NPROCESSORS_ONLN) V= \
&& make install \
&& apk del .build-deps-full \
&& cd .. \
&& rm -Rf "node-v$NODE_VERSION" \
&& rm "node-v$NODE_VERSION.tar.xz" SHASUMS256.txt.asc SHASUMS256.txt; \
fi \
&& rm -f "node-v$NODE_VERSION-linux-$ARCH-musl.tar.xz" \
&& apk del .build-deps \
# smoke tests
&& node --version \
&& npm --version
#
# end of docker node stuff
#
# vips required to run sharp library for image comparison
RUN echo "http://dl-4.alpinelinux.org/alpine/v3.14/community" >> /etc/apk/repositories \
&& apk --no-cache add vips
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
WORKDIR /usr/app
ARG data_dir=/config
VOLUME $data_dir
ENV DATA_DIR=$data_dir
COPY package*.json ./
COPY tsconfig.json .
COPY docker/root/ /
# no prebuild support for node-canvas on alpine so need to compile
# https://github.com/Automattic/node-canvas#compiling
RUN npm install --build-from-source
WORKDIR /app
ADD . /usr/app
FROM base as build
RUN npm run build
COPY --chown=abc:abc package*.json ./
COPY --chown=abc:abc tsconfig.json .
RUN npm install
COPY --chown=abc:abc . /app
RUN npm run build && rm -rf node_modules
FROM base as app
COPY --from=build --chown=abc:abc /app /app
RUN npm install --production \
&& npm cache clean --force \
&& chown abc:abc node_modules \
&& rm -rf node_modules/ts-node \
&& rm -rf node_modules/typescript
ENV NPM_CONFIG_LOGLEVEL debug
ARG log_dir=/home/node/logs
RUN mkdir -p $log_dir
VOLUME $log_dir
ENV LOG_DIR=$log_dir
# can set database to use more performant better-sqlite3 since we control everything
ENV DB_DRIVER=better-sqlite3
# NODE_ARGS are expanded after `node` command in the entrypoint IE "node {NODE_ARGS} src/index.js run"
# by default enforce better memory mangement by limiting max long-lived GC space to 512MB
ENV NODE_ARGS="--max_old_space_size=512"
ARG webPort=8085
ENV PORT=$webPort
EXPOSE $PORT
CMD [ "node", "src/index.js", "run" ]
# convenience variable for more helpful error messages
ENV IS_DOCKER=true

View File

@@ -1,37 +1,52 @@
[![Latest Release](https://img.shields.io/github/v/release/foxxmd/context-mod)](https://github.com/FoxxMD/context-mod/releases)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Docker Pulls](https://img.shields.io/docker/pulls/foxxmd/context-mod)](https://hub.docker.com/r/foxxmd/context-mod)
# ContextMod [![Latest Release](https://img.shields.io/github/v/release/foxxmd/context-mod)](https://github.com/FoxxMD/context-mod/releases) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Docker Pulls](https://img.shields.io/docker/pulls/foxxmd/context-mod)](https://hub.docker.com/r/foxxmd/context-mod)
<img src="/docs/logo.png" align="right"
alt="ContextMod logo" width="180" height="176">
**Context Mod** (CM) is an event-based, [reddit](https://reddit.com) moderation bot built on top of [snoowrap](https://github.com/not-an-aardvark/snoowrap) and written in [typescript](https://www.typescriptlang.org/).
It is designed to help fill in the gaps for [automoderator](https://www.reddit.com/wiki/automoderator/full-documentation) in regard to more complex behavior with a focus on **user-history based moderation.**
An example of the above that Context Bot can do now:
An example of the above that Context Bot can do:
> * On a new submission, check if the user has also posted the same link in **N** number of other subreddits within a timeframe/# of posts
> * On a new submission or comment, check if the user has had any activity (sub/comment) in **N** set of subreddits within a timeframe/# of posts
>
>In either instance Context Bot can then perform any action a moderator can (comment, report, remove, lock, etc...) against that user, comment, or submission.
Some feature highlights:
* Simple rule-action behavior can be combined to create any level of complexity in behavior
* Server/client architecture
Feature Highlights for **Moderators:**
* Complete bot **autonomy**. YAML config is [stored in your subreddit's wiki](/docs/subreddit/gettingStarted.md#setup-wiki-page) (like automoderator)
* Simple rule-action behavior can be combined to create **complex behavior detection**
* Support Activity filtering based on:
* [Author criteria](/docs/subreddit/components/README.md#author-filter) (name, css flair/text, age, karma, moderator status, [Toolbox User Notes](https://www.reddit.com/r/toolbox/wiki/docs/usernotes), and more!)
* [Activity state](/docs/subreddit/components/README.md#item-filter) (removed, locked, distinguished, etc...)
* State of Subreddit Activity is in [Subreddit](/docs/subreddit/components/README.md#subreddit-filter) (nsfw, name, profile, etc...)
* Rules and Actions support [named references](/docs/subreddit/components/README.md#named-rules) -- **write once, reference anywhere**
* Powerful [logic control](/docs/subreddit/components/advancedConcepts/flowControl.md) (if, then, goto)
* [Delay/re-process activities](/docs/subreddit/components/README.md#dispatch) using arbitrary rules
* [**Image Comparisons**](/docs/imageComparison.md) via fingerprinting and/or pixel differences
* [**Repost detection**](/docs/subreddit/components/repost) with support for external services (youtube, etc...)
* Event notification via Discord
* [**Web interface**](#web-ui-and-screenshots) for monitoring, administration, and oauth bot authentication
* [**Placeholders**](/docs/subreddit/actionTemplating.md) (like automoderator) can be configured via a wiki page or raw text and supports [mustache](https://mustache.github.io) templating
* [**Partial Configurations**](/docs/subreddit/components/README.md#partial-configurations) -- offload parts of your configuration to shared locations to consolidate logic between multiple subreddits
* [Guest Access](/docs/subreddit/README.md#guest-access) enables collaboration and easier setup by allowing temporary access
* [Toxic content prediction](/docs/subreddit/components/README.md#moderatehatespeechcom-predictions) using [moderatehatespeech.com](https://moderatehatespeech.com) machine learning model
Feature highlights for **Developers and Hosting (Operators):**
* [Server/client architecture](/docs/serverClientArchitecture.md)
* Default/no configuration runs "All In One" behavior
* Additional configuration allows web interface to connect to multiple servers
* Each server instance can run multiple reddit accounts as bots
* **Per-subreddit configuration** is handled by JSON stored in the subreddit wiki
* Any text-based actions (comment, submission, message, usernotes, ban, etc...) can be configured via a wiki page or raw text in JSON and support [mustache](https://mustache.github.io) [templating](/docs/actionTemplating.md)
* History-based rules support multiple "valid window" types -- [ISO 8601 Durations](https://en.wikipedia.org/wiki/ISO_8601#Durations), [Day.js Durations](https://day.js.org/docs/en/durations/creating), and submission/comment count limits.
* Support Activity skipping based on:
* Author criteria (name, css flair/text, age, karma, moderator status, and [Toolbox User Notes](https://www.reddit.com/r/toolbox/wiki/docs/usernotes))
* Activity state (removed, locked, distinguished, etc.)
* Rules and Actions support named references (write once, reference anywhere)
* [**Image Comparisons**](/docs/imageComparison.md) via fingerprinting and/or pixel differences
* Global/subreddit-level **API caching**
* Support for [Toolbox User Notes](https://www.reddit.com/r/toolbox/wiki/docs/usernotes) as criteria or Actions (writing notes)
* Docker container support
* Event notification via Discord
* **Web interface** for monitoring, administration, and oauth bot authentication
* Global/subreddit-level [**caching**](/docs/operator/caching.md) of Reddit APIs responses and CM results
* [Database Persistence](/docs/operator/database.md) using SQLite, MySql, or Postgres
* Audit trails for bot activity
* Historical statistics
* [Docker container support](/docs/operator/installation.md#docker-recommended)
* Easy, UI-based [OAuth authentication](/docs/operator/addingBot.md) for adding Bots and moderator dashboard
* Integration with [InfluxDB](https://www.influxdata.com) for detailed [time-series metrics](/docs/operator/database.md#influx) and a pre-built [Grafana](https://grafana.com) [dashboard](/docs/operator/database.md#grafana)
# Table of Contents
@@ -44,7 +59,7 @@ Some feature highlights:
Each subreddit using the RCB bot configures its behavior via their own wiki page.
When a monitored **Event** (new comment/submission, new modqueue item, etc.) is detected the bot runs through a list of **Checks** to determine what to do with the **Activity** from that Event. Each **Check** consists of :
When a monitored **Activity** (new comment/submission, new modqueue item, etc.) is detected the bot runs through a list of [**Checks**](/docs/subreddit/components/README.md#checks) to determine what to do with the **Activity** from that Event. Each **Check** consists of :
#### Kind
@@ -52,11 +67,11 @@ Is this check for a submission or comment?
#### Rules
A list of **Rule** objects to run against the **Activity**. Triggered Rules can cause the whole Check to trigger and run its **Actions**
A list of [**Rules**](/docs/subreddit/components/README.md#rules) to run against the **Activity**. Triggered Rules can cause the whole Check to trigger and run its **Actions**
#### Actions
A list of **Action** objects that describe what the bot should do with the **Activity** or **Author** of the activity (comment, remove, approve, etc.). The bot will run **all** Actions in this list.
A list of [**Actions**](/docs/subreddit/components/README.md#actions) that describe what the bot should do with the **Activity** or **Author** of the activity (comment, remove, approve, etc.). The bot will run **all** Actions in this list.
___
@@ -66,7 +81,7 @@ When an Event occurs all Checks of that type are run in the order they were list
___
[Learn more about the RCB lifecycle and core concepts in the docs.](/docs#how-it-works)
[Learn more about the RCB lifecycle and core concepts in the docs.](/docs/README.md#how-it-works)
## Getting Started
@@ -74,20 +89,20 @@ ___
This guide is for users who want to **run their own bot on a ContextMod instance.**
See the [Operator's Getting Started Guide](/docs/gettingStartedOperator.md)
See the [Operator's Getting Started Guide](/docs/operator/gettingStarted.md)
#### Moderators
This guide is for **reddit moderators** who want to configure an existing CM bot to run on their subreddit.
See the [Moderator's Getting Started Guide](/docs/gettingStartedMod.md)
See the [Moderator's Getting Started Guide](/docs/subreddit/gettingStarted.md)
## Configuration and Documentation
Context Bot's configuration can be written in JSON, [JSON5](https://json5.org/) or YAML. Its schema conforms to [JSON Schema Draft 7](https://json-schema.org/). Additionally, many **operator** settings can be passed via command line or environmental variables.
Context Bot's configuration can be written in YAML (like automoderator) or [JSON5](https://json5.org/). Its schema conforms to [JSON Schema Draft 7](https://json-schema.org/). Additionally, many **operator** settings can be passed via command line or environmental variables.
* For **operators** (running the bot instance) see the [Operator Configuration](/docs/operatorConfiguration.md) guide
* For **moderators** consult the [app schema and examples folder](/docs/#configuration-and-usage)
* For **operators** (running the bot instance) see the [Operator Configuration](/docs/operator/configuration.md) guide
* For **moderators** consult the [app schema and examples folder](/docs/README.md#configuration-and-usage)
[**Check the full docs for in-depth explanations of all concepts and examples**](/docs)
@@ -107,32 +122,43 @@ CM comes equipped with a dashboard designed for use by both moderators and bot o
* View **real-time logs** of what the bot is doing on your subreddit
* **Run bot on any permalink**
![Subreddit View](docs/screenshots/subredditStatus.jpg)
![Subreddit View](docs/images/subredditStatus.jpg)
### Bot Setup/Authentication
A bot oauth helper allows operators to define oauth credentials/permissions and then generate unique, one-time invite links that allow moderators to authenticate their own bots without operator assistance. [Learn more about using the oauth helper.](docs/botAuthentication.md#cm-oauth-helper-recommended)
A bot oauth helper allows operators to define oauth credentials/permissions and then generate unique, one-time invite links that allow moderators to authenticate their own bots without operator assistance. [Learn more about using the oauth helper.](docs/operator/addingBot.md#cm-oauth-helper-recommended)
Operator view/invite link generation:
![Oauth View](docs/screenshots/oauth.jpg)
![Oauth View](docs/images/oauth.jpg)
Moderator view/invite and authorization:
![Invite View](docs/screenshots/oauth-invite.jpg)
![Invite View](docs/images/oauth-invite.jpg)
A similar helper and invitation experience is available for adding **subreddits to an existing bot.**
![Subreddit Invite View](docs/images/subredditInvite.jpg)
### Configuration Editor
A built-in editor using [monaco-editor](https://microsoft.github.io/monaco-editor/) makes editing configurations easy:
* Automatic JSON syntax validation and formatting
* Automatic JSON or YAML syntax validation and formatting
* Automatic Schema (subreddit or operator) validation
* All properties are annotated via hover popups
* Unauthenticated view via `yourdomain.com/config`
* Authenticated view loads subreddit configurations by simple link found on the subreddit dashboard
* Switch schemas to edit either subreddit or operator configurations
![Configuration View](docs/screenshots/editor.jpg)
![Configuration View](docs/images/editor.jpg)
### [Grafana Dashboard](/docs/operator/database.md#grafana)
* Overall stats (active bots/subreddits, api calls, per second/hour/minute activity ingest)
* Over time graphs for events, per subreddit, and for individual rules/check/actions
![Grafana Dashboard](/docs/images/grafana.jpg)
## License

3
act.env.example Normal file
View File

@@ -0,0 +1,3 @@
GITHUB_TOKEN=
DOCKERHUB_USERNAME=
DOCKER_PASSWORD=

View File

@@ -17,12 +17,22 @@
"REFRESH_TOKEN": {
"description": "Refresh token retrieved from authenticating an account with your Reddit Application",
"value": "",
"required": true
"required": false
},
"ACCESS_TOKEN": {
"description": "Access token retrieved from authenticating an account with your Reddit Application",
"value": "",
"required": true
"required": false
},
"REDIRECT_URI": {
"description": "Redirect URI you specified when creating your Reddit Application. Required if you want to use the web interface. In the provided example replace 'your-heroku-app-name' with the name of your HEROKU app.",
"value": "https://your-heroku-6app-name.herokuapp.com/callback",
"required": false
},
"OPERATOR": {
"description": "Your reddit username WITHOUT any prefixes EXAMPLE /u/FoxxMD => FoxxMD. Specified user will be recognized as an admin.",
"value": "",
"required": false
},
"WIKI_CONFIG": {
"description": "Relative url to contextbot wiki page EX https://reddit.com/r/subreddit/wiki/<path>",

View File

@@ -0,0 +1,14 @@
#!/usr/bin/with-contenv bash
# used https://github.com/linuxserver/docker-plex as a template
# make data folder if not /config
if [ ! -d "${DATA_DIR}" ]; then \
mkdir -p "${DATA_DIR}"
chown -R abc:abc /config
fi
# permissions
chown abc:abc \
/config \
/config/*

View File

@@ -0,0 +1,9 @@
#!/usr/bin/with-contenv bash
# used https://github.com/linuxserver/docker-plex as a template
# NODE_ARGS can be passed by ENV in docker command like "docker run foxxmd/context-mod -e NODE_ARGS=--optimize_for_size"
exec \
s6-setuidgid abc \
/usr/local/bin/node $NODE_ARGS /app/src/index.js run

View File

@@ -5,103 +5,119 @@
* [Getting Started](#getting-started)
* [How It Works](#how-it-works)
* [Concepts](#concepts)
* [Event](#event)
* [Activity](#activity)
* [Run](#runs)
* [Check](#checks)
* [Rule](#rule)
* [Examples](#available-rules)
* [Available Rules](#available-rules)
* [Rule Set](#rule-set)
* [Examples](#rule-set-examples)
* [Action](#action)
* [Examples](#available-actions)
* [Available Actions](#available-actions)
* [Filters](#filters)
* [Configuration and Usage](#configuration-and-usage)
* [Common Resources](#common-resources)
* [Activities `window`](#activities-window)
* [Comparisons](#thresholds-and-comparisons)
* [Activity Templating](/docs/actionTemplating.md)
* [Image Comparisons](#image-comparisons)
* [Best Practices](#best-practices)
* [Named Rules](#named-rules)
* [Rule Order](#rule-order)
* [Caching](#caching)
* FAQ
## Getting Started
Review **at least** the **How It Works** and **Concepts** below, then:
* For **Operators** (running a bot instance) refer to [**Operator Getting Started**](/docs/gettingStartedOperator.md) guide
* For **Moderators** (configuring an existing bot for your subreddit) refer to the [**Moderator Getting Started**](/docs/gettingStartedMod.md) guide
* For **Operators** (running a bot instance) refer to [**Operator Getting Started**](/docs/operator/gettingStarted.md) guide
* For **Moderators** (configuring an existing bot for your subreddit) refer to the [**Moderator Getting Started**](/docs/subreddit/gettingStarted.md) guide
## How It Works
Where possible Context Mod (CM) uses the same terminology as, and emulates the behavior, of **automoderator** so if you are familiar with that much of this may seem familiar to you.
### Diagram
Expand the section below for a simplified flow diagram of how CM processes an incoming Activity. Then refer the text description of the diagram below as well as [Concepts](#Concepts) for descriptions of individual components.
<details>
<summary>Diagram</summary>
![Flow Diagram](/docs/images/diagram-highlevel.jpg)
</details>
CM's lifecycle looks like this:
#### 1) A new event in your subreddit is received by CM
#### 1) A new Activity in your subreddit is received by CM
The events CM watches for are configured by you. These can be new modqueue/unmoderated items, submissions, or comments.
The Activities CM watches for are configured by you. These can be new modqueue/unmoderated items, submissions, or comments.
#### 2) CM sequentially processes each Check in your configuration
#### 2) CM sequentially processes each Run in your configuration
A [**Run**](#Runs) is made up of a set of [**Checks**](#Checks)
#### 3) CM sequentially processes each Check in the current Run
A **Check** is a set of:
* One or more **Rules** that define what conditions should **trigger** this Check
* One or more **Actions** that define what the bot should do once the Check is **triggered**
* One or more [**Rules**](#Rule) that define what conditions should **trigger** this Check
* One or more [**Actions**](#Action) that define what the bot should do once the Check is **triggered**
#### 3) Each Check is processed, *in order*, until a Check is triggered
#### 4) Each Check is processed, *in order*, until a Check is **triggered**
Once a Check is **triggered** no more Checks will be processed. This means all subsequent Checks in your configuration (in the order you listed them) are basically skipped.
In CM's default configuration, once a Check is **triggered** no more Checks will be processed. This means all subsequent Checks in this Run (in the order you listed them) are skipped.
#### 4) All Actions from that Check are executed
#### 5) All Actions from the triggered Check are executed
After all Actions are executed CM returns to waiting for the next Event.
After all **Actions** from the triggered **Check** are executed CM begins processing the next **Run**
#### 6) Rinse and Repeat from #3
Until all Runs have been processed.
## Concepts
Core, high-level concepts regarding how CM works.
### Event
An **Event** refers to the [Activity](#activity) (Comment or Submission) CM receives to process as well as the results of processing that Activity.
### Activity
An Activity is a Comment or Submission from Reddit.
### Runs
A **Run** is made up of a set of **Checks** that represent a group of related behaviors the bot should check for or perform -- that are independent of any other behaviors the Bot should perform.
An example of Runs:
* A group of Checks that look for missing flairs on a user or a new submission and flair accordingly
* A group of Checks that detect spam or self-promotion and then remove those activities
Both group of Checks are independent of each other (don't have any patterns or actions in common).
[Full Documentation for Runs](/docs/subreddit/components/README.md#runs)
### Checks
A **Check** is the main logical unit of behavior for the bot. It is equivalent to "if X then Y". A Check is comprised of:
A **Check** is the main logical unit of behavior for the bot. It is equivalent to "if X then Y". A Check is composed of:
* One or more **Rules** that are tested against an **Activity**
* One of more **Actions** that are performed when the **Rules** are satisfied
The bot's configuration can be made up of one or more **Checks** that are processed **in the order they are listed in the configuration.**
A Run can be made up of one or more **Checks** that are processed **in the order they are listed in the Run.**
Once a Check is **triggered** (its Rules are satisfied and Actions performed) all subsequent Checks are skipped.
Some other important concepts regarding Checks:
* All Checks have a **kind** (defined in the configuration) that determine if they should run on **Submissions** or **Comments**
* Checks have a **condition** property that determines when they are considered **triggered**
* If the **condition** is `AND` then ALL of their **Rules** must be **triggered** for the Check to be **triggered**
* If the **condition** is `OR` then if ANY **Rules** is triggered **triggered** then the Check is **triggered**
Examples of different types of Checks can be found in the [subreddit-ready examples.](/docs/examples/subredditReady)
[Full Documentation for Checks](/docs/subreddit/components/README.md#checks)
### Rule
A **Rule** is some set of **criteria** (conditions) that are tested against an Activity (comment/submission), a User, or a User's history. A Rule is considered **triggered** when the **criteria** for that rule are found to be **true** for whatever is being tested against.
There are generally three main properties for a Rule:
* **Critiera** -- The conditions/values you want to test for.
* **Activities Window** -- If applicable, the range of activities that the **criteria** will be tested against.
* **Rule-specific options** -- Any number of options that modify how the **criteria** are tested.
CM has different **Rules** that can test against different types of behavior and aspects of a User, their history, and the Activity (submission/common) being checked.
#### Available Rules
Find detailed descriptions of all the Rules, with examples, below:
[Full Documentation for Rules](/docs/subreddit/components/README.md#rules)
* [Attribution](/docs/examples/attribution)
* [Recent Activity](/docs/examples/recentActivity)
* [Repeat Activity](/docs/examples/repeatActivity)
* [History](/docs/examples/history)
* [Author](/docs/examples/author)
* [Regex](/docs/examples/regex)
#### Available Rules
All available rules can be found in the [components documentation](/docs/subreddit/components/README.md#rules)
### Rule Set
@@ -111,25 +127,7 @@ Rule Sets can be used interchangeably with other **Rules** and **Rule Sets** in
They allow you to create more complex trigger behavior by combining multiple rules into one "parent rule".
It consists of:
* **condition** -- Under what condition should the Rule Set be considered triggered?
* `AND` -- ALL Rules in the Rule Set must **trigger** in order for the Rule Set to **trigger.**
* `OR` -- ANY Rule in the Rule Set that is **triggered** will trigger the whole Rule Set.
* **rules** -- The **Rules** for the Rule Set.
Example
```json5
{
"condition": "AND",
"rules": [
// all the rules go here
]
}
```
#### Rule Set Examples
* [**Detailed Example**](/docs/examples/advancedConcepts/ruleSets.json5)
[Rule Sets Documentation](/docs/subreddit/components/README.md#rule-sets)
### Action
@@ -137,207 +135,28 @@ An **Action** is some action the bot can take against the checked Activity (comm
#### Available Actions
* Remove (Comment/Submission)
* Flair (Submission)
* Ban (User)
* Approve (Comment/Submission)
* Comment (Reply to Comment/Submission)
* Lock (Comment/Submission)
* Report (Comment/Submission)
* [UserNote](/docs/examples/userNotes) (User, when /r/Toolbox is used)
For detailed explanation and options of what individual Actions can do [see the links in the `actions` property in the schema.](https://json-schema.app/view/%23/%23%2Fdefinitions%2FSubmissionCheckJson?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json)
[Available Actions Documentation](/docs/subreddit/components/README.md#list-of-actions)
### Filters
**Checks, Rules, and Actions** all have two additional (optional) criteria "tests". These tests behave differently than rule/check triggers in that:
**Runs, Checks, Rules, and Actions** all have two additional (optional) criteria "pre-tests". These tests are different from rules/checks in these ways:
* When they **pass** the thing being tested continues to process as usual
* When they **fail** the thing being tested **is skipped, not failed.**
* Filters test against the **current state** of the Activity, the Author of the Activity, or the Subreddit of the Activity -- rather than looking at history/context/etc...
* Filter test results only determine if the Run, Check, Rule, or Action **should run** -- rather than triggering it
* When the filter test **passes** the thing being tested continues to process as usual
* When the filter test **fails** the thing being tested **fails**.
For **Checks** and **Actions** skipping means that the thing is not processed. The Action is not run, the Check is not triggered.
In the context of **Rules** (in a Check) skipping means the Rule does not get run BUT it does not fail. The Check will continue processing as if the Rule did not exist. However, if ALL Rules in a Check are skipped then the Check does "fail" (is not triggered).
#### Available Filters
##### Item Filter (`itemIs`)
This filter will test against the **state of the Activity currently being run.** Some criteria available to test against IE "Is the activity...":
* removed
* nsfw
* locked
* stickied
* deleted
* etc...
The `itemIs` filter is made up of an array (list) of `State` criteria objects. **All** criteria in the array must pass for this filter to pass.
There are two different State criteria depending on what type of Activity is being tested:
* Submission -- [SubmissionState](https://json-schema.app/view/%23/%23%2Fdefinitions%2FSubmissionCheckJson/%23%2Fdefinitions%2FSubmissionState?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json)
* Comment -- [CommentState](https://json-schema.app/view/%23/%23%2Fdefinitions%2FCommentCheckJson/%23%2Fdefinitions%2FCommentState?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json)
##### Author Filter (`authorIs`)
This filter will test against the **Author of the Activity currently being run.** Some criteria available to test against:
* account age
* comment, link, and total karma
* subreddit flair text/css
* name
* User Notes
* verified
* etc...
The `authorIs` filter is made up two (optional) lists of [`AuthorCriteria`](https://json-schema.app/view/%23/%23%2Fdefinitions%2FSubmissionCheckJson/%23%2Fdefinitions%2FAuthorOptions/%23%2Fdefinitions%2FAuthorCriteria?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) criteria objects that define how the test behaves:
* `include` list -- If **any** `AuthorCriteria` from this list passes then the `authorIs` test passes
* `exclude` list -- If **any** `AuthorCriteria` from this list **does not pass** then the `authorIs` test passes. **Note:** This property is ignored if `include` is also present IE you cannot use both properties at the same time.
Refer to the [app schema for `AuthorCriteria`](https://json-schema.app/view/%23/%23%2Fdefinitions%2FSubmissionCheckJson/%23%2Fdefinitions%2FAuthorOptions/%23%2Fdefinitions%2FAuthorCriteria?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) for all available properties to test against.
Some examples of using `authorIs` can be found in the [Author examples.](/docs/examples/author)
[Full Documentation for Filters](/docs/subreddit/components/README.md#filters)
## Configuration And Usage
* For **Operator/Bot maintainers** see **[Operation Configuration](/docs/operatorConfiguration.md)**
* [CLI Usage](docs/operatorConfiguration.md#cli-usage)
* For **Operator/Bot maintainers** see **[Operation Guide](/docs/operator/README.md)**
* For **Moderators**
* Refer to the [examples folder](/docs/examples) or the [subreddit-ready examples](/docs/examples/subredditReady)
* as well as the [schema editor](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) which has
* Start with the [Subreddit/Moderator docs](/docs/subreddit/README.md) or [Moderator Getting Started guide](/docs/subreddit/gettingStarted.md)
* Refer to the [Subreddit Components Documentation](/docs/subreddit/components) or the [subreddit-ready examples](/docs/subreddit/components/subredditReady)
* as well as the [schema](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) which has
* fully annotated configuration data/structure
* generated examples in json/yaml
* built-in editor that automatically validates your config
## Common Resources
Technical information on recurring, common data/patterns used in CM.
### Activities `window`
Most **Rules** must define the **range of Activities (submissions and/or comments)** that will be used to check the criteria of the Rule. This range is defined wherever you see a `window` property in configuration.
Refer to the [Activities Window](/docs/activitiesWindow.md) documentation for a technical explanation with examples.
### Thresholds and Comparisons
Most rules/filters have criteria that require you to define a specific condition to test against. This can be anything from repeats of activities to account age.
In all of these scenarios the condition is defined using a subset of [comparison operators](https://www.codecademy.com/articles/fwd-js-comparison-logical) (very similar to how automoderator does things).
Available operators:
* `<` -- **less than** => `5 < 6` => 5 is less than 6
* `>` -- **greater than** => `6 > 5` => 6 is greater than 5
* `<=` -- **less than or equal to** => `5 <= 5` => 5 is less than **or equal to** 5
* `>=` -- **greater than or equal to** => `5 >= 5` => 5 is greater than **or equal to** 5
In the context of a rule/filter comparison you provide the comparison **omitting** the value that is being tested. An example...
The RepeatActivity rule has a `threshold` comparison to test against the number of repeat activities it finds
* You want the rule to trigger if it finds **4 or more repeat activities**
* The rule would be configured like this `"threshold": ">= 4"`
Essentially what this is telling the rule is `threshold: "x >= 4"` where `x` is the largest repeat of activities it finds.
#### Other Comparison Types
Other than comparison numeric values there are two other values that can be compared (depending on the criteria)
##### Percentages
Some criteria accept an optional **percentage** to compare against:
```
"threshold": "> 20%"
```
Refer to the individual rule/criteria schema to see what this percentage is comparing against.
##### Durations
Some criteria accept an optional **duration** to compare against:
```
"threshold": "< 1 month"
```
The duration value compares a time range from **now** to `duration value` time in the past.
Refer to [duration values in activity window documentation](/docs/activitiesWindow.md#duration-values) as well as the individual rule/criteria schema to see what this duration is comparing against.
### Image Comparisons
ContextMod implements two methods for comparing **image content**, perceptual hashing and pixel-to-pixel comparisons. Comparisons can be used to filter activities in some activities.
See [image comparison documentation](/docs/imageComparison.md) for a full reference.
## Best Practices
### Named Rules
All **Rules** in a subreddit's configuration can be assigned a **name** that can then be referenced from any other Check.
Create general-use rules so they can be reused and de-clutter your configuration. Additionally, CM will automatically cache the result of a rule so there is a performance and api usage benefit to re-using Rules.
See [ruleNameReuse.json5](/docs/examples/advancedConcepts/ruleNameReuse.json5) for a detailed configuration with annotations.
### Check Order
Checks are run in the order they appear in your configuration, therefore you should place your highest requirement/severe action checks at the top and lowest requirement/moderate actions at the bottom.
This is so that if an Activity warrants a more serious reaction that Check is triggered first rather than having a lower requirement check with less severe actions triggered and causing all subsequent Checks to be skipped.
* Attribution >50% AND Repeat Activity 8x AND Recent Activity in 2 subs => remove submission + ban
* Attribution >20% AND Repeat Activity 4x AND Recent Activity in 5 subs => remove submission + flair user restricted
* Attribution >20% AND Repeat Activity 2x => remove submission
* Attribution >20% AND History comments <30% => remove submission
* Attribution >15% => report
* Repeat Activity 2x => report
* Recent Activity in 3 subs => report
* Author not vetted => flair new user submission
### Rule Order
The ordering of your Rules within a Check/RuleSet can have an impact on Check performance (speed) as well as API usage.
Consider these three rules:
* Rule A -- Recent Activity => 3 subreddits => last 15 submissions
* Rule B -- Repeat Activity => last 3 days
* Rule C -- Attribution => >10% => last 90 days or 300 submissions
The first two rules are lightweight in their requirements -- Rule A can be completed in 1 API call, Rule B potentially completed in 1 Api call.
However, depending on how active the Author is, Rule C will take *at least* 3 API calls just to get all activities (Reddit limit 100 items per call).
If the Check is using `AND` condition for its rules (default) then if either Rule A or Rule B fail then Rule C will never run. This means 3 API calls never made plus the time waiting for each to return.
**It is therefore advantageous to list your lightweight Rules first in each Check.**
### Caching
ContextMod implements caching functionality for:
* author history (`window` criteria in rules)
* `authorIs` results
* `content` that uses wiki pages (on Comment/Report/Ban Actions)
* and User Notes
All of these use api requests so caching them reduces api usage.
Cached results can be re-used if the criteria in configuration is identical to a previously cached result. So...
* author history cache results are re-used if **`window` criteria on a Rule is identical to the `window` on another Rule** IE always use **7 Days** or always use **50 Items** for absolute counts.
* `authorIs` criteria is identical to another `authorIs` elsewhere in configuration..
* etc...
Re-use will result in less API calls and faster Check times.
PROTIP: You can monitor the re-use of cache in the `Cache` section of your subreddit on the web interface. See the tooltips in that section for a better breakdown of cache statistics.
## FAQ

View File

@@ -1,72 +0,0 @@
Actions that can submit text (Report, Comment) will have their `content` values run through a [Mustache Template](https://mustache.github.io/). This means you can insert data generated by Rules into your text before the Action is performed.
See here for a [cheatsheet](https://gist.github.com/FoxxMD/d365707cf99fdb526a504b8b833a5b78) and [here](https://www.tsmean.com/articles/mustache/the-ultimate-mustache-tutorial/) for a more thorough tutorial.
All Actions with `content` have access to this data:
```json5
{
item: {
kind: 'string', // the type of item (comment/submission)
author: 'string', // name of the item author (reddit user)
permalink: 'string', // a url to the item
url: 'string', // if the item is a Submission then its URL (external for link type submission, reddit link for self-posts)
title: 'string', // if the item is a Submission, then the title of the Submission,
botLink: 'string' // a link to the bot's FAQ
},
rules: {
// contains all rules that were run and are accessible using the name, lowercased, with all spaces/dashes/underscores removed
}
}
```
The properties of `rules` are accessible using the name, lower-cased, with all spaces/dashes/underscores. If no name is given `kind` is used as `name` Example:
```
"rules": [
{
"name": "My Custom-Recent Activity Rule", // mycustomrecentactivityrule
"kind": "recentActivity"
},
{
// name = repeatsubmission
"kind": "repeatActivity",
}
]
```
**To see what data is available for individual Rules [consult the schema](#configuration) for each Rule.**
#### Quick Templating Tutorial
As a quick example for how you will most likely be using templating -- wrapping a variable in curly brackets, `{{variable}}`, will cause the variable value to be rendered instead of the brackets:
```
myVariable = 50;
myOtherVariable = "a text fragment"
template = "This is my template, the variable is {{myVariable}}, my other variable is {{myOtherVariable}}, and that's it!";
console.log(Mustache.render(template, {myVariable});
// will render...
"This is my template, the variable is 50, my other variable is a text fragment, and that's it!";
```
**Note: When accessing an object or its properties you must use dot notation**
```
const item = {
aProperty: 'something',
anotherObject: {
bProperty: 'something else'
}
}
const content = "My content will render the property {{item.aProperty}} like this, and another nested property {{item.anotherObject.bProperty}} like this."
```

View File

@@ -1,217 +0,0 @@
# Activity Window
Most **Rules** have a `window` property somewhere within their configuration. This property defines the range of **Activities** (submission and/or comments) that should be retrieved for checking the criteria of the Rule.
As an example if you want to run an **Recent Activity Rule** to check if a user has had activity in /r/mealtimevideos you also need to define what range of activities you want to look at from that user's history.
## `window` property overview (tldr)
The value of `window` can be any of these types:
* `number` count of activities
* `string` [duration](#duration-string-recommended) or [iso 8601](#an-iso-8601-duration-string)
* [duration `object`](#duration-object)
* [ActivityWindowCriteria `object`](#activitywindowcriteria)
Examples of all of the above
<details>
```
// count, last 100 activities
{
"window": 100
}
// duration string, last 10 days
{
"window": "10 days"
}
// duration object, last 2 months and 5 days
{
"window": {
"months": 2,
"days": 5,
}
}
// iso 8601 string, last 15 minutes
{
"window": "PT15M"
}
// ActivityWindowCriteria, last 100 activities or 6 weeks of activities (whichever is found first)
{
"window": {
"count": 100,
"duration": "6 weeks"
}
}
```
</details>
## Types of Ranges
There are two types of values that can be used when defining a range:
### Count
This is the **number** of activities you want to retrieve. It's straightforward -- if you want to look at the last 100 activities for a user you can use `100` as the value.
### Duration
A **duration of time** between which all activities will be retrieved. This is a **relative value** that calculates the actual range based on **the duration of time subtracted from when the rule is run.**
For example:
* Today is **July 15th**
* You define a duration of **10 days**
Then the range of activities to be retrieved will be between **July 5th and July 15th** (10 days).
#### Duration Values
The value used to define the duration can be **any of these three types**:
##### Duration String (recommended)
A string consisting of
* A [Dayjs unit of time](https://day.js.org/docs/en/durations/creating#list-of-all-available-units)
* The value of that unit of time
Examples:
* `9 days`
* `14 hours`
* `80 seconds`
You can ensure your string is valid by testing it [here.](https://regexr.com/61em3)
##### Duration Object
If you need to specify multiple units of time for your duration you can instead provide a [Dayjs duration **object**](https://day.js.org/docs/en/durations/creating#list-of-all-available-units) consisting of Dayjs unit-values.
Example
```json
{
"days": 4,
"hours": 6,
"minutes": 20
}
```
##### An ISO 8601 duration string
If you're a real nerd you can also use a [standard duration](https://en.wikipedia.org/wiki/ISO_8601#Durations)) string.
Examples
* `PT15M` (15 minutes)
Ensure your string is valid by testing it [here.](https://regexr.com/61em9)
## ActivityWindowCriteria
This is an object that lets you specify more granular conditions for your range.
The full object looks like this:
```json
{
"count": 100,
"duration": "10 days",
"satisfyOn": "any",
"subreddits": {
"include": ["mealtimevideos","pooptimevideos"],
"exclude": ["videos"]
}
}
```
### Specifying Range
You may use **one or both range properties.**
If both range properties are specified then the value `satisfyOn` determines how the final range is determined
#### Using `"satisfyOn": "any"` (default)
If **any** then Activities will be retrieved until one of the range properties is met, **whichever occurs first.**
Example
```json
{
"count": 80,
"duration": "90 days",
"satisfyOn": "any"
}
```
Activities are retrieved in chunks of 100 (or `count`, whichever is smaller)
* If 90 days of activities returns only 40 activities => returns 40 activities
* If 80 activities is only 20 days of range => 80 activities
#### Using `"satisfyOn": "all"`
If **all** then both ranges must be satisfied. Effectively, whichever range produces the most Activities will be the one that is used.
Example
```json
{
"count": 100,
"duration": "90 days",
"satisfyOn": "all"
}
```
Activities are retrieved in chunks of 100 (or `count`, whichever is smaller)
* If at 90 days of activities => 40 activities retrieved
* continue retrieving results until 100 activities
* so range is >90 days of activities
* If at 100 activities => 20 days of activities retrieved
* continue retrieving results until 90 days of range
* so results in >100 activities
### Filtering Activities
You may filter retrieved Activities using an array of subreddits.
**Note:** Activities are filtered **before** range check is made so you will always end up with specified range (but may require more api calls if many activities are filtered out)
#### Include
Use **include** to specify which subreddits should be included from results
Example where only activities from /r/mealtimevideos and /r/modsupport will be returned
```json
{
"count": 100,
"duration": "90 days",
"satisfyOn": "any",
"subreddits": {
"include": ["mealtimevideos","modsupport"]
}
}
```
#### Exclude
Use **exclude** to specify which subreddits should NOT be in the results
Example where activities from /r/mealtimevideos and /r/modsupport will not be returned in results
```json
{
"count": 100,
"duration": "90 days",
"satisfyOn": "any",
"subreddits": {
"exclude": ["mealtimevideos","modsupport"]
}
}
```
**Note:** `exclude` will be ignored if `include` is also present.

View File

@@ -1,109 +0,0 @@
**Note:** This is for **bot operators.** If you are a subreddit moderator check out the **[Getting Started Guide](/docs/gettingStartedMod.md)**
Before you can start using your bot on reddit there are a few steps you must take:
* Create your bot account IE the reddit account that will be the "bot"
* Create a Reddit application
* Authenticate your bot account with the application
At the end of this process you will have this info:
* clientId
* clientSecret
* refreshToken
* accessToken
* redirectUri
**Note:** If you already have this information you can skip this guide **but make sure your redirect uri is correct if you plan on using the web interface.**
# Table Of Contents
* [Creating an Application](#create-application)
* [Authenticate Your Bot](#authenticate-your-bot-account)
* [Using CM OAuth Helper](#cm-oauth-helper-recommended)
* [Using Aardvark OAuth Helper](#aardvark-oauth-helper)
* [Provide Credentials to CM](#provide-credentials-to-cm)
# Create Application
Visit [your reddit preferences](https://www.reddit.com/prefs/apps) and at the bottom of the page go through the **create an(other) app** process.
* Give it a **name**
* Choose **web app**
* If you know what you will use for **redirect uri** go ahead and use it, otherwise use **http://localhost:8085/callback**
Click **create app**.
Then write down your **Client ID, Client Secret, and Redirect Uri** somewhere (or keep this webpage open)
# Authenticate Your Bot Account
There are **two ways** you can authenticate your bot account. It is recommended to use the CM oauth helper.
## CM OAuth Helper (Recommended)
This method will use CM's built in oauth flow. It is recommended because it will ensure your bot is authenticated with the correct oauth permissions.
### Start CM with Client ID/Secret and Operator
Start the application and provide these to your configuration:
* **Client ID**
* **Client Secret**
* **Redirect URI**
* **Operator**
It is important you define **Operator** because the auth route is **protected.** You must login to the application in order to access the route.
Refer to the [operator config guide](/docs/operatorConfiguration.md) if you need help with this.
Examples:
* CLI - `node src/index.js --clientId=myId --clientSecret=mySecret --redirectUri="http://localhost:8085/callback" --operator=FoxxMD`
* Docker - `docker run -e "CLIENT_ID=myId" -e "CLIENT_SECRET=mySecret" -e "OPERATOR=FoxxMD" -e "REDIRECT_URI=http://localhost:8085/callback" foxxmd/context-mod`
### Create An Auth Invite
Then open the CM web interface (default is [http://localhost:8085](http://localhost:8085)) and login.
After logging in you should be automatically redirected the auth page. If you are not then visit [http://localhost:8085/auth/helper](http://localhost:8085/auth/helper))
Follow the directions in the helper to create an **auth invite link.** Open this link and then follow the directions to authenticate your bot. At the end of the process you will receive an **Access Token** and **Refresh Token**
## Aardvark OAuth Helper
This method should only be used if you cannot use the [CM OAuth Helper method](#cm-oauth-helper-recommended) because you cannot access the CM web interface.
* Visit [https://not-an-aardvark.github.io/reddit-oauth-helper/](https://not-an-aardvark.github.io/reddit-oauth-helper/) and follow the instructions given.
* **Note:** You will need to update your **redirect uri.**
* Input your **Client ID** and **Client Secret** in the text boxes with those names.
* Choose scopes. **It is very important you check everything on this list or CM may not work correctly**
* edit
* flair
* history
* identity
* modcontributors
* modflair
* modposts
* modself
* mysubreddits
* read
* report
* submit
* wikiread
* wikiedit (if you are using Toolbox User Notes)
* Click **Generate tokens**, you will get a popup asking you to approve access (or login) -- **the account you approve access with is the account that Bot will control.**
* After approving an **Access Token** and **Refresh Token** will be shown at the bottom of the page. Save these to use with CM.
# Provide Credentials to CM
At the end of the last step you chose you should now have this information saved somewhere:
* clientId
* clientSecret
* refreshToken
* accessToken
* redirectUri
This is all the information you need to run your bot with CM.
Using these credentials follow the [operator config guide](/docs/operatorConfiguration.md) to finish setting up your CM instance.

392
docs/development.md Normal file
View File

@@ -0,0 +1,392 @@
TODO add more development sections...
# Developing/Testing Github Actions
Use [act](https://github.com/nektos/act) to run Github actions locally.
An example secrets file can be found in the project working directory at [act.env.example](act.env.example)
Modify [push-hook-sample.json](.github/push-hook-sample.json) to point to the local branch you want to run a `push` event trigger on, then run this command from the project working directory:
```bash
act -e .github/push-hook-sample.json --secret-file act.env
```
# Mocking Reddit API
Using [MockServer](https://www.mock-server.com/)
## Installation
https://www.mock-server.com/mock_server/running_mock_server.html
Easiest way is to install the [docker container](https://www.mock-server.com/mock_server/running_mock_server.html#pull_docker_image) ([from here](https://hub.docker.com/r/mockserver/mockserver))
Map port `1080:1080` -- acts as both the proxy port and the UI endpoint with the below URL:
```
http(s)://localhost:1080/mockserver/dashboard
```
In your [operator configuration](/docs/operator/operatorConfiguration.md) define a proxy for snoowrap at the top-level:
```yaml
snoowrap:
proxy: 'http://localhost:8010'
#debug: true # optionally set debug to true to make snoowrap requests output to log
```
## Usage
### Forwarding Requests (Monitoring Behavior)
This is what will make MockServer act as an actual **proxy server**. In this state CM will operate normally. In the MockServer UI you will be able to monitor all requests/responses made.
```HTTP
PUT /mockserver/expectation HTTP/1.1
Host: localhost:8010
Content-Type: application/json
Content-Length: 155
```
<details>
<summary>CURL</summary>
```bash
curl --location --request PUT 'http://localhost:8010/mockserver/expectation' \
--header 'Content-Type: application/json' \
--data-raw '{
"httpRequest": {},
"priority": 0,
"httpForward": {
"host": "oauth.reddit.com",
"port": 443,
"scheme": "HTTPS"
}
}'
```
</details>
### Mocking Network Issues
MockServer is a bit confusing and regex'ing for specific paths don't work well (for me??)
The lifecycle of a mock call I do:
* Make sure [forwarding](#forwarding-requests-monitoring-behavior) is set, to begin with
* Breakpoint before the code you want to test with mocking
* [Mock the network issue](#create-network-issue-behavior)
* Once the mock behavior should be "done" then
* [Clear all exceptions](#clearing-behavior)
* Set [forwarding behavior](#forwarding-requests-monitoring-behavior) again
### Create Network Issue Behavior
#### All Responses return 403
<details>
<summary>HTTP</summary>
```HTTP
PUT /mockserver/expectation HTTP/1.1
Host: localhost:8010
Content-Type: application/json
Content-Length: 1757
```
</details>
<details>
<summary>CURL</summary>
```bash
curl --location --request PUT 'http://localhost:8010/mockserver/expectation' \
--header 'Content-Type: application/json' \
--data-raw '{
"id": "error",
"httpRequest": {
"path": ".*"
},
"priority": 1,
"httpResponse": {
"statusCode": 403,
"reasonPhrase": "Forbidden",
"headers": {
"Connection": [
"keep-alive"
],
"Content-Type": [
"application/json; charset=UTF-8"
],
"x-ua-compatible": [
"IE=edge"
],
"x-frame-options": [
"SAMEORIGIN"
],
"x-content-type-options": [
"nosniff"
],
"x-xss-protection": [
"1; mode=block"
],
"expires": [
"-1"
],
"cache-control": [
"private, s-maxage=0, max-age=0, must-revalidate, no-store, max-age=0, must-revalidate"
],
"x-ratelimit-remaining": [
"575.0"
],
"x-ratelimit-used": [
"25"
],
"x-ratelimit-reset": [
"143"
],
"X-Moose": [
"majestic"
],
"Accept-Ranges": [
"bytes"
],
"Date": [
"Wed, 05 Jan 2022 14:37:37 GMT"
],
"Via": [
"1.1 varnish"
],
"Vary": [
"accept-encoding"
],
"Strict-Transport-Security": [
"max-age=15552000; includeSubDomains; preload"
],
"Server": [
"snooserv"
],
"X-Clacks-Overhead": [
"GNU Terry Pratchett"
]
}
}
}'
```
</details>
#### All Responses Timeout
<details>
<summary>HTTP</summary>
```HTTP
PUT /mockserver/expectation HTTP/1.1
Host: localhost:8010
Content-Type: application/json
Content-Length: 251
```
</details>
<details>
<summary>CURL</summary>
```bash
curl --location --request PUT 'http://localhost:8010/mockserver/expectation' \
--header 'Content-Type: application/json' \
--data-raw '{
"id": "error",
"httpRequest": {
"path": ".*"
},
"priority": 1,
"httpResponse": {
"body": "should never receive this",
"delay": {
"timeUnit": "SECONDS",
"value": 60
}
}
}'
```
</details>
#### All Responses Drop After Delay (Connection Closed by Server)
<details>
<summary>HTTP</summary>
```HTTP
PUT /mockserver/expectation HTTP/1.1
Host: localhost:8010
Content-Type: application/json
Content-Length: 234
```
</details>
<details>
<summary>CURL</summary>
```bash
curl --location --request PUT 'http://localhost:8010/mockserver/expectation' \
--header 'Content-Type: application/json' \
--data-raw '{
"id": "error",
"httpRequest": {
"path": ".*"
},
"priority": 1,
"httpError": {
"dropConnection": true,
"delay": {
"timeUnit": "SECONDS",
"value": 2
}
}
}'
```
</details>
### Clearing Behavior
```HTTP
PUT /mockserver/clear?type=EXPECTATIONS HTTP/1.1
Host: localhost:8010
Content-Type: application/json
Content-Length: 26
```
<details>
<summary>CURL</summary>
```bash
curl --location --request PUT 'http://localhost:8010/mockserver/clear?type=EXPECTATIONS' \
--header 'Content-Type: application/json' \
--data-raw '{
"path": "/.*"
}'
```
</details>
}
}
}
```
</details>
<details>
<summary>CURL</summary>
```bash
curl --location --request PUT 'http://localhost:8010/mockserver/expectation' \
--header 'Content-Type: application/json' \
--data-raw '{
"id": "error",
"httpRequest": {
"path": ".*"
},
"priority": 1,
"httpResponse": {
"body": "should never receive this",
"delay": {
"timeUnit": "SECONDS",
"value": 60
}
}
}'
```
</details>
#### All Responses Drop After Delay (Connection Closed by Server)
<details>
<summary>HTTP</summary>
```HTTP
PUT /mockserver/expectation HTTP/1.1
Host: localhost:8010
Content-Type: application/json
Content-Length: 234
{
"id": "error",
"httpRequest": {
"path": ".*"
},
"priority": 1,
"httpError": {
"dropConnection": true,
"delay": {
"timeUnit": "SECONDS",
"value": 2
}
}
}
```
</details>
<details>
<summary>CURL</summary>
```bash
curl --location --request PUT 'http://localhost:8010/mockserver/expectation' \
--header 'Content-Type: application/json' \
--data-raw '{
"id": "error",
"httpRequest": {
"path": ".*"
},
"priority": 1,
"httpError": {
"dropConnection": true,
"delay": {
"timeUnit": "SECONDS",
"value": 2
}
}
}'
```
</details>
### Clearing Behavior
```HTTP
PUT /mockserver/clear?type=EXPECTATIONS HTTP/1.1
Host: localhost:8010
Content-Type: application/json
Content-Length: 26
{
"path": "/user/.*"
}
```
<details>
<summary>CURL</summary>
```bash
curl --location --request PUT 'http://localhost:8010/mockserver/clear?type=EXPECTATIONS' \
--header 'Content-Type: application/json' \
--data-raw '{
"path": "/.*"
}'
```
</details>

View File

@@ -1,27 +0,0 @@
# Examples
This directory contains example of valid, ready-to-go configurations for Context Mod for the purpose of:
* showcasing what the bot can do
* providing best practices for writing your configuration
* providing generally useful configurations **that can be used immediately** or as a jumping-off point for your configuration
### Examples Overview
* Rules
* [Attribution](/docs/examples/attribution)
* [Recent Activity](/docs/examples/recentActivity)
* [Repeat Activity](/docs/examples/repeatActivity)
* [History](/docs/examples/history)
* [Author](/docs/examples/author)
* [Regex](/docs/examples/regex)
* [Toolbox User Notes](/docs/examples/userNotes)
* [Advanced Concepts](/docs/examples/advancedConcepts)
* [Rule Sets](/docs/examples/advancedConcepts/ruleSets.json5)
* [Name Rules](/docs/examples/advancedConcepts/ruleNameReuse.json5)
* [Check Ordering](/docs/examples/advancedConcepts)
* [Subreddit-ready examples](/docs/examples/subredditReady)
PROTIP: You can edit/build on examples by using the [schema editor.](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json)

View File

@@ -1,56 +0,0 @@
### Named Rules
See [ruleNameReuse.json5](/docs/examples/advancedConcepts/ruleNameReuse.json5)
### Check Order
Checks are run in the order they appear in your configuration, therefore you should place your highest requirement/severe action checks at the top and lowest requirement/moderate actions at the bottom.
This is so that if an Activity warrants a more serious reaction that Check is triggered first rather than having a lower requirement check with less severe actions triggered and causing all subsequent Checks to be skipped.
* Attribution >50% AND Repeat Activity 8x AND Recent Activity in 2 subs => remove submission + ban
* Attribution >20% AND Repeat Activity 4x AND Recent Activity in 5 subs => remove submission + flair user restricted
* Attribution >20% AND Repeat Activity 2x => remove submission
* Attribution >20% AND History comments <30% => remove submission
* Attribution >15% => report
* Repeat Activity 2x => report
* Recent Activity in 3 subs => report
* Author not vetted => flair new user submission
### Rule Sets
The `rules` array on a `Checks` can contain both `Rule` objects and `RuleSet` objects.
A **Rule Set** is a "nested" set of `Rule` objects with a passing condition specified. These allow you to create more complex trigger behavior by combining multiple rules.
See **[ruleSets.json5](/docs/examples/advancedConcepts/ruleSets.json5)** for a complete example as well as consulting the [schema](https://json-schema.app/view/%23%2Fdefinitions%2FRuleSetJson?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json).
### Rule Order
The ordering of your Rules within a Check/RuleSet can have an impact on Check performance (speed) as well as API usage.
Consider these three rules:
* Rule A -- Recent Activity => 3 subreddits => last 15 submissions
* Rule B -- Repeat Activity => last 3 days
* Rule C -- Attribution => >10% => last 90 days or 300 submissions
The first two rules are lightweight in their requirements -- Rule A can be completed in 1 API call, Rule B potentially completed in 1 Api call.
However, depending on how active the Author is, Rule C will take *at least* 3 API calls just to get all activities (Reddit limit 100 items per call).
If the Check is using `AND` condition for its rules (default) then if either Rule A or Rule B fail then Rule C will never run. This means 3 API calls never made plus the time waiting for each to return.
**It is therefore advantageous to list your lightweight Rules first in each Check.**
### API Caching
Context Mod implements some basic caching functionality for **Author Activities** and wiki pages (on Comment/Report Actions).
**Author Activities** are cached for a subreddit-configurable amount of time (10 seconds by default). A cached activities set can be re-used if the **window on a Rule is identical to the window on another Rule**.
This means that when possible you should re-use window values.
IE If you want to check an Author's Activities for a time range try to always use **7 Days** or always use **50 Items** for absolute counts.
Re-use will result in less API calls and faster Check times.

View File

@@ -1,75 +0,0 @@
{
"checks": [
{
"name": "Auto Remove SP Karma",
"description": "Remove submission because author has self-promo >10% and posted in karma subs recently",
"kind": "submission",
"rules": [
// named rules can be referenced at any point in the configuration (where they occur does not matter)
// and can be used in any Check
// Note: rules do not transfer between subreddit configurations
"freekarmasub",
{
"name": "attr10all",
"kind": "attribution",
"criteria": [
{
"threshold": "> 10%",
"window": "90 days"
},
{
"threshold": "> 10%",
"window": 100
}
],
}
],
"actions": [
{
"kind": "remove"
},
{
"kind": "comment",
"content": "Your submission was removed because you are over reddit's threshold for self-promotion and recently posted this content in a karma sub"
}
]
},
{
"name": "Free Karma On Submission Alert",
"description": "Check if author has posted this submission in 'freekarma' subreddits",
"kind": "submission",
"rules": [
{
// rules can be re-used throughout a configuration by referencing them by name
//
// The rule name itself can only contain spaces, hyphens and underscores
// The value used to reference it will have all of these removed, and lower-cased
//
// so to reference this rule use the value 'freekarmasub'
"name": "Free_Karma-SUB",
"kind": "recentActivity",
"lookAt": "submissions",
"useSubmissionAsReference":true,
"thresholds": [
{
"threshold": ">= 1",
"subreddits": [
"DeFreeKarma",
"FreeKarma4U",
"FreeKarma4You",
"upvote"
]
}
],
"window": "7 days"
}
],
"actions": [
{
"kind": "report",
"content": "Submission posted {{rules.freekarmasub.totalCount}} times in karma {{rules.freekarmasub.subCount}} subs over {{rules.freekarmasub.window}}: {{rules.freekarmasub.subSummary}}"
}
]
},
]
}

View File

@@ -1,84 +0,0 @@
{
"checks": [
{
"name": "Self Promo All or low comment",
"description": "SP >10% of all activities or >10% of submissions with low comment engagement",
"kind": "submission",
"rules": [
{
// this attribution rule is looking at all activities
//
// we want want this one rule to trigger the check because >10% of all activity (submission AND comments) is a good requirement
"name": "attr10all",
"kind": "attribution",
"criteria": [
{
"threshold": "> 10%",
"window": "90 days"
},
{
"threshold": "> 10%",
"window": 100
}
],
},
{
// this is a **Rule Set**
//
// it is made up of "nested" rules with a pass condition (AND/OR)
// if the nested rules pass the condition then the Rule Set triggers the Check
//
// AND = all nested rules must be triggered to make the Rule Set trigger
// AND = any of the nested Rules will be the Rule Set trigger
"condition": "AND",
// in this check we use an Attribution >10% on ONLY submissions, which is a lower requirement then the above attribution rule
// and combine it with a History rule looking for low comment engagement
// to make a "higher" requirement Rule Set our of two low requirement Rules
"rules": [
{
"name": "attr20sub",
"kind": "attribution",
"criteria": [
{
"threshold": "> 10%",
"thresholdOn": "submissions",
"window": "90 days"
},
{
"threshold": "> 10%",
"thresholdOn": "submissions",
"window": 100
}
],
"lookAt": "media"
},
{
"name": "lowOrOpComm",
"kind": "history",
"criteriaJoin": "OR",
"criteria": [
{
"window": "90 days",
"comment": "< 50%"
},
{
"window": "90 days",
"comment": "> 40% OP"
}
]
}
]
}
],
"actions": [
{
"kind": "remove"
},
{
"kind": "comment",
"content": "Your submission was removed because you are over reddit's threshold for self-promotion or exhibit low comment engagement"
}
]
},
],
}

View File

@@ -1,39 +0,0 @@
{
"checks": [
{
"name": "Self Promo Activities",
"description": "Check if any of Author's aggregated submission origins are >10% of entire history",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "attr10all",
"kind": "attribution",
// criteria defaults to OR -- so either of these criteria will trigger the rule
"criteria": [
{
// threshold can be a percent or an absolute number
"threshold": "> 10%",
// The default is "all" -- calculate percentage of entire history (submissions & comments)
// "thresholdOn": "all",
// look at last 90 days of Author's activities (comments and submissions)
"window": "90 days"
},
{
"threshold": "> 10%",
// look at Author's last 100 activities (comments and submissions)
"window": 100
}
],
}
],
"actions": [
{
"kind": "report",
"content": "{{rules.attr10all.largestPercent}}% of {{rules.attr10all.activityTotal}} items over {{rules.attr10all.window}}"
}
]
}
]
}

View File

@@ -1,40 +0,0 @@
{
"checks": [
{
"name": "Self Promo Submissions",
"description": "Check if any of Author's aggregated submission origins are >10% of their submissions",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "attr10sub",
"kind": "attribution",
// criteria defaults to OR -- so either of these criteria will trigger the rule
"criteria": [
{
// threshold can be a percent or an absolute number
"threshold": "> 10%",
// calculate percentage of submissions, rather than entire history (submissions & comments)
"thresholdOn": "submissions",
// look at last 90 days of Author's activities (comments and submissions)
"window": "90 days"
},
{
"threshold": "> 10%",
"thresholdOn": "submissions",
// look at Author's last 100 activities (comments and submissions)
"window": 100
}
],
}
],
"actions": [
{
"kind": "report",
"content": "{{rules.attr10sub.largestPercent}}% of {{rules.attr10sub.activityTotal}} items over {{rules.attr10sub.window}}"
}
]
}
]
}

View File

@@ -1,38 +0,0 @@
# Author
## Rule
The **Author** rule triggers if any [AuthorCriteria](https://json-schema.app/view/%23%2Fdefinitions%2FAuthorCriteria?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) from a list are either **included** or **excluded**, depending on which property you put them in.
**AuthorCriteria** that can be checked:
* name (u/userName)
* author's subreddit flair text
* author's subreddit flair css
* author's subreddit mod status
* [Toolbox User Notes](/docs/examples/userNotes)
The Author **Rule** is best used in conjunction with other Rules to short-circuit a Check based on who the Author is. It is easier to use a Rule to do this then to write **author filters** for every Rule (and makes Rules more re-useable).
Consult the [schema](https://json-schema.app/view/%23%2Fdefinitions%2FAuthorRuleJSONConfig?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) for a complete reference of the rule's properties.
### Examples
* Basic examples
* [Flair new user Submission](/docs/examples/author/flairNewUserSubmission.json5) - If the Author does not have the `vet` flair then flair the Submission with `New User`
* [Flair vetted user Submission](/docs/examples/author/flairNewUserSubmission.json5) - If the Author does have the `vet` flair then flair the Submission with `Vetted`
* Used with other Rules
* [Ignore vetted user](/docs/examples/author/flairNewUserSubmission.json5) - Short-circuit the Check if the Author has the `vet` flair
## Filter
All **Rules** and **Checks** have an optional `authorIs` property that takes an [AuthorOptions](https://json-schema.app/view/%23%2Fdefinitions%2FAuthorOptions?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) object.
**This property works the same as the Author Rule except that:**
* On **Rules** if all criteria fail the Rule is **skipped.**
* If a Rule is skipped **it does not fail or pass** and so does not affect the outcome of the Check.
* However, if all Rules on a Check are skipped the Check will fail.
* On **Checks** if all criteria fail the Check **fails**.
### Examples
* [Skip recent activity check based on author](/docs/examples/author/authorFilter.json5) - Skip a Recent Activity check for a set of subreddits if the Author of the Submission has any set of flairs.

View File

@@ -1,69 +0,0 @@
{
"checks": [
{
"name": "Karma/Meme Sub Activity",
"description": "Report on karma sub activity or meme sub activity if user isn't a memelord",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "freekarma",
"kind": "recentActivity",
"lookAt": "submissions",
"thresholds": [
{
"threshold": ">= 1",
"subreddits": [
"DeFreeKarma",
"FreeKarma4U",
]
}
],
"window": "7 days"
},
{
"name": "noobmemer",
"kind": "recentActivity",
// authors filter will be checked before a rule is run. If anything passes then the Rule is skipped -- it is not failed or triggered.
// if *all* Rules for a Check are skipped due to authors filter then the Check will fail
"authorIs": {
// each property (include/exclude) can contain multiple AuthorCriteria
// if any AuthorCriteria passes its test the Rule is skipped
//
// for an AuthorCriteria to pass all properties present on it must pass
//
// if "include" is present it will always run and exclude will be skipped
// "include:" []
"exclude": [
// for this to pass the Author of the Submission must not have the flair "Supreme Memer" and have the name "user1" or "user2"
{
"flairText": ["Supreme Memer"],
"names": ["user1","user2"]
},
{
// for this to pass the Author of the Submission must not have the flair "Decent Memer"
"flairText": ["Decent Memer"]
}
]
},
"lookAt": "submissions",
"thresholds": [
{
"threshold": ">= 1",
"subreddits": [
"dankmemes",
]
}
],
"window": "7 days"
}
],
"actions": [
{
"kind": "report",
"content": "Author has posted in free karma sub, or in /r/dankmemes and does not have meme flair in this subreddit"
}
]
}
]
}

View File

@@ -1,29 +0,0 @@
{
"checks": [
{
"name": "Flair New User Sub",
"description": "Flair submission as sketchy if user does not have vet flair",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "newflair",
"kind": "author",
// rule will trigger if Author does not have "vet" flair text
"exclude": [
{
"flairText": ["vet"]
}
]
}
],
"actions": [
{
"kind": "flair",
"text": "New User",
"css": "orange"
}
]
}
]
}

View File

@@ -1,29 +0,0 @@
{
"checks": [
{
"name": "Flair Vetted User Submission",
"description": "Flair submission as Approved if user has vet flair",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "newflair",
"kind": "author",
// rule will trigger if Author has "vet" flair text
"include": [
{
"flairText": ["vet"]
}
]
}
],
"actions": [
{
"kind": "flair",
"text": "Vetted",
"css": "green"
}
]
}
]
}

View File

@@ -1,75 +0,0 @@
{
"checks": [
{
"name": "non-vetted karma/meme activity",
"description": "Report if Author has SP and has recent karma/meme sub activity and isn't vetted",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
// The Author Rule is best used in conjunction with other Rules --
// instead of having to write an AuthorFilter for every Rule where you want to skip it based on Author criteria
// you can write one Author Rule and make it fail on the required criteria
// so that the check fails and Actions don't run
"name": "nonvet",
"kind": "author",
"exclude": [
{
"flairText": ["vet"]
}
]
},
{
"name": "attr10",
"kind": "attribution",
"criteria": [
{
"threshold": "> 10%",
"window": "90 days"
},
{
"threshold": "> 10%",
"window": 100
}
],
},
{
"name": "freekarma",
"kind": "recentActivity",
"lookAt": "submissions",
"thresholds": [
{
"threshold": ">= 1",
"subreddits": [
"DeFreeKarma",
"FreeKarma4U",
]
}
],
"window": "7 days"
},
{
"name": "memes",
"kind": "recentActivity",
"lookAt": "submissions",
"thresholds": [
{
"threshold": ">= 3",
"subreddits": [
"dankmemes",
]
}
],
"window": "7 days"
}
],
// will NOT run if the Author for this Submission has the flair "vet"
"actions": [
{
"kind": "report",
"content": "Author has posted in free karma or meme subs recently"
}
]
}
]
}

View File

@@ -1,13 +0,0 @@
# History
The **History** rule can check an Author's submission/comment statistics over a time period:
* Submission total or percentage of All Activity
* Comment total or percentage of all Activity
* Comments made as OP (commented in their own Submission) total or percentage of all Comments
Consult the [schema](https://json-schema.app/view/%23%2Fdefinitions%2FHistoryJSONConfig?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) for a complete reference of the rule's properties.
### Examples
* [Low Comment Engagement](/docs/examples/history/lowEngagement.json5) - Check if Author is submitting much more than they comment.
* [OP Comment Engagement](/docs/examples/history/opOnlyEngagement.json5) - Check if Author is mostly engaging only in their own content

View File

@@ -1,30 +0,0 @@
{
"checks": [
{
"name": "Low Comment Engagement",
"description": "Check if Author is submitting much more than they comment",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "lowComm",
"kind": "history",
"criteria": [
{
// look at last 90 days of Author's activities
"window": "90 days",
// trigger if less than 30% of their activities in this time period are comments
"comment": "< 30%"
},
]
}
],
"actions": [
{
"kind": "report",
"content": "Low engagement: comments were {{rules.lowcomm.commentPercent}} of {{rules.lowcomm.activityTotal}} over {{rules.lowcomm.window}}"
}
]
}
]
}

View File

@@ -1,30 +0,0 @@
{
"checks": [
{
"name": "Engaging Own Content Only",
"description": "Check if Author is mostly engaging in their own content only",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "opOnly",
"kind": "history",
"criteria": [
{
// look at last 90 days of Author's activities
"window": "90 days",
// trigger if more than 60% of their activities in this time period are comments as OP
"comment": "> 60% OP"
},
]
}
],
"actions": [
{
"kind": "report",
"content": "Selfish OP: {{rules.oponly.opPercent}} of {{rules.oponly.commentTotal}} comments over {{rules.oponly.window}} are as OP"
}
]
}
]
}

View File

@@ -1,10 +0,0 @@
# Recent Activity
The **Recent Activity** rule can check if an Author has made any Submissions/Comments in a list of defined Subreddits.
Consult the [schema](https://json-schema.app/view/%23%2Fdefinitions%2FRecentActivityRuleJSONConfig?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) for a complete reference of the rule's properties.
### Examples
* [Free Karma Subreddits](/docs/examples/recentActivity/freeKarma.json5) - Check if the Author has recently posted in any "free karma" subreddits
* [Submission in Free Karma Subreddits](/docs/examples/recentActivity/freeKarmaOnSubmission.json5) - Check if the Author has posted the Submission this check is running on in any "free karma" subreddits recently

View File

@@ -1,40 +0,0 @@
{
"checks": [
{
"name": "Free Karma Alert",
"description": "Check if author has posted in 'freekarma' subreddits",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "freekarma",
"kind": "recentActivity",
"useSubmissionAsReference": false,
// when `lookAt` is not present this rule will look for submissions and comments
// lookAt: "submissions"
// lookAt: "comments"
"thresholds": [
{
// for all subreddits, if the number of activities (sub/comment) is equal to or greater than 1 then the rule is triggered
"threshold": ">= 1",
"subreddits": [
"DeFreeKarma",
"FreeKarma4U",
"FreeKarma4You",
"upvote"
]
}
],
// will look at all of the Author's activities in the last 7 days
"window": "7 days"
}
],
"actions": [
{
"kind": "report",
"content": "{{rules.freekarma.totalCount}} activities in karma {{rules.freekarma.subCount}} subs over {{rules.freekarma.window}}: {{rules.freekarma.subSummary}}"
}
]
}
]
}

View File

@@ -1,41 +0,0 @@
{
"checks": [
{
"name": "Free Karma On Submission Alert",
"description": "Check if author has posted this submission in 'freekarma' subreddits",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "freekarmasub",
"kind": "recentActivity",
// rule will only look at Author's submissions in these subreddits
"lookAt": "submissions",
// rule will only look at Author's submissions in these subreddits that have the same content (link) as the submission this event was made on
// In simpler terms -- rule will only check to see if the same link the author just posted is also posted in these subreddits
"useSubmissionAsReference":true,
"thresholds": [
{
// for all subreddits, if the number of activities (sub/comment) is equal to or greater than 1 then the rule is triggered
"threshold": ">= 1",
"subreddits": [
"DeFreeKarma",
"FreeKarma4U",
"FreeKarma4You",
"upvote"
]
}
],
// look at all of the Author's submissions in the last 7 days
"window": "7 days"
}
],
"actions": [
{
"kind": "report",
"content": "Submission posted {{rules.freekarmasub.totalCount}} times in karma {{rules.freekarmasub.subCount}} subs over {{rules.freekarmasub.window}}: {{rules.freekarmasub.subSummary}}"
}
]
}
]
}

View File

@@ -1,20 +0,0 @@
The **Regex** rule matches on text content from a comment or submission in the same way automod uses regex. The rule, however, provides additional functionality automod does not:
* Can set the **number** of matches that trigger the rule (`matchThreshold`)
Which can then be used in conjunction with a [`window`](https://github.com/FoxxMD/context-mod/blob/master/docs/activitiesWindow.md) to match against activities from the history of the Author of the Activity being checked (including the Activity being checked):
* Can set the **number of Activities** that meet the `matchThreshold` to trigger the rule (`activityMatchThreshold`)
* Can set the **number of total matches** across all Activities to trigger the rule (`totalMatchThreshold`)
* Can set the **type of Activities** to check (`lookAt`)
* When an Activity is a Submission can **specify which parts of the Submission to match against** IE title, body, and/or url (`testOn`)
### Examples
* [Trigger if regex matches against the current activity](/docs/examples/regex/matchAnyCurrentActivity.json5)
* [Trigger if regex matches 5 times against the current activity](/docs/examples/regex/matchThresholdCurrentActivity.json5)
* [Trigger if regex matches against any part of a Submission](/docs/examples/regex/matchSubmissionParts.json5)
* [Trigger if regex matches any of Author's last 10 activities](/docs/examples/regex/matchHistoryActivity.json5)
* [Trigger if regex matches at least 3 of Author's last 10 activities](/docs/examples/regex/matchActivityThresholdHistory.json5)
* [Trigger if there are 5 regex matches in the Author's last 10 activities](/docs/examples/regex/matchTotalHistoryActivity.json5)
* [Trigger if there are 5 regex matches in the Author's last 10 comments](/docs/examples/regex/matchSubsetHistoryActivity.json5)

View File

@@ -1,30 +0,0 @@
{
"checks": [
{
"name": "Burstpost Spam",
"description": "Check if Author is crossposting in short bursts",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "burstpost",
"kind": "repeatActivity",
// will only look at Submissions in Author's history that contain the same content (link) as the Submission this check was initiated by
"useSubmissionAsReference": true,
// the number of non-repeat activities (submissions or comments) to ignore between repeat submissions
"gapAllowance": 3,
// if the Author has posted this Submission 6 times, ignoring 3 non-repeat activities between each repeat, then this rule will trigger
"threshold": ">= 6",
// look at all of the Author's submissions in the last 7 days
"window": "7 days"
}
],
"actions": [
{
"kind": "report",
"content": "Author has burst-posted this link {{rules.burstpost.largestRepeat}} times over {{rules.burstpost.window}}"
}
]
}
]
}

View File

@@ -1,28 +0,0 @@
{
"checks": [
{
"name": "Crosspost Spam",
"description": "Check if Author is spamming Submissions across subreddits",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "xpostspam",
"kind": "repeatActivity",
// will only look at Submissions in Author's history that contain the same content (link) as the Submission this check was initiated by
"useSubmissionAsReference": true,
// if the Author has posted this Submission 5 times consecutively then this rule will trigger
"threshold": ">= 5",
// look at all of the Author's submissions in the last 7 days
"window": "7 days"
}
],
"actions": [
{
"kind": "report",
"content": "Author has posted this link {{rules.xpostspam.largestRepeat}} times over {{rules.xpostspam.window}}"
}
]
}
]
}

View File

@@ -1,41 +0,0 @@
Provided here are **complete, ready-to-go configuration** that can copy-pasted straight into your configuration wiki page to get going with ContextMod immediately.
These configurations attempt to provide sensible, non-destructive, default behavior for some common scenarios and subreddit types.
In most cases these will perform decently out-of-the-box but they are not perfect. You should still monitor bot behavior to see how it performs and will most likely still need to tweak these configurations to get your desired behavior.
All actions for these configurations are non-destructive in that:
* All instances where an activity would be modified (remove/ban/approve) will have `dryRun: true` set to prevent the action from actually being performed
* These instances will also have a `report` action detailing the action would have been performed
**You will have to remove the `report` action and `dryRun` settings yourself.** This is to ensure that you understand the behavior the bot will be performing. If you are unsure of this you should leave them in place until you are certain the behavior the bot is performing is acceptable.
## Submission-based Behavior
### [Remove submissions from users who have used 'freekarma' subs to bypass karma checks](/docs/examples/subredditReady/freekarma.json5)
If the user has any activity (comment/submission) in known freekarma subreddits in the past (50 activities or 6 months) then remove the submission.
### [Remove submissions from users who have crossposted the same submission 4 or more times](/docs/examples/subredditReady/crosspostSpam.json5)
If the user has crossposted the same submission in the past (50 activities or 6 months) 4 or more times in a row then remove the submission.
### [Remove submissions from users who have crossposted or used 'freekarma' subs](/docs/examples/subredditReady/freeKarmaOrCrosspostSpam.json5)
Will remove submission if either of the above two behaviors is detected
### [Remove link submissions where the user's history is comprised of 10% or more of the same link](/docs/examples/subredditReady/selfPromo.json5)
If the link origin (youtube author, twitter author, etc. or regular domain for non-media links)
* comprises 10% or more of the users **entire** history in the past (100 activities or 6 months)
* or comprises 10% or more of the users **submission** history in the past (100 activities or 6 months) and the user has low engagement (<50% of history is comments or 40%> of comment are as OP)
then remove the submission
## Comment-based behavior
### [Remove comment if the user has posted the same comment 4 or more times in a row](/docs/examples/subredditReady/commentSpam.json5)
If the user made the same comment (with some fuzzy matching) 4 or more times in a row in the past (50 activities or 6 months) then remove the comment.

View File

@@ -1,42 +0,0 @@
{
"polling": ["newComm"],
"checks": [
{
//
// Stop users who spam the same comment many times
//
// Remove a COMMENT if the user has crossposted it at least 4 times in recent history
//
"name": "low xp comment spam",
"description": "X-posted comment >=4x",
"kind": "comment",
"condition": "AND",
"rules": [
{
"name": "xPostLow",
"kind": "repeatActivity",
"gapAllowance": 2,
"threshold": ">= 4",
"window": {
"count": 50,
"duration": "6 months"
}
},
],
"actions": [
// remove this after confirming behavior is acceptable
{
"kind": "report",
"content": "Remove=> Posted same comment {{rules.xpostlow.largestRepeat}}x times"
},
//
//
{
"kind": "remove",
// remove the line below after confirming behavior is acceptable
"dryRun": true
}
]
}
]
}

View File

@@ -1,77 +0,0 @@
{
"polling": ["unmoderated"],
"checks": [
{
//
// Stop users who post low-effort, crossposted spam
//
// Remove a SUBMISSION if the user has crossposted it at least 4 times in recent history AND
// less than 50% of their activity is comments OR more than 40% of those comments are as OP (in the own submissions)
//
"name": "low xp spam and engagement",
"description": "X-posted 4x and low comment engagement",
"kind": "submission",
"itemIs": [
{
"removed": false
}
],
"condition": "AND",
"rules": [
{
"name": "xPostLow",
"kind": "repeatActivity",
"gapAllowance": 2,
"threshold": ">= 4",
"window": {
"count": 50,
"duration": "6 months"
}
},
{
"name": "lowOrOpComm",
"kind": "history",
"criteriaJoin": "OR",
"criteria": [
{
"window": {
"count": 100,
"duration": "6 months"
},
"comment": "< 50%"
},
{
"window": {
"count": 100,
"duration": "6 months"
},
"comment": "> 40% OP"
}
]
}
],
"actions": [
// remove this after confirming behavior is acceptable
{
"kind": "report",
"content": "Remove=>{{rules.xpostlow.largestRepeat}} X-P => {{rules.loworopcomm.thresholdSummary}}"
},
//
//
{
"kind": "remove",
// remove the line below after confirming behavior is acceptable
"dryRun": true
},
// optionally remove "dryRun" from below if you want to leave a comment on removal
// PROTIP: the comment is bland, you should make it better
{
"kind": "comment",
"content": "Your submission has been removed because you cross-posted it {{rules.xpostlow.largestRepeat}} times and you have very low engagement outside of making submissions",
"distinguish": true,
"dryRun": true
}
]
}
]
}

View File

@@ -1,138 +0,0 @@
{
"polling": [
"unmoderated"
],
"checks": [
{
//
// Stop users who post low-effort, crossposted spam
//
// Remove a SUBMISSION if the user has crossposted it at least 4 times in recent history AND
// less than 50% of their activity is comments OR more than 40% of those comments are as OP (in the own submissions)
//
"name": "remove on low xp spam and engagement",
"description": "X-posted 4x and low comment engagement",
"kind": "submission",
"itemIs": [
{
"removed": false
}
],
"condition": "AND",
"rules": [
{
"name": "xPostLow",
"kind": "repeatActivity",
"gapAllowance": 2,
"threshold": ">= 4",
"window": {
"count": 50,
"duration": "6 months"
}
},
{
"name": "lowOrOpComm",
"kind": "history",
"criteriaJoin": "OR",
"criteria": [
{
"window": {
"count": 100,
"duration": "6 months"
},
"comment": "< 50%"
},
{
"window": {
"count": 100,
"duration": "6 months"
},
"comment": "> 40% OP"
}
]
}
],
"actions": [
// remove this after confirming behavior is acceptable
{
"kind": "report",
"content": "Remove=>{{rules.xpostlow.largestRepeat}} X-P => {{rules.loworopcomm.thresholdSummary}}"
},
//
//
{
"kind": "remove",
// remove the line below after confirming behavior is acceptable
"dryRun": true
},
// optionally remove "dryRun" from below if you want to leave a comment on removal
// PROTIP: the comment is bland, you should make it better
{
"kind": "comment",
"content": "Your submission has been removed because you cross-posted it {{rules.xpostlow.largestRepeat}} times and you have very low engagement outside of making submissions",
"distinguish": true,
"dryRun": true
}
]
},
{
//
// Remove submissions from users who have recent activity in freekarma subs within the last 50 activities or 6 months (whichever is less)
//
"name": "freekarma removal",
"description": "Remove submission if user has used freekarma sub recently",
"kind": "submission",
"itemIs": [
{
"removed": false
}
],
"condition": "AND",
"rules": [
{
"name": "freekarma",
"kind": "recentActivity",
"window": {
"count": 50,
"duration": "6 months"
},
"useSubmissionAsReference": false,
"thresholds": [
{
"subreddits": [
"FreeKarma4U",
"FreeKarma4You",
"KarmaStore",
"promote",
"shamelessplug",
"upvote"
]
}
]
}
],
"actions": [
// remove this after confirming behavior is acceptable
{
"kind": "report",
"content": "Remove=> {{rules.newtube.totalCount}} activities in freekarma subs"
},
//
//
{
"kind": "remove",
// remove the line below after confirming behavior is acceptable
"dryRun": true
},
// optionally remove "dryRun" from below if you want to leave a comment on removal
// PROTIP: the comment is bland, you should make it better
{
"kind": "comment",
"content": "Your submission has been removed because you have recent activity in 'freekarma' subs",
"distinguish": true,
"dryRun": true
}
]
}
]
}

View File

@@ -1,64 +0,0 @@
{
"polling": [
"unmoderated"
],
"checks": [
{
//
// Remove submissions from users who have recent activity in freekarma subs within the last 50 activities or 6 months (whichever is less)
//
"name": "freekarma removal",
"description": "Remove submission if user has used freekarma sub recently",
"kind": "submission",
"itemIs": [
{
"removed": false
}
],
"condition": "AND",
"rules": [
{
"name": "freekarma",
"kind": "recentActivity",
"window": {
"count": 50,
"duration": "6 months"
},
"useSubmissionAsReference": false,
"thresholds": [
{
"subreddits": [
"FreeKarma4U",
"FreeKarma4You",
"KarmaStore",
"upvote"
]
}
]
}
],
"actions": [
// remove this after confirming behavior is acceptable
{
"kind": "report",
"content": "Remove=> {{rules.newtube.totalCount}} activities in freekarma subs"
},
//
//
{
"kind": "remove",
// remove the line below after confirming behavior is acceptable
"dryRun": true,
},
// optionally remove "dryRun" from below if you want to leave a comment on removal
// PROTIP: the comment is bland, you should make it better
{
"kind": "comment",
"content": "Your submission has been removed because you have recent activity in 'freekarma' subs",
"distinguish": true,
"dryRun": true,
}
]
}
]
}

View File

@@ -1,104 +0,0 @@
{
"polling": [
"unmoderated"
],
"checks": [
{
//
// Stop users who make link submissions with a self-promotional agenda (with reddit's suggested 10% rule)
// https://www.reddit.com/wiki/selfpromotion#wiki_guidelines_for_self-promotion_on_reddit
//
// Remove a SUBMISSION if the link comprises more than or equal to 10% of users history (100 activities or 6 months) OR
//
// if link comprises 10% of submission history (100 activities or 6 months)
// AND less than 50% of their activity is comments OR more than 40% of those comments are as OP (in the own submissions)
//
"name": "Self-promo all AND low engagement",
"description": "Self-promo is >10% for all or just sub and low comment engagement",
"kind": "submission",
"condition": "OR",
"rules": [
{
"name": "attr",
"kind": "attribution",
"criteria": [
{
"threshold": ">= 10%",
"window": {
"count": 100,
"duration": "6 months"
},
"domains": [
"AGG:SELF"
]
}
],
},
{
"condition": "AND",
"rules": [
{
"name": "attrsub",
"kind": "attribution",
"criteria": [
{
"threshold": ">= 10%",
"thresholdOn": "submissions",
"window": {
"count": 100,
"duration": "6 months"
},
"domains": [
"AGG:SELF"
]
}
]
},
{
"name": "lowOrOpComm",
"kind": "history",
"criteriaJoin": "OR",
"criteria": [
{
"window": {
"count": 100,
"duration": "6 months"
},
"comment": "< 50%"
},
{
"window": {
"count": 100,
"duration": "6 months"
},
"comment": "> 40% OP"
}
]
}
]
}
],
"actions": [
{
"kind": "report",
"content": "{{rules.attr.largestPercent}}{{rules.attrsub.largestPercent}} of {{rules.attr.activityTotal}}{{rules.attrsub.activityTotal}} items ({{rules.attr.window}}{{rules.attrsub.window}}){{#rules.loworopcomm.thresholdSummary}} => {{rules.loworopcomm.thresholdSummary}}{{/rules.loworopcomm.thresholdSummary}}"
},
//
//
{
"kind": "remove",
// remove the line below after confirming behavior is acceptable
"dryRun": true
},
// optionally remove "dryRun" from below if you want to leave a comment on removal
// PROTIP: the comment is bland, you should make it better
{
"kind": "comment",
"content": "Your submission has been removed it comprises 10% or more of your recent history ({{rules.attr.largestPercent}}{{rules.attrsub.largestPercent}}). This is against [reddit's self promotional guidelines.](https://www.reddit.com/wiki/selfpromotion#wiki_guidelines_for_self-promotion_on_reddit)",
"distinguish": true,
"dryRun": true
}
]
}
]
}

View File

@@ -1,45 +0,0 @@
{
"checks": [
{
"name": "Self Promo Activities",
"description": "Tag SP only if user does not have good contributor user note",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "attr10all",
"kind": "attribution",
"author": {
"exclude": [
{
// the key of the usernote type to look for https://github.com/toolbox-team/reddit-moderator-toolbox/wiki/Subreddit-Wikis%3A-usernotes#working-with-note-types
// rule will not run if current usernote on Author is of type 'gooduser'
"type": "gooduser"
}
]
},
"criteria": [
{
"threshold": "> 10%",
"window": "90 days"
},
{
"threshold": "> 10%",
"window": 100
}
],
}
],
"actions": [
{
"kind": "usernote",
// the key of usernote type
// https://github.com/toolbox-team/reddit-moderator-toolbox/wiki/Subreddit-Wikis%3A-usernotes#working-with-note-types
"type": "spamwarn",
// content is mustache templated as usual
"content": "Self Promotion: {{rules.attr10all.titlesDelim}} {{rules.attr10sub.largestPercent}}%"
}
]
}
]
}

View File

@@ -1,36 +0,0 @@
{
"checks": [
{
"name": "Self Promo Activities",
"description": "Check if any of Author's aggregated submission origins are >10% of entire history",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "attr10all",
"kind": "attribution",
"criteria": [
{
"threshold": "> 10%",
"window": "90 days"
},
{
"threshold": "> 10%",
"window": 100
}
],
}
],
"actions": [
{
"kind": "usernote",
// the key of usernote type
// https://github.com/toolbox-team/reddit-moderator-toolbox/wiki/Subreddit-Wikis%3A-usernotes#working-with-note-types
"type": "spamwarn",
// content is mustache templated as usual
"content": "Self Promotion: {{rules.attr10all.titlesDelim}} {{rules.attr10sub.largestPercent}}%"
}
]
}
]
}

View File

@@ -1,71 +0,0 @@
This getting started guide is for **Operators** -- that is, someone who wants to run the actual software for a ContentMod bot. If you are a **Moderator** check out the [moderator getting started](/docs/gettingStartedMod.md) guide instead.
# Table of Contents
* [Installation](#installation)
* [Docker](#docker-recommended)
* [Locally](#locally)
* [Heroku](#heroku-quick-deployhttpsherokucomabout)
* [Bot Authentication](#bot-authentication)
* [Instance Configuration](#instance-configuration)
* [Run Your Bot and Start Moderating](#run-your-bot-and-start-moderating)
# Installation
In order to run a ContextMod instance you must first you must install it somewhere.
ContextMod can be run on almost any operating system but it is recommended to use Docker due to ease of deployment.
## Docker (Recommended)
PROTIP: Using a container management tool like [Portainer.io CE](https://www.portainer.io/products/community-edition) will help with setup/configuration tremendously.
### [Dockerhub](https://hub.docker.com/r/foxxmd/context-mod)
```
foxxmd/context-mod:latest
```
Adding **environmental variables** to your `docker run` command will pass them through to the app EX:
```
docker run -d -e "CLIENT_ID=myId" ... foxxmd/context-mod
```
### Locally
Requirements:
* Typescript >=4.3.5
* Node >=15
Clone this repository somewhere and then install from the working directory
```bash
git clone https://github.com/FoxxMD/context-mod.git .
cd context-mod
npm install
tsc -p .
```
### [Heroku Quick Deploy](https://heroku.com/about)
[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://dashboard.heroku.com/new?template=https://github.com/FoxxMD/context-mod)
# Bot Authentication
Next you need to create a bot and authenticate it with Reddit. Follow the [bot authentication guide](/docs/botAuthentication.md) to complete this step.
# Instance Configuration
Finally, you must provide the credentials you received from the **Bot Authentication** step to the ContextMod instance you installed earlier. Refer to the [Operator Configuration](/docs/operatorConfiguration.md) guide to learn how this can be done as there are multiple approaches depending on how you installed the software.
Additionally, at this step you can also tweak many more settings and behavior concerning how your CM bot will operate.
# Run Your Bot and Start Moderating
Congratulations! You should now have a fully authenticated bot running on ContextMod software.
In order for your Bot to operate on reddit though it **must be a moderator in the subreddit you want it to run in.** This may be your own subreddit or someone else's.
**Note: ContextMod does not currently handle moderation invites automatically** and may never have this functionality. Due to the fact that many of its behaviors are api-heavy and that subreddits can control their own configuration the api and resource (cpu/memory) usage of a ContextMod instance can be highly variable. It therefore does not make sense to allow any/all subreddits to automatically have access to an instance through automatically accepting moderator invites. So...if you are planning to run a ContextMod instance for subreddits other than those you moderate you should establish solid trust with moderators of that subreddit as well as a solid line of communication in order to ensure their configurations can be tailored to best fit their needs and your resources.
Once you have logged in as your bot and manually accepted the moderator invite you will need to restart your ContextMod instance in order for these changes to take effect.

View File

@@ -2,11 +2,12 @@
ContextMod supports comparing image content, for the purpose of detecting duplicates, with two different but complimentary systems. Image comparison behavior is available for the following rules:
* [Recent Activity](/docs/examples/recentActivity)
* [Recent Activity](/docs/subreddit/components/recentActivity)
* Repeat Activity (In-progress)
To enable comparisons reference the example below (at the top-level of your rule) and configure as needed:
JSON
```json5
{
"name": "ruleWithImageDetection",
@@ -31,9 +32,18 @@ To enable comparisons reference the example below (at the top-level of your rule
},
//
// And above ^^^
...
//...
}
```
YAML
```yaml
name: ruleWithImageDetection
kind: recentActivity
enable: true
threshold: 5
fetchBehavior: extension
```
**Perceptual Hashing** (`hash`) and **Pixel Comparisons** (`pixel`) may be used at the same time. Refer to the documentation below to see how they interact.
@@ -114,10 +124,12 @@ To further configure hashing refer to this code block:
// the higher the bits the more accurate the comparison
//
// NOTE: Hashes of different sizes (bits) cannot be compared. If you are caching hashes make sure all rules where results may be shared use the same bit count to ensure hashes can be compared. Otherwise hashes will be recomputed.
"bits": 32, // default is 32 if not defined
"bits": 32,
// default is 32 if not defined
//
// number of seconds to cache an image hash
"ttl": 60, // default is 60 if not defined
"ttl": 60,
// default is 60 if not defined
//
// "High Confidence" Threshold
// If the difference in comparison is equal to or less than this number the images are considered the same and pixel comparison WILL NOT occur
@@ -139,8 +151,22 @@ To further configure hashing refer to this code block:
//
// And above ^^^
//"pixel": {...}
},
}
//...
}
```
YAML
```yaml
name: ruleWithImageDetectionAndConfiguredHashing
kind: recentActivity
imageDetection:
enable: true
hash:
enable: true
bits: 32
ttl: 60
hardThreshold: 5
softThreshold: 0
```
## Pixel Comparison
@@ -184,18 +210,28 @@ To configure pixel comparisons refer to this code block:
```json5
{
"name": "ruleWithImageDetectionAndPixelEnabled",
"kind": "recentActivity",
"imageDetection": {
//"hash": {...}
"pixel": {
// enable or disable pixel comparisons (disabled by default)
"enable": true,
// if the comparison difference percentage is equal to or less than this value the images are considered the same
//
// if not defined the value from imageDetection.threshold will be used
"threshold": 5
}
},
//...
"name": "ruleWithImageDetectionAndPixelEnabled",
"kind": "recentActivity",
"imageDetection": {
//"hash": {...}
"pixel": {
// enable or disable pixel comparisons (disabled by default)
"enable": true,
// if the comparison difference percentage is equal to or less than this value the images are considered the same
//
// if not defined the value from imageDetection.threshold will be used
"threshold": 5
}
},
//...
}
```
YAML
```yaml
name: ruleWithImageDetectionAndPixelEnabled
kind: recentActivity
imageDetection:
pixel:
enable: true
threshold: 5
```

Binary file not shown.

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 93 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

BIN
docs/images/config/save.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 11 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

BIN
docs/images/configBox.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 45 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 111 KiB

BIN
docs/images/editor.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 133 KiB

BIN
docs/images/grafana.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 183 KiB

BIN
docs/images/guests.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

BIN
docs/images/logs.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 133 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 137 KiB

BIN
docs/images/oauth.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 158 KiB

BIN
docs/images/runInput.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 141 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 225 KiB

BIN
docs/logo.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

78
docs/operator/README.md Normal file
View File

@@ -0,0 +1,78 @@
# Operator Guide
An **Operator** is the user **running the ContextMod software.**
They are responsible for configuring the software at a high-level and managing associated infrastructure such as:
* Creating cache/database servers and configuring their connections in CM
* Provisioning the [Reddit Clients](#provisioning-a-reddit-client) needed to run bots and the CM UI
* Providing [global-level configuration](/docs/operator/configuration.md) that affects general bot/subreddit behavior
* Onboarding new bots/subreddits
# Table of Contents
* [Overview](#overview)
* [Client-Server Architecture](/docs/serverClientArchitecture.md)
* [Getting Started](/docs/operator/gettingStarted.md)
* [Installation](/docs/operator/installation.md)
* [Provisioning a Reddit Client](#provisioning-a-reddit-client)
* [Configuration](/docs/operator/configuration.md)
* [Adding A Bot](/docs/operator/addingBot.md)
# Overview
CM is composed of two applications that operate independently but are packaged together such that they act as one piece of software:
* **Server** -- Responsible for **running the bot(s)** and providing an API to retrieve information on and interact with them EX start/stop bot, reload config, retrieve operational status, etc.
* **Client** -- Responsible for serving the **web interface** and handling the bot oauth authentication flow between operators and subreddits/bots.
Both applications authenticate, and are primarily operated, by using [Reddit's API through OAuth.](https://github.com/reddit-archive/reddit/wiki/OAuth2) The **Client** uses OAuth to verify the identity of moderators logging into the web interface. The **Server** uses oauth tokens to interact with Reddit's API and operate all the configured bots.
In its default mode of operation CM takes care of all the interaction between **Server** and **Client** for you so that you can effectively treat it as a monolithic application. Learn more about CM's architecture and other operation modes in the [Server-Client Architecture documentation.](/docs/serverClientArchitecture.md)
# [Getting Started](/docs/operator/gettingStarted.md)
The [Getting Started](/docs/operator/gettingStarted.md) guide serves as a straight-forward "how-to" for standing up a CM server from scratch with minimal explanation.
# [Installation](/docs/operator/installation.md)
CM has many installation options:
* Locally, from source, as a typescript project
* Built/pulled from a Docker image hosted on Dockerhub
* Deployed to Heroku with a Quick Deploy template (experimental)
Refer to the [Installation](/docs/operator/installation.md) docs for more information.
# Provisioning A Reddit Client
As mentioning in the [Overview](#overview), CM operates primarily using Reddit's API through OAuth. You must create a [Reddit Client](https://github.com/reddit-archive/reddit/wiki/OAuth2#getting-started) in order to interact with the API.
## Create Application
Visit [your reddit preferences](https://www.reddit.com/prefs/apps) and at the bottom of the page go through the **create an(other) app** process.
* Give it a **name**
* Choose **web app**
* If you know what you will use for **redirect uri** go ahead and use it, otherwise use `http://localhost:8085/callback`
Click **create app**.
Then write down your **Client ID, Client Secret, and Redirect Uri** somewhere
# [Configuration](/docs/operator/configuration.md)
The [Configuration](/docs/operator/configuration.md) documentation covers:
* How CM's configuration can be defined
* How to create and define location for a config file
* Running CM from the command line
* Documentation for configuration on Bots, the web client, API, and more...
# [Adding A Bot](/docs/operator/addingBot.md)
The [Adding A Bot](/docs/operator/addingBot.md) documentation covers:
* What is a Bot?
* What is needed to add a Bot to CM?
* Different approaches to authenticating and adding a Bot to CM

View File

@@ -0,0 +1,86 @@
# Table of Contents
* [What is a Bot?](#what-is-a-bot)
* [Prerequisites](#Prerequisites)
* [Adding a Bot to CM](#adding-a-bot-to-cm)
* [Using CM OAuth Helper (Recommended)](#cm-oauth-helper-recommended)
* [Using Aardvark OAuth Helper](#aardvark-oauth-helper)
# What is a Bot?
A **reddit bot** is composed of two components:
* A normal **reddit account** like `u/MyRedditAccount`
* Software that performs actions **on behalf of that reddit account** using Reddit's API
There is nothing special about the account! What's special is how its used -- through the API *with bot software* like ContextMod.
# Prerequisites
These things need to be done before a Bot can be added to CM:
* [Provisioned a Reddit Client](/docs/operator/README.md#provisioning-a-reddit-client)
* You or the person who controls the Bot account must have account credentials (username/password). Logging in to reddit is part of the setup process.
* If the bot does not exist **create a reddit account for it.**
* If the bot does exist make sure you are in communication with the owner of the account.
# Adding A Bot to CM
## CM OAuth Helper (Recommended)
This method will use CM's built in oauth flow. It is recommended because:
* It's easy!
* Will ensure your bot is authenticated with the correct oauth permissions
### Start CM with the Minimum Configuration (Initial Setup)
If this is your **first time adding a bot** you must make sure you have
* done the [prerequisites](#prerequisites)
* created a [minimum operator configuration](/docs/operator/configuration.md#minimum-config)
* that specifies the client id/secret from provisioning your reddit client
* specified **Operator Name** in the configuration
It is important you define **Operator Name** because the auth route is **protected.** You must login to CM's web interface in order to access the route.
### Create A Bot Invite
Open the CM web interface (default is [http://localhost:8085](http://localhost:8085)) and login with the reddit account specified in **Operator Name.**
If this is your first time setting up a bot you should be automatically redirected to the auth page. Otherwise, visit [http://localhost:8085/auth/helper](http://localhost:8085/auth/helper)
Follow the directions in the helper to create a **Bot Invite Link.**
### Onboard the Bot
Visit the **Bot Invite Link** while **logged in to reddit as the bot account** to begin the onboarding process. Refer to the [Onboarding Your Bot]() subreddit documentation for more information on this process.
At the end of the onboarding process the bot should be automatically added to your operator configuration. If there is an issue with automatically adding it then the oauth credentials will be displayed at the end of onboarding and can be [manually added to the configuration.](/docs/operator/configuration.md#manually-adding-a-bot)
## Aardvark OAuth Helper
This method should only be used if you cannot use the [CM OAuth Helper method.](#cm-oauth-helper-recommended)
* Visit [https://not-an-aardvark.github.io/reddit-oauth-helper/](https://not-an-aardvark.github.io/reddit-oauth-helper/) and follow the instructions given.
* **Note:** You will need to update the **redirect uri** you set when [provisioning your reddit client.](/docs/operator/README.md#provisioning-a-reddit-client)
* Input your **Client ID** and **Client Secret** in the text boxes with those names.
* Choose scopes. **It is very important you check everything on this list or CM may not work correctly**
* edit
* flair
* history
* identity
* modcontributors
* modflair
* modposts
* modself
* modnote
* mysubreddits
* read
* report
* submit
* wikiread
* wikiedit (if you are using Toolbox User Notes)
* Click **Generate tokens**, you will get a popup asking you to approve access (or login) -- **the account you approve access with is the account that Bot will control.**
* After approving an **Access Token** and **Refresh Token** will be shown at the bottom of the page. Use these to [manually add a bot to your operator configuration.](/docs/operator/configuration.md#manually-adding-a-bot)
* After adding the bot you will need to restart CM.

161
docs/operator/caching.md Normal file
View File

@@ -0,0 +1,161 @@
# Table of Contents
* [Overview](#overview)
* [What Is Cache?](#what-is-cache)
* [How CM Uses Caching](#how-cm-uses-caching)
* [Reddit API Calls](#reddit-api-calls)
* [Rules and Filters](#rules-and-filters)
* [Configuration](#configuration)
* [Cache Provider](#cache-provider)
# Overview
**Caching** is a major factor in CM's performance and optimization of Reddit API usage. Leveraging caching effectively in your operator configuration and in individual subreddit configurations can make or break your CM instance.
### What Is Cache?
A **Cache** is a storage medium with high **write** and **read** speed that is generally used to store **temporary, but frequently accessed data.**
## How CM Uses Caching
CM primarily **caches** two types of data:
### Reddit API Calls
#### How Reddit's API Works
In order to communicate with Reddit to retrieve posts, comments, user information, etc... CM uses API calls. Each API call is composed of a
* **request** -- CM asks Reddit for certain information
* **response** -- Reddit responds with the request information
[Reddit imposes an **api quota**](https://github.com/reddit-archive/reddit/wiki/API#rules) on every **individual account** using the API through an application. This quota is **600 requests per 10 minutes.** At the end of the 10 minutes period the quota is reset.
Additionally, some API calls have limits on how much data they can return. The most relevant of these is **user history can only be returned 100 activities (submission/comments) per API call**. EX if you want to get **500** activities from a user's history you will need to make **5** api calls.
#### Caching API Responses
In order to most effectively use the limited quota of API calls CM will **automatically cache API responses based on the "fingerprint" of the request sent.**
On an individual "item" basis that means these resources are always cached:
* General user information (name, karma, age, profile description, etc..)
* General subreddit information (name, nsfw, quarantined, etc...)
* Individually processed activities (comment body, is comment author op, submission title, reports, link, etc...)
Additionally (and most importantly), responses for **user history** are cached **based on what was requested**. Example "fingerprint":
* username
* type of activities to retrieve (overview, only submissions, only comments)
* range of activities to retrieve (last 100, last 6 months, etc...)
If the above "fingerprint" is used in three different Rules then
* First fingerprint appearance -> CM make API call and caches response
* Second fingerprint appearance -> CM uses cached response
* Third fingerprint appearance -> CM uses cached response
So only **one** API call is made even though the history is used three times.
It is therefore **important to re-use window criteria** wherever possible to take advantage of this caching.
### Rules and Filters
Once CM has processed a Rule or Filter (`itemIs` or `authorIs`) the results of that component is stored in cache. For Rules the result is stored for the lifecycle of the Activity being processed and then discarded. For Filters the result is stored for a short time in cache and can be re-used by other Activities.
Re-using Rules and Filters by either using the exact same configuration or by using **names** provides:
* A major performance benefit since these do not need to be re-calculated
* A low-to-medium optimization on API caching, depending on what criteria are being tested.
In general re-use should always be a goal.
# Configuration
## Cache Provider
CM supports two cache **providers**. By default all providers use `memory`:
* `memory` -- in-memory (non-persistent) backend
* Cache will be lost when CM is restarted/exits
* `redis` -- [Redis](https://redis.io/) backend
Each `provider` object in configuration can be specified as:
* one of the above **strings** to use the **defaults settings** or
* an **object** with keys to override default settings
[Refer to full documentation on cache providers in the schema](https://json-schema.app/view/%23/%23%2Fdefinitions%2FOperatorCacheConfig/%23%2Fdefinitions%2FCacheOptions?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FOperatorConfig.json)
Some examples:
```json5
{
"provider": {
"store": "memory", // one of "memory" or "redis"
"ttl": 60, // the default max age of a key in seconds
"max": 500, // the maximum number of keys in the cache (for "memory" only)
// the below properties only apply to 'redis' provider
"host": 'localhost',
"port": 6379,
"auth_pass": null,
"db": 0,
}
}
```
YAML
```yaml
provider:
store: redis
ttl: 60
max: 500
host: localhost
port: 6379
auth_pass: null
db: 0
```
Providers can be specified in multiple locations, with each more specific location overriding the parent-level config:
* top-level config
* in individual bot configurations
* in the web config
```yaml
operator:
name: example
# top level
caching:
provider:
...
bots:
- name: u/MyBot
# overrides top level
caching:
provider:
...
web:
# overrides top level
caching:
provider:
...
```
## Cache TTL
The **Time To Live (TTL)** -- how long data may live in the cache before "expiring" -- can be controlled indepedently for different data types. Sane defaults are provided for all types but tweaking these can improve API caching optimization depending on the subreddit's configuration (use case).
Each of these can be specified in the `caching` property. TTL is measured in seconds.
* `authorTTL` (default 60) -- user activity (overview, submissions, comments)
* `commentTTL` (default 60) -- individually fetched comments
* `submissionTTL` (default 60) -- individually fetched submissions
* `filterCriteriaTTL` (default 60) -- filter results (`itemIs` and `authorIs`)
* `selfTTL` (default 60) -- actions performed by the bot (creating comment reply, report). If action is in cache the bot ignores it if found during polling.
* This helps prevent bot from reacting to things it did itself IE you don't want it to remove a comment because it reported the comment itself
* `subredditTTL` (default 60) -- general information on fetched subreddit
* `userNotesTTL` (default 300) -- Amount of time [Toolbox User Notes](https://www.reddit.com/r/toolbox/wiki/docs/usernotes) are cached
* `wikiTTL` (default 300) -- Wiki pages used for content (in messages, reports, bans, etc...) are cached for this amount of time

View File

@@ -2,32 +2,23 @@ The **Operator** configuration refers to configuration used configure to the act
from the **Subreddit** configuration that is defined in each Subreddit's wiki and determines the rules/actions for
activities the Bot runs on.
**The full documentation** for all options in the operator configuration can be found [**here in the operator schema.**](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FOperatorConfig.json)
# Table of Contents
* [Minimum Required Configuration](#minimum-required-configuration)
* [Defining Configuration](#defining-configuration)
* [CLI Usage](#cli-usage)
* [Minimum Configuration](#minimum-configuration)
* [Bots](#bots)
* [Examples](#example-configurations)
* [Minimum Config](#minimum-config)
* [Using Config Overrides](#using-config-overrides)
* [Cache Configuration](#cache-configuration)
# Minimum Required Configuration
| property | Server And Web | Server Only | Web/Bot-Auth Only |
|:--------------:|:------------------:|:------------------:|:------------------:|
| `clientId` | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| `clientSecret` | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| `redirectUri` | :heavy_check_mark: | :x: | :heavy_check_mark: |
| `refreshToken` | :heavy_check_mark: | :heavy_check_mark: | :x: |
| `accessToken` | :heavy_check_mark: | :heavy_check_mark: | :x: |
Refer to the **[Bot Authentication guide](/docs/botAuthentication.md)** to retrieve credentials.
* [Database Configuration](#database-configuration)
# Defining Configuration
CM can be configured using **any or all** of the approaches below. Note that **at each level ALL configuration values are
optional** but the "required configuration" mentioned above must be available when all levels are combined.
CM can be configured using **any or all** of the approaches below. **It is recommended to use FILE ([File Configuration](#file-configuration-recommended))**
Any values defined at a **lower-listed** level of configuration will override any values from a higher-listed
configuration.
@@ -35,18 +26,37 @@ configuration.
* **ENV** -- Environment variables loaded from an [`.env`](https://github.com/toddbluhm/env-cmd) file (path may be
specified with `--file` cli argument)
* **ENV** -- Any already existing environment variables (exported on command line/terminal profile/etc.)
* **FILE** -- Values specified in a JSON configuration file using the structure [in the schema](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FOperatorConfig.json)
* **FILE** -- Values specified in a YAML/JSON configuration file using the structure [in the schema](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FOperatorConfig.json)
* When reading the **schema** if the variable is available at a level of configuration other than **FILE** it will be
noted with the same symbol as above. The value shown is the default.
* **ARG** -- Values specified as CLI arguments to the program (see [ClI Usage](#cli-usage) below)
**Note:** When reading the **schema** if the variable is available at a level of configuration other than **FILE** it will be
noted with the same symbol as above. The value shown is the default.
## File Configuration (Recommended)
* To load a JSON configuration (for **FILE**) **from the command line** use the `-c` cli argument EX: `node src/index.js -c /path/to/JSON/config.json`
* To load a JSON configuration (for **FILE**) **using an environmental variable** use `OPERATOR_CONFIG` EX: `OPERATOR_CONFIG=/path/to/JSON/config.json`
Using a file has many benefits over using ARG or ENV:
[**See the Operator Config Schema here**](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FOperatorConfig.json)
* CM can automatically update your configuration
* CM can automatically add bots via the [CM OAuth Helper](/docs/operator/addingBot.md#cm-oauth-helper-recommended)
* CM has a built-in configuration editor that can help you build and validate your configuration file
* File config is **required** if adding multiple bots to CM
## Defining Multiple Bots or CM Instances
### Specify File Location
By default CM will look for `config.yaml` or `config.json` in the `DATA_DIR` directory:
* [Local installation](/docs/operator/installation.md#locally) -- `DATA_DIR` is the root of your installation directory (same folder as `package.json`)
* [Docker](/docs/operator/installation.md#docker-recommended) -- `DATA_DIR` is at `/config` in the container
The `DATA_DIR` directory can be changed by passing `DATA_DIR` as an environmental variable EX `DATA_DIR=/path/to/directory`
The name of the config file can be changed by passing `OPERATOR_CONFIG` as an environmental variable:
* As filename -- `OPERATOR_CONFIG=myConfig.yaml` -> CM looks for `/path/to/directory/myConfig.yaml`
* As absolute path -- `OPERATOR_CONFIG=/a/path/myConfig.yaml` -> CM looks for `/a/path/myConfig.yaml`
[**Refer to the Operator Config File Schema for full documentation**](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FOperatorConfig.json)
### Defining Multiple Bots or CM Instances
One ContextMod instance can
@@ -112,6 +122,87 @@ Options:
</details>
# Minimum Configuration
The minimum configuration required to run CM assumes you have no bots and want to use CM to [add your first bot.](/docs/operator/addingBot.md#cm-oauth-helper-recommended)
You will need have this information available:
* From [provision a reddit client](/docs/operator/README.md#provisioning-a-reddit-client)
* Client ID
* Client Secret
* Redirect URI (if different from default `http://localhost:8085/callback`)
* Operator Name -- username of the reddit account you want to use to administer CM with
See the [**example minimum configuration** below.](#minimum-config)
This configuration can also be **generated** by CM if you start CM with **no configuration defined** and visit the web interface.
# Bots
Configured using the `bots` top-level property. Bot configuration can override and specify many more options than are available at the operator-level. Many of these can also set the defaults for each subreddit the bot runs:
* Of the subreddits this bot moderates, specify a subset of subreddits to run or exclude from running
* default caching behavior
* control the soft/hard api usage limits
* Flow Control defaults
* Filter Criteria defaults
* default Polling behavior
[Full documentation for all bot instance options can be found in the schema.](https://json-schema.app/view/%23/%23%2Fdefinitions%2FBotInstanceJsonConfig?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FOperatorConfig.json)
## Adding A Bot
If you use the [CM OAuth Helper](/docs/operator/addingBot.md#cm-oauth-helper-recommended) and it works successfully then the configuration for the Bot will be automatically added.
### Manually Adding a Bot
Add a new *object* to the `bots` property at the top-level of your configuration. If `bots` does not exist create it now.
Minimum information required for a valid bot:
* Client Id
* Client Secret
* Refresh Token
* Access Token
<details>
<summary>Example</summary>
```yaml
operator:
name: YourRedditUsername
bots:
- name: u/MyRedditBot # name is optional but highly recommend for readability in both config and web interface
credentials:
reddit:
clientId: f4b4df1c7b2
clientSecret: 34v5q1c56ub
accessToken: 34_f1w1v4
refreshToken: p75_1c467b2
web:
credentials:
clientId: f4b4df1c7b2
clientSecret: 34v5q1c56ub
redirectUri: 'http://localhost:8085/callback'
```
</details>
# Web Client
Configured using the `web` top-level property. Allows specifying settings related to:
* UI port
* Database and caching connection, if different from global settings
* Session max age and secret
* Invite max age
* Connections to CM API instances (if using multiple)
[Full documentation for all web settings can be found in the schema.](https://json-schema.app/view/%23/%23%2Fproperties%2Fweb?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FOperatorConfig.json)
# Example Configurations
## Minimum Config
@@ -119,20 +210,37 @@ Options:
Below are examples of the minimum required config to run the application using all three config approaches independently.
Using **FILE**
<details>
See [Specify File Location](#specify-file-location) for where this file would be located.
YAML (`config.yaml`)
```yaml
operator:
name: YourRedditUsername
web:
credentials:
clientId: f4b4df1c7b2
clientSecret: 34v5q1c56ub
redirectUri: 'http://localhost:8085/callback'
```
JSON (`config.json5`)
```json5
{
"bots": [
{
"credentials": {
"clientId": "f4b4df1c7b2",
"clientSecret": "34v5q1c56ub",
"refreshToken": "34_f1w1v4",
"accessToken": "p75_1c467b2"
}
"operator": {
"name": "YourRedditUsername"
},
"web": {
"credentials": {
"clientId": "f4b4df1c7b2",
"clientSecret": "34v5q1c56ub",
"redirectUri": "http://localhost:8085/callback"
}
]
}
}
```
@@ -143,10 +251,10 @@ Using **ENV** (`.env`)
<details>
```
OPERATOR=YourRedditUsername
CLIENT_ID=f4b4df1c7b2
CLIENT_SECRET=34v5q1c56ub
REFRESH_TOKEN=34_f1w1v4
ACCESS_TOKEN=p75_1c467b2
REDIRECT_URI=http://localhost:8085/callback
```
</details>
@@ -156,7 +264,7 @@ Using **ARG**
<details>
```
node src/index.js run --clientId=f4b4df1c7b2 --clientSecret=34v5q1c56ub --refreshToken=34_f1w1v4 --accessToken=p75_1c467b2
node src/index.js run --clientId=f4b4df1c7b2 --clientSecret=34v5q1c56ub --redirectUri=http://localhost:8085/callback
```
</details>
@@ -166,6 +274,7 @@ node src/index.js run --clientId=f4b4df1c7b2 --clientSecret=34v5q1c56ub --refres
An example of using multiple configuration levels together IE all are provided to the application:
**FILE**
<details>
```json
@@ -176,6 +285,13 @@ An example of using multiple configuration levels together IE all are provided t
}
```
YAML
```yaml
logging:
level: debug
```
</details>
**ENV** (`.env`)
@@ -184,8 +300,6 @@ An example of using multiple configuration levels together IE all are provided t
```
CLIENT_SECRET=34v5q1c56ub
REFRESH_TOKEN=34_f1w1v4
ACCESS_TOKEN=p75_1c467b2
SUBREDDITS=sub1,sub2,sub3
PORT=9008
```
@@ -207,8 +321,6 @@ When all three are used together they produce these variables at runtime for the
```
clientId: f4b4df1c7b2
clientSecret: 34v5q1c56ub
refreshToken: 34_f1w1v4
accessToken: accessToken
subreddits: sub1
port: 9008
log level: debug
@@ -220,6 +332,33 @@ See the [Architecture Docs](/docs/serverClientArchitecture.md) for more informat
<details>
YAML
```yaml
bots:
- credentials:
clientId: f4b4df1c7b2
clientSecret: 34v5q1c56ub
refreshToken: 34_f1w1v4
accessToken: p75_1c467b2
web:
credentials:
clientId: f4b4df1c7b2
clientSecret: 34v5q1c56ub
redirectUri: 'http://localhost:8085/callback'
clients:
# server application running on this same CM instance
- host: 'localhost:8095'
secret: localSecret
# a server application running somewhere else
- host: 'mySecondContextMod.com:8095'
secret: anotherSecret
api:
secret: localSecret
```
JSON
```json5
{
"bots": [
@@ -262,30 +401,8 @@ See the [Architecture Docs](/docs/serverClientArchitecture.md) for more informat
# Cache Configuration
CM implements two caching backend **providers**. By default all providers use `memory`:
See the [Cache Configuration](/docs/operator/caching.md) documentation.
* `memory` -- in-memory (non-persistent) backend
* `redis` -- [Redis](https://redis.io/) backend
# Database Configuration
Each `provider` object in configuration can be specified as:
* one of the above **strings** to use the **defaults settings** or
* an **object** with keys to override default settings
A caching object in the json configuration:
```json5
{
"provider": {
"store": "memory", // one of "memory" or "redis"
"ttl": 60, // the default max age of a key in seconds
"max": 500, // the maximum number of keys in the cache (for "memory" only)
// the below properties only apply to 'redis' provider
"host": 'localhost',
"port": 6379,
"auth_pass": null,
"db": 0,
}
}
```
See the [Database Configuration](/docs/operator/database.md) documentation.

188
docs/operator/database.md Normal file
View File

@@ -0,0 +1,188 @@
# Overview
CM uses a database to store three types of data:
* **Recorded Events** -- an "audit" of how CM processed an Activity (Comment/Submission) and what actions it took based on the result of processing it (comment, report, remove, etc.)
* Persistent Settings/Data
* Settings like the last known state of a Subreddit's bot before the application exited
* Web Client sessions and invites -- stuff that should survive a restart
* Statistics
* All-time and time-series high-level statistics like # of events, # of checks run, etc...
CM does NOT store subreddit configurations or any runtime alterations of these configurations. This is to keep configurations **portable** -- on principle, if you (a moderator) choose to use a different CM instance to run your subreddit's bot then it should not function differently.
# Providers
CM uses [TypeORM](https://typeorm.io/) as the database access layer and specifically supports three database types:
* SQLite -- using either [SQL.js](https://sql.js.org) or native SQLite through [better-sqlite3](https://github.com/JoshuaWise/better-sqlite3)
* MySQL/MariaDB
* Postgres
The database configuration is specified in the top-level `databaseConfig.connection` property in the operator configuration. EX:
```yaml
operator:
name: u/MyRedditAccount
databaseConfig:
connection:
...
```
## SQLite
When using a [local installation](/docs/installation.md#locally) the default database is `sqljs`, which requires no binary dependencies. When using [docker](/docs/operator/installation.md#docker-recommended) the default is `better-sqlite3`.
**NOTE:** It is **NOT RECOMMENDED** to use `sqljs` in a production environment for performance reasons. You should at least switch to `better-sqlite3` or preferably MySql/Postgres.
* [`sqljs` connection options](https://typeorm.io/data-source-options#sqljs-data-source-options)
* [`better-sqlite3` connection options](https://typeorm.io/data-source-options#better-sqlite3-data-source-options)
For both sqlite types, if no database/location is specified, it will be created in the [`DATA_DIR` directory.](/docs/operator/configuration.md#specify-file-location)
If CM detects it cannot **read and write** to the database files, or directory if no files exist, it will fallback to using an in-memory database that will be lost when CM restarts. If you have trouble with r/w permissions and are using docker make sure [file permissions are correct for your mounted volume.](/docs/operator/installation.md#linux-host)
## MySQL/MariaDB
[MySQL/MariaDB connection options](https://typeorm.io/data-source-options#mysql--mariadb-data-source-options)
The `database` you specify should exist before using CM.
## Postgres
[Postgres connection options](https://typeorm.io/data-source-options#postgres--cockroachdb-data-source-options)
The `database` and `schema` you specify should exist before using CM.
# Migrations
CM implements database migrations. On startup it will check for any pending migrations. If the database doesn't exist (sqlite) or is empty or no tables conflict it will automatically execute migrations.
If there is any kind of conflict it will **pause startup** and display a prompt in the user interface to confirm migration execution. **You should always backup your database before running migrations.**
To force CM to always run migrations without confirmation set `force` to `true` in the `migrations` property within `databaseConfig`:
```yaml
databaseConfig:
migrations:
force: true # always run migrations
```
### SQLite
When using a SQLite driver CM can create automatic backups for you. Another prompt will be displayed on the migrations page in the web interface to make a copy of your database. You can make CM automatically backup and continue with migrations like so:
```yaml
databaseConfig:
migrations:
continueOnAutomatedBackup: true # try to backup sqlite files automatically and continue with migrations if successful
```
# Recorded Event Retention
The **Recorded Events** CM stores in the database can be controlled per subreddit. By default events will be stored indefinitely.
A **Retention Policy** is a metric to determine what "range" of Recorded Events CM should keep in the database. It can be either:
* A **number** -- The last X number of Recorded Events will be kept
* EX `1000` -> Keep the last 1000 events and discard any others.
* A **duration** -- A time period, starting from now until X `duration` in the past, for which events will be kept
* EX `3 months` -> Keep all events created between now and 3 months ago. Anything older than 3 months ago will be discarded.
The Retention Policy can be specified at operator level, bot, subreddit *override*, and subreddit configuration level:
```yaml
operator:
name: u/MyRedditAccount
databaseConfig:
retention: '3 months' # each subreddit will retain 3 more of recorded events
bots:
# all subreddits this bot moderates will have 3 month retention
- name: u/OneBotAccount
credentials:
...
subreddits:
overrides:
- name: aSpecialSubredit
databaseConfig:
# overrides retention for THIS SUBBREDIT ONLY, will retain last 1000 events
# -- also overrides any retention set in the subreddit's actual configuration
retention: 1000
- name: u/TwoBotAccount
credentials:
...
databaseConfig:
retention: '1 month' # overrides top-level rentention for all subeddits this bot moderates
```
In a subreddit's config:
```yaml
polling:
- unmoderated
# will retain last 2000 events
# -- will override top-level operator retention or bot-specific retention
# -- will NOT override a subreddit override specified in bot coniguration
retention: 2000
runs:
...
```
# Influx
ContextMod supports writing detailed time-series data to [InfluxDB](https://www.influxdata.com/).
This data can be used to monitor the overall health, performance, and metrics for a ContextMod server. Currently, this data can **only be used by an Operator** as it requires access to the operator configuration and CM instance.
CM supports InfluxDB OSS > 2.3 or InfluxDB Cloud.
**Note:** This is an **advanced feature** and assumes you have enough technical knowledge to follow the documentation provided by each application to deploy and configure them. No support is guaranteed for installation, configuration, or use of Influx and Grafana.
## Supported Metrics
TBA
## Setup
### InfluxDB OSS
* Install [InfluxDB](https://docs.influxdata.com/influxdb/v2.3/install/)
* [Configure InfluxDB using the UI](https://docs.influxdata.com/influxdb/v2.3/install/#set-up-influxdb-through-the-ui)
* You will need **Username**, **Password**, **Organization Name**, and **Bucket Name** later for Grafana setup so make sure to record them somewhere
* [Create a Token](https://docs.influxdata.com/influxdb/v2.3/security/tokens/create-token/) with enough permissions to write/read to the bucket you configured
* After the token is created **view/copy the token** to clipboard by clicking the token name. You will need this for Grafana setup.
### ContextMod
Add the following block to the top-level of your operator configuration:
```yaml
influxConfig:
credentials:
url: 'http://localhost:8086' # URL to your influx DB instance
token: '9RtZ5YZ6bfEXAMPLENJsTSKg==' # token created in the previous step
org: MyOrg # organization created in the previous step
bucket: contextmod # name of the bucket created in the previous step
```
## Grafana
A pre-built dashboard for [Grafana](https://grafana.com) can be imported to display overall metrics/stats using InfluxDB data.
![Grafana Dashboard](/docs/images/grafana.jpg)
* Create a new Data Source using **InfluxDB** type
* Choose **Flux** for the **Query Language**
* Fill in the details for **URL**, **Basic Auth Details** and **InfluxDB Details** using the data you created in the [Influx Setup step](#influxdb-oss)
* Set **Min time interval** to `60s`
* Click **Save and test**
* Import Dashboard
* **Browse** the Dashboard pane
* Click **Import** and **upload** the [grafana dashboard json file](/docs/operator/grafana.json)
* Chose the data source you created from the **InfluxDB CM** dropdown
* Click **Import**
The dashboard can be filtered by **Bots** and **Subreddits** dropdowns at the top of the page to get more specific details.

View File

@@ -0,0 +1,54 @@
This getting started guide is for **Operators** -- that is, someone who wants to run the actual software for a ContentMod bot. If you are a **Moderator** check out the [moderator getting started](/docs/subreddit/gettingStarted.md) guide instead.
# Table of Contents
* [Installation](#installation)
* [Create a Reddit Client](#create-a-reddit-client)
* [Start ContextMod](#start-contextmod)
* [Add a Bot to CM](#add-a-bot-to-cm)
* [Access The Dashboard](#access-the-dashboard)
* [What's Next?](#whats-next)
# Installation
Follow the [installation](/docs/operator/installation.md) documentation. It is recommended to use **Docker** since it is self-contained.
# Create a Reddit Client
[Create a reddit client](/docs/operator/README.md#provisioning-a-reddit-client)
# Start ContextMod
Start CM using the example command from your [installation](#installation) and visit http://localhost:8085
The First Time Setup page will ask you to input:
* Client ID (from [Create a Reddit Client](#create-a-reddit-client))
* Client Secret (from [Create a Reddit Client](#create-a-reddit-client))
* Operator -- this is the username of your main Reddit account.
**Write Config** and then restart CM. You have now created the [minimum configuration](/docs/operator/configuration.md#minimum-configuration) required to run CM.
# Add A Bot to CM
You should automatically be directed to the [Bot Invite Helper](/docs/operator/addingBot.md#cm-oauth-helper-recommended) used to authorize and add a Bot to your CM instance.
Follow the directions here and **create an Authorization Invite** at the bottom of the page.
Next, login to Reddit with the account you will be using as the Bot and then visit the **Authorization Invite** link you created. Follow the steps there to finish adding the Bot to your CM instance.
# Access The Dashboard
Congratulations! You should now have a fully authenticated bot running on a ContextMod instance.
In order for your Bot to operate in a subreddit it **must be a moderator in that subreddit.** This may be your own subreddit or someone else's.
To monitor the behavior of bots running on your instance visit http://localhost:8085.
# What's Next?
As an operator you should familiarize yourself with how the [operator configuration](/docs/operator/configuration.md) you made works. This will help you understand how to get the most of your CM instance by leveraging the [Cache](/docs/oeprator/caching.md) and [Database](/docs/operator/database.md) effectively as well as provide you will all possible options for configuring CM using the [schema.](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FOperatorConfig.json)
If you are also the moderator of the subreddit the bot will be running you should check out the [moderator getting started guide.](/docs/subreddit/gettingStarted.md#setup-wiki-page)
You might also be interested in these [quick tips for using the web interface](/docs/webInterface.md). Additionally, on the dashboard click the **Help** button at the top of the page to get a guided tour of the dashboard.

3148
docs/operator/grafana.json Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,100 @@
# Installation
In order to run a ContextMod instance you must first you must install it somewhere.
ContextMod can be run on almost any operating system but it is recommended to use Docker due to ease of deployment.
## Docker (Recommended)
PROTIP: Using a container management tool like [Portainer.io CE](https://www.portainer.io/products/community-edition) will help with setup/configuration tremendously.
Images available from these registeries:
* [Dockerhub](https://hub.docker.com/r/foxxmd/context-mod) - `docker.io/foxxmd/context-mod`
* [GHCR](https://github.com/foxxmd/context-mod/pkgs/container/context-mod) - `ghcr.io/foxxmd/context-mod`
An example of starting the container using the [minimum configuration](/docs/operator/configuration.md#minimum-config):
* Bind the directory where your config file, logs, and database are located on your host machine into the container's default `DATA_DIR` by using `-v /host/path/folder:/config`
* Note: **You must do this** or else your configuration will be lost next time your container is updated.
* Expose the web interface using the container port `8085`
```
docker run -d -v /host/path/folder:/config -p 8085:8085 ghcr.io/foxxmd/context-mod:latest
```
The location of `DATA_DIR` in the container can be changed by passing it as an environmental variable EX `-e "DATA_DIR=/home/abc/config`
### Linux Host
**NOTE:** If you are using [rootless containers with Podman](https://developers.redhat.com/blog/2020/09/25/rootless-containers-with-podman-the-basics#why_podman_) this DOES NOT apply to you.
If you are running Docker on a **Linux Host** you must specify `user:group` permissions of the user who owns the **configuration directory** on the host to avoid [docker file permission problems.](https://ikriv.com/blog/?p=4698) These can be specified using the [environmental variables **PUID** and **PGID**.](https://docs.linuxserver.io/general/understanding-puid-and-pgid)
To get the UID and GID for the current user run these commands from a terminal:
* `id -u` -- prints UID
* `id -g` -- prints GID
```
docker run -d -v /host/path/folder:/config -p 8085:8085 -e PUID=1000 -e PGID=1000 ghcr.io/foxxmd/context-mod:latest
```
## Locally
Requirements:
* [Typescript](https://www.typescriptlang.org/) >=4.3.5
* [Node](https://nodejs.org) >=16
* [NPM](https://www.npmjs.com/) >=8 (usually bundled with Node)
Clone this repository somewhere and then install from the working directory
```bash
git clone https://github.com/FoxxMD/context-mod.git .
cd context-mod
npm install
tsc -p .
```
An example of running CM using the [minimum configuration](/docs/operator/configuration.md#minimum-config) with a [configuration file](/docs/operator/configuration.md#file-configuration-recommended):
```bash
node src/index.js run
```
## [Heroku Quick Deploy](https://heroku.com/about)
**NOTE:** This is still experimental and requires more testing.
[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://dashboard.heroku.com/new?template=https://github.com/FoxxMD/context-mod)
This template provides a **web** and **worker** dyno for heroku.
* **Web** -- Will run the bot **and** the web interface for ContextMod.
* **Worker** -- Will run **just** the bot.
Be aware that Heroku's [free dyno plan](https://devcenter.heroku.com/articles/free-dyno-hours#dyno-sleeping) enacts some limits:
* A **Web** dyno will go to sleep (pause) after 30 minutes without web activity -- so your bot will ALSO go to sleep at this time
* The **Worker** dyno **will not** go to sleep but you will NOT be able to access the web interface. You can, however, still see how Cm is running by reading the logs for the dyno.
If you want to use a free dyno it is recommended you perform first-time setup (bot authentication and configuration, testing, etc...) with the **Web** dyno, then SWITCH to a **Worker** dyno so it can run 24/7.
# Memory Management
Node exhibits [lazy GC cleanup](https://github.com/FoxxMD/context-mod/issues/90#issuecomment-1190384006) which can result in memory usage for long-running CM instances increasing to unreasonable levels. This problem does not seem to be an issue with CM itself but with Node's GC approach. The increase does not affect CM's performance and, for systems with less memory, the Node *should* limit memory usage based on total available.
In practice CM uses ~130MB for a single bot, single subreddit setup. Up to ~350MB for many (10+) bots or many (20+) subreddits.
If you need to reign in CM's memory usage for some reason this can be addressed by setting an upper limit for memory usage with `node` args by using either:
**--max_old_space_size=**
Value is megabytes. This sets an explicit limit on GC memory usage.
This is set by default in the [Docker](#docker-recommended) container using the env `NODE_ARGS` to `--max_old_space_size=512`. It can be disabled by overriding the ENV.
**--optimize_for_size**
Tells Node to optimize for (less) memory usage rather than some performance optimizations. This option is not memory size dependent. In practice performance does not seem to be affected and it reduces (but not entirely prevents) memory increases over long periods.

Binary file not shown.

Before

Width:  |  Height:  |  Size: 125 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 148 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 226 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 479 KiB

View File

@@ -1,7 +1,6 @@
# Overview
ContextMod's high-level functionality is separated into two **independently run** applications.
ContextMod's high-level functionality is separated into two **independently run** applications.
Each application consists of an [Express](https://expressjs.com/) web server that executes the core logic for that application and communicates via HTTP API calls:
@@ -20,7 +19,7 @@ Communication between the applications is secured using [Json Web Tokens](https:
## Default Mode
**ContextMod is designed to operate in a "monolith" mode by default.**
**ContextMod is designed to operate in a "monolith" mode by default.**
This is done by assuming that when configuration is provided by **environmental variables or CLI arguments** the user's intention is to run the client/server together with only one bot, as if ContextMod is a monolith application. When using these configuration types the same values are parsed to both the server/client to ensure interoperability/transparent usage for the operator. Some examples of this in the **operator configuration**:

95
docs/subreddit/README.md Normal file
View File

@@ -0,0 +1,95 @@
This section is for **reddit moderators**. It covers how to use a CM bot for your subreddit.
If you are trying to run a ContextMod instance (the actual software) please refer to the [operator section](/docs/operator/README.md).
# Table of Contents
* [Overview](#overview)
* [Your Relationship to CM](#your-relationship-to-cm)
* [Operator](#operator)
* [Your Bot](#your-bot)
* [Getting Started](#getting-started)
* [Accessing The Bot](#accessing-the-bot)
* [Editing The Bot](#editing-the-bot)
* [Configuration](#configuration)
* [Guest Access](#guest-access)
# Overview
The Context Mod **software** can manage multiple **bots** (reddit accounts used as bots, like `/u/MyCMBot`). Each bot can manage (run) multiple **subreddits** which is determined by the subreddits the account is a moderator of.
You, the moderator of a subreddit a CM bot runs in, can access/manage the Bot using the CM software's [web interface](/docs/images/subredditStatus.jpg) and control its behavior using the [web editor.](/docs/images/editor.jpg)
## Your Relationship to CM
It is important to understand the relationship between you (the moderator), the bot, and the operator (the person running the CM software).
The easiest way to think about this is in relation to how you use Automoderator and interact with Reddit as a moderator. As an analogy:
### Operator
The operator is the person running the actual server/machine the Context Mod software is on.
They are best thought of as **Reddit:**
* Mostly hands-off when it comes to the bot and interacting with your subreddit
* You must interact with Reddit first before you can use automoderator (login, create a subreddit, etc...)
Unlike reddit, though, there is a greater level of trust required between you and the Operator because what you make the Bot do ultimately affects the Operator since they are the ones actually running your Bot and making API calls to reddit.
### Your Bot
Your bot is like an **invite-only version of Automoderator**:
* Unlike automoderator, you **must** interact with the Operator in order to get the bot working. It is not public for anyone to use.
* Like automoderator, you **must** create a [configuration](/docs/subreddit/components/README.md) for it do anything.
* The bot does not come pre-configured for you. It is a blank slate and requires user input to be useful.
* Also like automoderator, you are **entirely in control of the bot.**
* You can start, stop, and edit its behavior at any time without needing to communicate with the Operator.
* CM provides you _tools_, different ways the Bot can detect patterns in your subreddit/users as well as actions it can, and you can decide to use them however you want.
* Your bot is **only accessible to moderators of your subreddit.**
# Getting Started
The [Getting Started](/docs/subreddit/gettingStarted.md) guide lays out the steps needed to go from nothing to a working Bot. If you are a moderator new to Context Mod this is where you want to begin.
# Accessing The Bot
All bot management and editing is done through the [web interface.](/docs/images/subredditStatus.jpg) The URL used for accessing this interface is given to you by the **Operator** once they have agreed to host your bot/subreddit.
NOTE: This interface is **only access to moderators of your subreddit** and [guests.](#guest-access) You must login to the web interface **with your moderator account** in order to access it.
A **guided tour** that helps show how to manage the bot at a high-level is available on the web interface by clicking the **Help** button in the top-right of the page.
## Editing The Bot
Find the [editor in the web interface](/docs/webInterface.md#editingupdating-your-config) to access the built-in editor for the bot.
[The editor](/docs/images/editor.jpg) should be your all-in-one location for viewing and editing your bot's behavior. **It is equivalent to Automoderator's editor page.**
The editor features:
* syntax validation and highlighting
* configuration auto-complete and documentation (hover over properties)
* built-in validation using Microsoft Word "squiggly lines" indicators and an error list at the bottom of the window
* built-in saving (at the top of the window)
# Configuration
Use the [Configuration Reference](/docs/subreddit/components/README.md) to learn about all the different components available for building a CM configuration.
Additionally, refer to [How It Works](/docs/README.md#how-it-works) and [Core Concepts](/docs/README.md#concepts) to learn the basic of CM configuration.
After you have the basics under your belt you could use the [subreddit-reddit example configurations](/docs/subreddit/components/subredditReady) to familiarize yourself with a complete configuration and ways to use CM.
# Guest Access
CM supports **Guest Access**. Reddit users who are given Guest Access to your bot are allowed to access the web interface even though they are not moderators.
Additionally, they can edit the subreddit's config using the bot. If a Guest edits your config their username will be mentioned in the wiki page edit reason.
Guests can do everything a regular mod can except view/add/remove Guest. They can be removed at any time or set with an expiration date that their access is removed on.
**Guests are helpful if you are new to CM and know reddit users that can help you get started.**
[Add guests from the Subreddit tab in the main interface.](/docs/images/guests.jpg)

View File

@@ -0,0 +1,175 @@
Actions that can submit text (Report, Comment, UserNote, Message, Ban, Submission) will have their `content` values run through a [Mustache Template](https://mustache.github.io/). This means you can insert data generated by Rules into your text before the Action is performed.
See here for a [cheatsheet](https://gist.github.com/FoxxMD/d365707cf99fdb526a504b8b833a5b78) and [here](https://www.tsmean.com/articles/mustache/the-ultimate-mustache-tutorial/) for a more thorough tutorial.
# Template Data
Some data can always be accessed at the top-level. Example
```
This action was run from {{manager}} in Check {{check}}.
The bot intro post is {{botLink}}
Message the moderators of this subreddit using this [compose link]({{modmailLink}})
```
| Name | Description | Example |
|---------------|---------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| `manager` | The name of the subreddit the bot is running in | mealtimevideos |
| `check` | The name of the Check that was triggered | myCheck |
| `botLink` | A link to the bot introduction | https://www.reddit.com/r/ContextModBot/comments/otz396/introduction_to_contextmodbot |
| `modmailLink` | A link that opens reddit's DM compose with the subject line as the Activity being processed | https://www.reddit.com/message/compose?to=/r/mealtimevideos&message=https://www.reddit.com/r/ContextModBot/comments/otz396/introduction_to_contextmodbot |
## Activity Data
**Activity data can be accessed using the `item` variable.** Example
```
This activity is a {{item.kind}} with {{item.votes}} votes, created {{item.age}} ago.
```
Produces:
> This activity is a submission with 10 votes created 5 minutes ago.
### Common
All Actions with `content` have access to this data:
| Name | Description | Example |
|--------------|-----------------------------------------------------------------------------------------------------|----------------------------------------------------------|
| `kind` | The Activity type (submission or comment) | submission |
| `author` | Name of the Author of the Activity being processed | FoxxMD |
| `permalink` | URL to the Activity | https://reddit.com/r/mySuibreddit/comments/ab23f/my_post |
| `votes` | Number of upvotes | 69 |
| `age` | The age of the Activity in a [human friendly format](https://day.js.org/docs/en/durations/humanize) | 5 minutes |
| `subreddit` | The name of the subreddit the Activity is from | mealtimevideos |
| `id` | The `Reddit Thing` ID for the Activity | t3_0tin1 |
| `title` | As comments => the body of the comment. As Submission => title | Test post please ignore |
| `shortTitle` | The same as `title` but truncated to 15 characters | test post pleas... |
### Submissions
If the **Activity** is a Submission these additional properties are accessible:
| Name | Description | Example |
|---------------|-----------------------------------------------------------------|-------------------------|
| `upvoteRatio` | The upvote ratio | 100% |
| `nsfw` | If the submission is marked as NSFW | true |
| `spoiler` | If the submission is marked as a spoiler | true |
| `url` | If the submission was a link then this is the URL for that link | http://example.com |
| `title` | The title of the submission | Test post please ignore |
### Comments
If the **Activity** is a Comment these additional properties are accessible:
| Name | Description | Example |
|------|--------------------------------------------------------------|---------|
| `op` | If the Author is the OP of the Submission this comment is in | true |
### Moderator
If the **Activity** occurred in a Subreddit the Bot moderates these properties are accessible:
| Name | Description | Example |
|---------------|-------------------------------------|---------|
| `reports` | The number of reports recieved | 1 |
| `modReports` | The number of reports by moderators | 1 |
| `userReports` | The number of reports by users | 1 |
## Rule Data
### Summary
A summary of what rules were processed and which were triggered, with results, is available using the `ruleSummary` variable. Example:
```
A summary of rules processed for this activity:
{{ruleSummary}}
```
Would produce:
> A summary of rules processed for this activity:
>
> * namedRegexRule - ✘
> * nameAttributionRule - ✓ - 1 Attribution(s) met the threshold of < 20%, with 1 (3%) of 32 Total -- window: 6 months
> * noXPost ✓ - ✓ 1 of 1 unique items repeated <= 3 times, largest repeat: 1
### Individual
Individual **Rules** can be accessed using the name of the rule, **lower-cased, with all spaces/dashes/underscores.** Example:
```
Submission was repeated {{rules.noxpost.largestRepeat}} times
```
Produces
> Submission was repeated 7 times
## Action Data
### Summary
A summary of what actions have already been run **when the template is rendered** is available using the `actionSummary` variable. It is therefore important that the Action you want to produce the summary is run **after** any other Actions you want to get a summary for.
Example:
```
A summary of actions processed for this activity, so far:
{{actionSummary}}
```
Would produce:
> A summary of actions processed for this activity, so far:
>
> * approve - ✘ - Item is already approved??
> * lock - ✓
> * modnote - ✓ - (SOLID_CONTRIBUTOR) User is good
### Individual
Individual **Actions** can be accessed using the name of the action, **lower-cased, with all spaces/dashes/underscores.** Example:
```
User was banned for {{actions.exampleban.duration}} for {{actions.exampleban.reason}}
```
Produces
> User was banned for 4 days for toxic behavior
# Quick Templating Tutorial
As a quick example for how you will most likely be using templating -- wrapping a variable in curly brackets, `{{variable}}`, will cause the variable value to be rendered instead of the brackets:
```
myVariable = 50;
myOtherVariable = "a text fragment"
template = "This is my template, the variable is {{myVariable}}, my other variable is {{myOtherVariable}}, and that's it!";
console.log(Mustache.render(template, {myVariable});
// will render...
"This is my template, the variable is 50, my other variable is a text fragment, and that's it!";
```
**Note: When accessing an object or its properties you must use dot notation**
```
const item = {
aProperty: 'something',
anotherObject: {
bProperty: 'something else'
}
}
const content = "My content will render the property {{item.aProperty}} like this, and another nested property {{item.anotherObject.bProperty}} like this."
```

View File

@@ -0,0 +1,491 @@
# Table Of Contents
* [Overview](#overview)
* [Lifecycle](#lifecycle)
* Window Components
* [Range](#range)
* [Count](#count)
* [Duration](#duration)
* [Duration String](#duration-string-recommended)
* [Duration Object](#duration-object)
* [Specifying Range](#specifying-range)
* [Specifying Activity Type](#types-of-activities)
* [Filters](#filters)
* [Properties](#filter-properties)
* [Lifecycle](#filter-lifecycle)
* [`pre`](#pre-filter)
* [`post`](#post-filter)
* [More Examples](#more-examples)
* [Count](#count-examples)
* [Duration](#duration-examples)
* [Count And Duration](#count-and-duration-examples)
* [Activity Types](#activity-type-examples)
* [Filter](#filter-examples)
# Overview
An **Activity Window** (`window`) is a group of properties that describe a **range** of [**Activities**](/docs/README.md#activity) to retrieve from Reddit and how to **filter** them.
The main components of an Activity Window:
* **Range** -- How many Activities ([`count`](#count)) or what time period ([`duration`](#duration)) of Activities to fetch
* **Type of Activities** -- When **fetching** from an Author's history, should it return overview (any Activities), just Submissions, or just Comments?
* **Filters** -- How the retrieved Activities should be [filtered](/docs/subreddit/components/README.md#filters) before returning them to a Rule
As an example, if you want to run a **Recent Activity Rule** to check if a user has had activity in /r/mealtimevideos you also need to define what range of activities you want to look at from that user's history.
## Lifecycle
Understanding the lifecycle for how CM uses the Activity Window to retrieve Activities is important to effectively using it.
```mermaid
graph TD
RangReq["Determine Range requirements (Parse `count` and `duration` values)"] --> EmptyList>Create empty Activities List]
EmptyList --> Fetch>"Fetch Activities chunk (1 API Call)"]
Fetch --> PreFilter>Invoke `pre` filter on Activities from chunk]
PreFilter --> Add[Add filtered chunk to Activities list]
Add --> CheckRange>Check Range Satisfaction]
CheckRange -->|`count` Range| CountReq[Are there `count` # of Activities?]
CheckRange -->|`duration` Range| DurationReq[Is the oldest Activity `duration` old?]
CheckRange -->|`count` and `duration` Range| CountDurReq[Is either `count` and/or `duration` satisfied?]
CountReq -->|No| Fetch
DurationReq -->|No| Fetch
CountDurReq -->|No| Fetch
CountReq -->|Yes| FetchDone
DurationReq -->|Yes| FetchDone
CountDurReq -->|Yes| FetchDone
FetchDone[Fetch Complete] --> PostFilter>Invoke `post` filter on all Activities]
PostFilter --> Return>Return Activities to Rule]
click PreFilter href "#pre-filter"
click PostFilter href "#post-filter"
click CountReq href "#count"
click DurationReq href "#duration"
click CountDurReq href "#specifying-range"
```
# Range
There are two types of values that can be used when defining a range:
## Count
This is the **number** of activities you want to retrieve. It's straightforward -- if you want to look at the last 100 activities for a user you can use `100` as the value.
### Shorthand
If **count** is the only property you want to define (leaving everything else as default) then `window` can be defined with just this value:
```yaml
window: 70 # retrieve 70 activities
```
****
Otherwise, use the `count` property on a full `window` object:
```yaml
window:
count: 70 # retrieve 70 activities
...
```
## Duration
A **duration of time** between which all activities will be retrieved. This is a **relative value** that calculates the actual range based on **the duration of time subtracted from when the rule is run.**
For example:
* Today is **July 15th**
* You define a duration of **10 days**
Then the range of activities to be retrieved will be between **July 5th and July 15th** (10 days).
### Shorthand
If a Duration string is the only property you want to define (leaving everything else as default) then `window` can be defined with just this value:
```yaml
window: '9 days' # retrieve last 9 days of activities
```
Otherwise, use the `duration` property on a full `window` object:
```yaml
window:
duration: '9 days'
...
```
### Duration Types
The value used to define the duration can be **one of these two types**:
#### Duration String (recommended)
A string consisting of
* A [Dayjs unit of time](https://day.js.org/docs/en/durations/creating#list-of-all-available-units)
* The value of that unit of time
Examples:
* `9 days`
* `14 hours`
* `80 seconds`
You can ensure your string is valid by testing it [here.](https://regexr.com/61em3)
##### As ISO 8601 duration string
If you're a real nerd you can also use a [standard duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) string.
Examples
* `PT15M` (15 minutes)
Ensure your string is valid by testing it [here.](https://regexr.com/61em9)
#### Duration Object
If you need to specify multiple units of time for your duration you can instead provide a [Dayjs duration **object**](https://day.js.org/docs/en/durations/creating#list-of-all-available-units) consisting of Dayjs unit-values.
```yaml
window:
duration:
days: 4
hours: 6
minutes: 20
```
## Specifying Range
You may use **one or both range properties.**
If both range properties are specified then the value `satisfyOn` determines how the final range is determined
### Using `satisfyOn: any` (default)
If **any** then Activities will be retrieved until one of the range properties is met, **whichever occurs first.**
```yaml
window:
count: 80
duration: 90 days
satisfyOn: any
```
Activities are retrieved in chunks of 100 (or `count`, whichever is smaller)
* If 90 days of activities returns only 40 activities => returns 40 activities
* If 80 activities is only 20 days of range => 80 activities
### Using `satisfyOn: all`
If **all** then both ranges must be satisfied. Effectively, whichever range produces the most Activities will be the one that is used.
```yaml
window:
count: 100
duration: 90 days
satisfyOn: all
```
Activities are retrieved in chunks of 100 (or `count`, whichever is smaller)
* If at 90 days of activities => 40 activities retrieved
* continue retrieving results until 100 activities
* so range is >90 days of activities
* If at 100 activities => 20 days of activities retrieved
* continue retrieving results until 90 days of range
* so results in >100 activities
# Types of Activities
When retrieving an Author's history the window can specify if it should return all Activities (overview), just Comments, or just Submissions, using the `fetch` property:
```yaml
window:
# defaults to 'overview'
fetch: 'submission' # must be one of: overview, comment, submission
```
# Filters
Activity Window can also specify [Item and Subreddit Filters](/docs/subreddit/components/README.md#filters) to filter the Activities retrieved from Reddit before they are returned to a Rule.
Activities can be filtered **during** (`pre`) retrieval or **after** (`post`) retrieval. **When**, during the window **lifecycle**, the Activities are filtered can change the set of Activities returned to a Rule drastically.
## Filter Properties
Regardless of when you are filtering Activities the shape of the filter is the same. Filter properties:
* `subreddits` -- A [Filter Shape](/docs/subreddit/components/README.md#filter-shapes) for filtering by the [Subreddit Criteria](/docs/subreddit/components/README.md#subreddit-filter) of each Activity
* `submissionState` -- A [Filter Shape](/docs/subreddit/components/README.md#filter-shapes) for [Submission Criteria](/docs/subreddit/components/README.md#item-filter). Will run only if filtering a Submission.
* `commentState` -- A [Filter Shape](/docs/subreddit/components/README.md#filter-shapes) for [Comment Criteria](/docs/subreddit/components/README.md#item-filter). Will run only if filtering a Comment.
* `activityState` -- A [Filter Shape](/docs/subreddit/components/README.md#filter-shapes) for either [Submission or Comment Criteria](/docs/subreddit/components/README.md#item-filter). Will run only if `submissionState` or `commentState` is not defined for their respective Activity types.
In this example the filter only returns Activities:
* With a subreddit that
* Is from the subreddit /r/mealtimevideos OR
* Has a name that matches the regex `/ask.*/i` (starts with `ask`) OR
* Is marked as NSFW
* AND if the Activity is a Submission:
* must be self text
* AND if the Activity is a Comment (because `activityState` is defined and `commentState` is not):
* Comment is NOT removed
```yaml
subreddits:
include:
- 'mealtimevideos'
- '/ask.*/i'
- over18: true
submissionState:
include:
- is_self: true
activityState:
exclude:
- removed: false
```
## Filter Lifecycle
Filters can be independently specified for two different "locations" during the window lifecycle using the `filterOn` property.
### `pre` Filter
`pre` will filter Activities from **each API call, before** they are added to the full list of retrieved Activities and, most importantly, **before CM checks if Range conditions have been satisfied.** See the [Lifecycle Diagram](#lifecycle).
This is useful when you want the Range conditions to be applied to an "already filtered" list of Activities -- as opposed to afterwards (like `post`).
#### `max` restriction
However, `pre` introduces the possibility of **near impossible range conditions.**
For example, if you want 200 activities from a subreddit a user has never created activities in then CM would fetch the user's **entire history** before finishing since each chunk of Activities would be filtered to 0.
To prevent this scenario all `pre` filters must also specify a `max` [range](#range) that tell CM:
> if X range of **non-filtered** Activities is reached stop immediately
#### `pre` Example
Let's follow an example scenario where you want the last 200 activities a user has in /r/mealtimevideos
```yaml
window:
count: 200
filterOn:
pre:
subreddits:
include:
- mealtimevideos
max: 400
```
* CM fetches the first chunk of 100 Activities (100 total unfiltered)
* `pre` Filter finds 70 of those 100 are from /r/mealtimevideos => Total Filtered Activities 70
* CM Checks range condition => 70 is less than 200
* CM Checks max condition => 100 unfiltered is less than 400
* CM fetches second chunk of 100 Activities (200 total unfiltered)
* `pre` Filter finds another 70 of those 100 are from /r/mealtimevideos => Total Filtered Activities 140
* CM Checks range condition => 140 is less than 200
* CM Checks max condition => 200 unfiltered is less than 400
* CM fetches third chunk of 100 activities (300 total unfiltered)
* `pre` Filter finds 90 of those 100 are from /r/mealtimevideos => Total Filtered Activities 230
* CM checks range condition => 230 is more than 200
* CM Checks max condition => 300 unfiltered is less than 400
**CM is done fetching activities with 230 Filtered Activities**
Now let's look at the same scenario but `max` is hit:
* CM fetches the first chunk of 100 Activities (100 total unfiltered)
* `pre` Filter finds 10 of those 100 are from /r/mealtimevideos => Total Filtered Activities 10
* CM Checks range condition => 10 is less than 200
* CM Checks max condition => 100 unfiltered is less than 400
* CM fetches second chunk of 100 Activities (200 total unfiltered)
* `pre` Filter finds another 15 of those 100 are from /r/mealtimevideos => Total Filtered Activities 25
* CM Checks range condition => 25 is less than 200
* CM Checks max condition => 200 unfiltered is less than 400
* CM fetches third chunk of 100 activities (300 total unfiltered)
* `pre` Filter finds 5 of those 100 are from /r/mealtimevideos => Total Filtered Activities 30
* CM checks range condition => 30 is less than 200
* CM Checks max condition => 300 unfiltered is less than 400
* CM fetches fourth chunk of 100 activities (400 total unfiltered)
* `pre` Filter finds 0 of those 100 are from /r/mealtimevideos => Total Filtered Activities 30
* CM checks range condition => 30 is less than 200
* CM Checks max condition => 400 unfiltered is NOT less than 400
* CM stops fetching due to max condition hit
**CM is done fetching activities with 30 Filtered Activities**
### `post` Filter
`post` will filter the full list of Activities **after range conditions are satisfied**. This also means it receives the list of Activities filtered by the [`pre` filter](#pre-filter), if one is defined.
The list returned by `post` is what the Rule receives.
#### Example
Let's follow an example scenario where you want to get the last 200 activities from a User's history **and then** return any of those 200 that were made in /r/mealtimevideos -- contrast this wording to the [`pre` example](#pre-example)
```yaml
window:
count: 200
filterOn:
post:
subreddits:
include:
- mealtimevideos
```
* CM fetches the first chunk of 100 Activities (100 total unfiltered)
* CM checks range condition => 100 is less than 200
* CM fetches the second chunk of 100 Activities (200 total unfiltered)
* CM checks range condition => 200 is equal to 200
* CM is done fetching activities with 200 unfiltered Activities
* `post` filter finds 10 of those 200 are from /r/mealtimevideos
* CM returns 10 Activities to the Rule
# More Examples
### Count Examples
#### Get last 100 activities in a User's history
```yaml
window: 100
```
or
```yaml
window:
count: 100
```
### Duration Examples
#### Get last 2 weeks of a User's history
```yaml
window: '2 weeks'
```
or
```yaml
window:
duration: '2 weeks'
```
#### Get last 24 hours and 30 minutes of User's history
```yaml
window:
duration:
hours: 24
minutes: 30
```
### Count and Duration Examples
#### Get EITHER last 6 months or last 200 activities of User's history
Whichever is [satisifed first](#using-satisfyon-any-default)
```yaml
window:
count: 200
duration: '6 months'
```
#### Get AT LEAST the last 6 months and last 200 activities of User's history
Both must be [satisifed](#using-satisfyon-all)
```yaml
window:
count: 200
duration: '6 months'
satisfyOn: 'all'
```
### Activity Type Examples
#### Get the last 200 submissions from User's history
```yaml
window:
count: 200
fetch: 'submission'
```
#### Get the last 200 comments from User's history
```yaml
window:
count: 200
fetch: 'comment'
```
#### Get the last 200 activities (submissions or comments) from User's history
```yaml
window:
count: 200
fetch: 'overview' # or do not include 'fetch' at all, this is the default
```
### Filter Examples
#### Get the all activities from NSFW subreddits in the User's last 200 activities
```yaml
window:
count: 200
filterOn:
post:
subreddits:
include:
- over18: true
```
#### Get the all comments from NSFW subreddits where user is OP, in the User's last 200 activities
```yaml
window:
count: 200
fetch: 'comment'
filterOn:
post:
subreddits:
include:
- over18: true
commentState:
include:
- op: true
```
#### Get the last 200 self-text submissions from a User's history and return any from ask* subreddits
```yaml
window:
count: 200
fetch: 'submission'
filterOn:
pre:
submissionState:
include:
- is_self: true
max: 500
post:
subreddits:
include:
- '/ask.*/i'
```

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,228 @@
Context Mod's behavior after a **Check** has been processed can be configured by a user. This allows a subreddit to control exactly what Runs/Checks will be processed based on the outcome (triggered or not) of a Check.
# Table of Contents
- [Post-Check Properties](#post-check-properties)
* [State](#state)
* [Behavior](#behavior)
+ [Next](#next)
+ [Next Run](#next-run)
+ [Stop](#stop)
+ [Goto](#goto)
- [Goto Syntax](#goto-syntax)
- [Default Behaviors](#default-behaviors)
* [Defining Default Behaviors](#defining-default-behaviors)
- [Examples](#examples)
# Post-Check Properties
## State
When a Check is finished processing it can be in one of two states:
* **Triggered** -- The **Rules** defined in the Check were **triggered** which caused the **Actions** for the Check to be run
* **Failure** -- The **Rules** defined in the check were **not triggered**, based on the conditions that were set (either from the [Check condition](/docs/README.md#Checks) or [Rule Sets](/docs/subreddit/components/advancedConcepts/README.md#Rule-Sets)), and no **Actions** were run
The behavior CM follows is based on which state it is in. The behavior can be specified **by one or both** of these **state properties** on the Check configuration:
* `postTrigger` -- Specifies what behavior to take when the check is **triggered**
* `postFail` -- Specifies what behavior to take when the check is **not triggered**
## Behavior
There are **four** behaviors CM can take. Both/either **state properties** can be defined with **any behavior.**
### Next
The **Next** behavior tells CM to continue to whatever comes *after the Check that was just processed.* This could be another Check or, if this is the last Check in a Run, the next Run.
NOTE: `next` is the **default behavior** for the `postFail` state
Example
```yaml
- name: MyCheck
# ...
postFail: next # if Check is not triggered then CM will start processing AnotherCheck
- name: AnotherCheck
# ...
```
### Next Run
The **Next Run** behavior tells CM to **skip all remaining Checks in the current Run and start processing the next Run in order.**
NOTE: `nextRun` is the **default behavior** for the `postTrigger` state
Example
```yaml
runs:
- name: MyFirstRun
checks:
- name: MyCheck
# ...
postTrigger: nextRun # if Check is triggered then CM will SKIP mySecondCheck and instead start processing MySecondRun
- name: MySecondCheck
# ...
- name: MySecondRun
checks:
- name: FooCheck
# ...
```
### Stop
The **Stop** behavior tells CM to **stop processing the Activity entirely.** This means all remaining Checks and Runs will not be processed.
Example
```yaml
runs:
- name: MyFirstRun
checks:
- name: MyCheck
# ...
postTrigger: stop # if Check is triggered CM will NOT process MySecondCheck OR MySecondRun. The activity is "done" being processed at this point
- name: MySecondCheck
# ...
- name: MySecondRun
checks:
- name: FooCheck
# ...
```
### Goto
The **Goto** behavior is an **advanced** behavior that allows you to specify that CM should "jump to" a specific place in your configuration, regardless of order/location, and continue processing the Activity from there. It can be used to do things like:
* create a loop/iteration to have CM re-process the Activity on an earlier executed part of your configuration because a later part modified the Activity (flaired, etc...)
* use a Check as a simplified *switch statement*
**Goto should be use with care.** If you do not fully understand how this mechanism works you should avoid using it as **most** behaviors can be accomplished using the other behaviors.
As an additional protection **goto depth is limited to 1 by default** which means if a `goto` would be executed more than once during an Activity's lifecycle CM will automatically stop processing that Activity. The `maxGotoDepth` can be raised by the [**Bot Operator**](/docs/operator/gettingStarted.md) per subreddit.
#### Goto Syntax
Location to "jump to" can be specified as:
* **Run** -- `goto:myRunName`
* **Check inside a different Run** -- `goto:myRunName.aCheckInsideTheRun`
* **Check inside the current Run** -- `goto:.myCheck`
Example
```yaml
runs:
- name: MyFirstRun
checks:
- name: FirstCheck
# ...
- name: MyCheck
# ...
postTrigger: 'goto:MyThirdRun' # jump to the run MyThirdRun
postFail: 'goto:MySecondRun.BuzzCheck' # jump to the Check BuzzCheck inside the Run MySecondRun
- name: MySecondRun
checks:
- name: FooCheck
# ...
- name: BuzzCheck
# ...
- name: MyThirdRun
checks:
- name: BarCheck
# ...
```
# Default Behaviors
It is **not required** to define post-Check behavior. CM uses sane defaults to mimic the behavior of automoderator as well as what is "intuitive" when reading a configuration -- that logic flows from top-to-bottom in the order it was defined. For each Check like this:
```yaml
- name: MyCheck
kind: comment
rules:
# ...
actions:
# ...
```
`postTrigger` and `postFail` have default behaviors (mentioned in the sections above) that make the Check end up working like this:
```yaml
- name: MyCheck
kind: comment
rules:
# ...
actions:
# ...
postTrigger: nextRun # check is triggered and actions were performed, skip remaining checks and go to the next Run
postFail: next # check is not triggered and no actions performed, continue to the next check in this Run
```
**So if you are fine with all Checks running in order until one triggered there is no need to define post-Check behaviors at all.**
## Defining Default Behaviors
Defining `postTrigger` and/or `postFail` on a **Run** will set the default behavior for any **Checks** in the Run that **do not have an explicit behavior set.**
```yaml
runs:
- name: MyFirstRun
postTrigger: stop # all Checks without postTrigger defined will have 'stop' as their behavior
checks:
- name: FooCheck # postTrigger is 'stop' since it is not defined
# ...
- name: BarCheck
# ...
postTrigger: next # overrides default behavior
```
# Examples
One **Run** with **default behavior** (no post-Check behavior explicitly defined)
```mermaid
flowchart TB
subgraph spam ["(Run) Spam"]
b1["(Check) self-promotion"] -- "postFail: next" --> b2
b2["(Check) repeat spam"] -- "postFail: next" --> b3
b3["(Check) Good user"]
end
b1 -- "postTrigger: nextRun" --> finish
b2 -- "postTrigger: nextRun" --> finish
b3 -- "postFail: next" --> finish
b3 -- "postTrigger: nextRun" --> finish
finish[Processing Finished]
```
Two **Runs** with **default behavior** (no post-Check behavior explicitly defined)
```mermaid
flowchart TB
subgraph flair ["(Run) Flairing"]
a1["(Check) Flair Submission based on history"]-- "postFail: next" -->a2
a2["(Check) Flair Submission based on user profile"] -- "postFail: next" --> a3
a3["(Check) Flair Submission based on self text"]
end
a1 -- "postTrigger: nextRun" --> b1
a2 -- "postTrigger: nextRun" --> b1
a3 -- "postFail: next" --> b1
a3 -- "postTrigger: nextRun" --> b1
subgraph spam ["(Run) Spam"]
b1["(Check) self-promotion"] -- "postFail: next" -->b2
b2["(Check) repeat spam"] -- "postFail: next" -->b3
b3["(Check) Good user"]
end
b1 -- "postTrigger: nextRun" --> finish
b2 -- "postTrigger: nextRun" --> finish
b3 -- "postFail: next" --> finish
b3 -- "postTrigger: nextRun" --> finish
finish[Processing Finished]
```

View File

@@ -0,0 +1,96 @@
runs:
- name: flairAndCategory
# Runs inherit the same filters as checks/rules/actions
# If these filters fail the Run is skipped and CM processes the next run in order
# authorIs:
# itemIs:
# Set the default behavior for check trigger/fail
# postTrigger:
# postFail:
# Defaults can also be set for check authorIs/itemIs
# same as at operator/subreddit level - any defined here will override "higher" defaults
# filterCriteriaDefaults:
checks:
- name: goodUserFlair
description: flair user if they have decent history in sub
kind: submission
authorIs:
exclude:
- flairText: 'Good User'
rules:
- kind: recentActivity
thresholds:
- threshold: '> 5'
karma: '> 10'
subreddits:
- mySubreddit
actions:
- kind: userflair
text: 'Good User'
# post-behavior after a check has run. Either the check is TRIGGERED or FAIL
# there are 4 possible behaviors for each post-behavior type:
#
# 'next' => Continue to next check in order
# 'nextRun' => Exit the current Run (skip all remaining Checks) and go to the next Run in order
# 'stop' => Exit the current Run and finish activity processing immediately (skip all remaining Runs)
# 'goto:run[.check]' => Specify a run[.check] to jump to. This can be anywhere in your config. CM will continue to process in order from the specified point.
#
# GOTO syntax --
# 'goto:normalFilters' => go to run "normalFilters"
# 'goto:normalFilters.myCheck' => go to run "normalFilters" and start at check "myCheck"
# 'goto:.goodUserFlair' => go to check 'goodUserFlair' IN THE SAME RUN currently processing
#
# this means if the check triggers then continue to 'good submission flair'
postTrigger: next # default is 'nextRun'
# postFail: # default is 'next'
- name: good submission flair
description: flair submission if from good user
kind: submission
authorIs:
include:
- flairText: 'Good User'
actions:
- kind: flair
text: 'Trusted Source'
- kind: approve
# this means if the check is triggered then stop processing the activity entirely
postTrigger: stop
- name: Determine Suspect
checks:
- name: is suspect
kind: submission
rules:
- kind: recentActivity
thresholds:
- subreddits:
- over_18: true
actions:
# do some actions
# if check is triggered then go to run 'suspectFilters'
postTrigger: 'goto:suspectFilters'
# if check is not triggered then go to run 'normalFilters'
postFail: 'goto:normalFilters'
- name: suspectFilters
postTrigger: stop
authorIs:
exclude:
- flairText: 'Good User'
checks:
# some checks for users that are suspicious
- name: normalFilters
authorIs:
exclude:
- flairText: 'Good User'
checks:
# some checks for general activities

View File

@@ -0,0 +1,79 @@
{
"runs": [
{
"checks": [
{
"name": "Auto Remove SP Karma",
"description": "Remove submission because author has self-promo >10% and posted in karma subs recently",
"kind": "submission",
"rules": [
// named rules can be referenced at any point in the configuration (where they occur does not matter)
// and can be used in any Check
// Note: rules do not transfer between subreddit configurations
"freekarmasub",
{
"name": "attr10all",
"kind": "attribution",
"criteria": [
{
"threshold": "> 10%",
"window": "90 days"
},
{
"threshold": "> 10%",
"window": 100
}
],
}
],
"actions": [
{
"kind": "remove"
},
{
"kind": "comment",
"content": "Your submission was removed because you are over reddit's threshold for self-promotion and recently posted this content in a karma sub"
}
]
},
{
"name": "Free Karma On Submission Alert",
"description": "Check if author has posted this submission in 'freekarma' subreddits",
"kind": "submission",
"rules": [
{
// rules can be re-used throughout a configuration by referencing them by name
//
// The rule name itself can only contain spaces, hyphens and underscores
// The value used to reference it will have all of these removed, and lower-cased
//
// so to reference this rule use the value 'freekarmasub'
"name": "Free_Karma-SUB",
"kind": "recentActivity",
"lookAt": "submissions",
"useSubmissionAsReference":true,
"thresholds": [
{
"threshold": ">= 1",
"subreddits": [
"DeFreeKarma",
"FreeKarma4U",
"FreeKarma4You",
"upvote"
]
}
],
"window": "7 days"
}
],
"actions": [
{
"kind": "report",
"content": "Submission posted {{rules.freekarmasub.totalCount}} times in karma {{rules.freekarmasub.subCount}} subs over {{rules.freekarmasub.window}}: {{rules.freekarmasub.subSummary}}"
}
]
},
]
}
]
}

View File

@@ -0,0 +1,53 @@
runs:
- checks:
- name: Auto Remove SP Karma
description: >-
Remove submission because author has self-promo >10% and posted in karma
subs recently
kind: submission
rules:
# named rules can be referenced at any point in the configuration (where they occur does not matter)
# and can be used in any Check
# Note: rules do not transfer between subreddit configurations
- freekarmasub
- name: attr10all
kind: attribution
criteria:
- threshold: '> 10%'
window: 90 days
- threshold: '> 10%'
window: 100
actions:
- kind: remove
- kind: comment
content: >-
Your submission was removed because you are over reddit's threshold
for self-promotion and recently posted this content in a karma sub
- name: Free Karma On Submission Alert
description: Check if author has posted this submission in 'freekarma' subreddits
kind: submission
rules:
# rules can be re-used throughout a configuration by referencing them by name
#
# The rule name itself can only contain spaces, hyphens and underscores
# The value used to reference it will have all of these removed, and lower-cased
#
# so to reference this rule use the value 'freekarmasub'
- name: Free_Karma-SUB
kind: recentActivity
lookAt: submissions
useSubmissionAsReference: true
thresholds:
- threshold: '>= 1'
subreddits:
- DeFreeKarma
- FreeKarma4U
- FreeKarma4You
- upvote
window: 7 days
actions:
- kind: report
content: >-
Submission posted {{rules.freekarmasub.totalCount}} times in karma
{{rules.freekarmasub.subCount}} subs over
{{rules.freekarmasub.window}}: {{rules.freekarmasub.subSummary}}

View File

@@ -0,0 +1,88 @@
{
"runs": [
{
"checks": [
{
"name": "Self Promo All or low comment",
"description": "SP >10% of all activities or >10% of submissions with low comment engagement",
"kind": "submission",
"rules": [
{
// this attribution rule is looking at all activities
//
// we want want this one rule to trigger the check because >10% of all activity (submission AND comments) is a good requirement
"name": "attr10all",
"kind": "attribution",
"criteria": [
{
"threshold": "> 10%",
"window": "90 days"
},
{
"threshold": "> 10%",
"window": 100
}
],
},
{
// this is a **Rule Set**
//
// it is made up of "nested" rules with a pass condition (AND/OR)
// if the nested rules pass the condition then the Rule Set triggers the Check
//
// AND = all nested rules must be triggered to make the Rule Set trigger
// OR = any of the nested Rules will be the Rule Set trigger
"condition": "AND",
// in this check we use an Attribution >10% on ONLY submissions, which is a lower requirement then the above attribution rule
// and combine it with a History rule looking for low comment engagement
// to make a "higher" requirement Rule Set our of two low requirement Rules
"rules": [
{
"name": "attr20sub",
"kind": "attribution",
"criteria": [
{
"threshold": "> 10%",
"thresholdOn": "submissions",
"window": "90 days"
},
{
"threshold": "> 10%",
"thresholdOn": "submissions",
"window": 100
}
],
"lookAt": "media"
},
{
"name": "lowOrOpComm",
"kind": "history",
"criteriaJoin": "OR",
"criteria": [
{
"window": "90 days",
"comment": "< 50%"
},
{
"window": "90 days",
"comment": "> 40% OP"
}
]
}
]
}
],
"actions": [
{
"kind": "remove"
},
{
"kind": "comment",
"content": "Your submission was removed because you are over reddit's threshold for self-promotion or exhibit low comment engagement"
}
]
},
]
}
]
}

View File

@@ -0,0 +1,54 @@
runs:
- checks:
- name: Self Promo All or low comment
description: >-
SP >10% of all activities or >10% of submissions with low comment
engagement
kind: submission
rules:
# this attribution rule is looking at all activities
#
# we want want this one rule to trigger the check because >10% of all activity (submission AND comments) is a good requirement
- name: attr10all
kind: attribution
criteria:
- threshold: '> 10%'
window: 90 days
- threshold: '> 10%'
window: 100
# this is a RULE SET
#
# it is made up of "nested" rules with a pass condition (AND/OR)
# if the nested rules pass the condition then the Rule Set triggers the Check
#
# AND = all nested rules must be triggered to make the Rule Set trigger
# OR = any of the nested Rules will be the Rule Set trigger
- condition: AND
# in this check we use an Attribution >10% on ONLY submissions, which is a lower requirement then the above attribution rule
# and combine it with a History rule looking for low comment engagement
# to make a "higher" requirement Rule Set our of two low requirement Rules
rules:
- name: attr20sub
kind: attribution
criteria:
- threshold: '> 10%'
thresholdOn: submissions
window: 90 days
- threshold: '> 10%'
thresholdOn: submissions
window: 100
lookAt: media
- name: lowOrOpComm
kind: history
criteriaJoin: OR
criteria:
- window: 90 days
comment: < 50%
- window: 90 days
comment: '> 40% OP'
actions:
- kind: remove
- kind: comment
content: >-
Your submission was removed because you are over reddit's threshold
for self-promotion or exhibit low comment engagement

View File

@@ -10,5 +10,5 @@ Consult the [schema](https://json-schema.app/view/%23/%23%2Fdefinitions%2FCheckJ
### Examples
* [Self Promotion as percentage of all Activities](/docs/examples/attribution/redditSelfPromoAll.json5) - Check if Author is submitting much more than they comment.
* [Self Promotion as percentage of Submissions](/docs/examplesm/attribution/redditSelfPromoSubmissionsOnly.json5) - Check if any of Author's aggregated submission origins are >10% of their submissions
* Self Promotion as percentage of all Activities [YAML](/docs/subreddit/componentscomponents/attribution/redditSelfPromoAll.yaml) | [JSON](/docs/subreddit/componentscomponents/attribution/redditSelfPromoAll.json5) - Check if Author is submitting much more than they comment.
* Self Promotion as percentage of Submissions [YAML](/docs/subreddit/components/attribution/redditSelfPromoSubmissionsOnly.yaml) | [JSON](/docs/examplesm/attribution/redditSelfPromoSubmissionsOnly.json5) - Check if any of Author's aggregated submission origins are >10% of their submissions

View File

@@ -0,0 +1,43 @@
{
"runs": [
{
"checks": [
{
"name": "Self Promo Activities",
"description": "Check if any of Author's aggregated submission origins are >10% of entire history",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "attr10all",
"kind": "attribution",
// criteria defaults to OR -- so either of these criteria will trigger the rule
"criteria": [
{
// threshold can be a percent or an absolute number
"threshold": "> 10%",
// The default is "all" -- calculate percentage of entire history (submissions & comments)
// "thresholdOn": "all",
// look at last 90 days of Author's activities (comments and submissions)
"window": "90 days"
},
{
"threshold": "> 10%",
// look at Author's last 100 activities (comments and submissions)
"window": 100
}
],
}
],
"actions": [
{
"kind": "report",
"content": "{{rules.attr10all.largestPercent}}% of {{rules.attr10all.activityTotal}} items over {{rules.attr10all.window}}"
}
]
}
]
}
]
}

View File

@@ -0,0 +1,28 @@
runs:
- checks:
- name: Self Promo Activities
description: >-
Check if any of Author's aggregated submission origins are >10% of entire
history
# check will run on a new submission in your subreddit and look at the Author of that submission
kind: submission
rules:
- name: attr10all
kind: attribution
# criteria defaults to OR -- so either of these criteria will trigger the rule
criteria:
- threshold: '> 10%' # threshold can be a percent or an absolute number
# The default is "all" -- calculate percentage of entire history (submissions & comments)
#thresholdOn: all
#
# look at last 90 days of Author's activities (comments and submissions)
window: 90 days
- threshold: '> 10%'
# look at Author's last 100 activities (comments and submissions)
window: 100
actions:
- kind: report
content: >-
{{rules.attr10all.largestPercent}}% of
{{rules.attr10all.activityTotal}} items over
{{rules.attr10all.window}}

View File

@@ -0,0 +1,25 @@
runs:
- checks:
- name: Self Promo Submissions
description: >-
Check if any of Author's aggregated submission origins are >10% of their
submissions
# check will run on a new submission in your subreddit and look at the Author of that submission
kind: submission
rules:
- name: attr10sub
kind: attribution
# criteria defaults to OR -- so either of these criteria will trigger the rule
criteria:
- threshold: '> 10%' # threshold can be a percent or an absolute number
thresholdOn: submissions # calculate percentage of submissions, rather than entire history (submissions & comments)
window: 90 days # look at last 90 days of Author's activities (comments and submissions)
- threshold: '> 10%'
thresholdOn: submissions
window: 100 # look at Author's last 100 activities (comments and submissions)
actions:
- kind: report
content: >-
{{rules.attr10sub.largestPercent}}% of
{{rules.attr10sub.activityTotal}} items over
{{rules.attr10sub.window}}

View File

@@ -0,0 +1,44 @@
{
"runs": [
{
"checks": [
{
"name": "Self Promo Submissions",
"description": "Check if any of Author's aggregated submission origins are >10% of their submissions",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "attr10sub",
"kind": "attribution",
// criteria defaults to OR -- so either of these criteria will trigger the rule
"criteria": [
{
// threshold can be a percent or an absolute number
"threshold": "> 10%",
// calculate percentage of submissions, rather than entire history (submissions & comments)
"thresholdOn": "submissions",
// look at last 90 days of Author's activities (comments and submissions)
"window": "90 days"
},
{
"threshold": "> 10%",
"thresholdOn": "submissions",
// look at Author's last 100 activities (comments and submissions)
"window": 100
}
],
}
],
"actions": [
{
"kind": "report",
"content": "{{rules.attr10sub.largestPercent}}% of {{rules.attr10sub.activityTotal}} items over {{rules.attr10sub.window}}"
}
]
}
]
}
]
}

View File

@@ -0,0 +1,48 @@
# Author
## Rule
The **Author** rule triggers if any [AuthorCriteria](https://json-schema.app/view/%23%2Fdefinitions%2FAuthorCriteria?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) from a list are either **included** or **excluded**, depending on which property you put them in.
**AuthorCriteria** that can be checked:
* name (u/userName)
* author's subreddit flair text
* author's subreddit flair css
* author's subreddit mod status
* [Toolbox User Notes](/docs/subreddit/componentscomponents/userNotes)
The Author **Rule** is best used in conjunction with other Rules to short-circuit a Check based on who the Author is. It is easier to use a Rule to do this then to write **author filters** for every Rule (and makes Rules more re-useable).
Consult the [schema](https://json-schema.app/view/%23%2Fdefinitions%2FAuthorRuleJSONConfig?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) for a complete reference of the rule's properties.
### Examples
* Basic examples
* Flair new user Submission [YAML](/docs/subreddit/componentscomponents/author/flairNewUserSubmission.yaml) | [JSON](/docs/subreddit/componentscomponents/author/flairNewUserSubmission.json5) - If the Author does not have the `vet` flair then flair the Submission with `New User`
* Flair vetted user Submission [YAML](/docs/subreddit/componentscomponents/author/flairNewUserSubmission.yaml) | [JSON](/docs/subreddit/componentscomponents/author/flairNewUserSubmission.json5) - If the Author does have the `vet` flair then flair the Submission with `Vetted`
* Used with other Rules
* Ignore vetted user [YAML](/docs/subreddit/componentscomponents/author/flairNewUserSubmission.yaml) | [JSON](/docs/subreddit/componentscomponents/author/flairNewUserSubmission.json5) - Short-circuit the Check if the Author has the `vet` flair
## Filter
All **Rules** and **Checks** have an optional `authorIs` property that takes an [AuthorOptions](https://json-schema.app/view/%23%2Fdefinitions%2FAuthorOptions?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) object.
**This property works the same as the Author Rule except that:**
* On **Rules** if all criteria fail the Rule is **skipped.**
* If a Rule is skipped **it does not fail or pass** and so does not affect the outcome of the Check.
* However, if all Rules on a Check are skipped the Check will fail.
* On **Checks** if all criteria fail the Check **fails**.
### Examples
* Skip recent activity check based on author [YAML](/docs/subreddit/componentscomponents/author/authorFilter.yaml) | [JSON](/docs/subreddit/componentscomponents/author/authorFilter.json5) - Skip a Recent Activity check for a set of subreddits if the Author of the Submission has any set of flairs.
## Flair users and submissions
Flair users and submissions based on certain keywords from submitter's profile.
Consult [User Flair schema](https://json-schema.app/view/%23%2Fdefinitions%2FUserFlairActionJson?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) and [Submission Flair schema](https://json-schema.app/view/%23%2Fdefinitions%2FFlairActionJson?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) for a complete reference of the rule's properties.
### Examples
* OnlyFans submissions [YAML](/docs/subreddit/componentscomponents/author/onlyfansFlair.yaml) | [JSON](/docs/subreddit/componentscomponents/author/onlyfansFlair.json5) - Check whether submitter has typical OF keywords in their profile and flair both author + submission accordingly.

Some files were not shown because too many files have changed in this diff Show More