* Render note content to check criteria so templated content isn't compared against rendered content
* Fix cached mod note acted on id to have correct prefix based on thing type
* Fix references mod action not checking note for acted on
* Replace hard-coded polling sources with string constants
* Implement string to PollOn parsing which is case-insensitive and forgives mispelling
* Check that manager specifies only one of each polling source type when build config
* Consolidate ACID check for author history results into authorActivities function so it can be used everywhere
* Remove self check in Recent Activity and used consolidated functionality so that filtering still occurs on current activity
* Generate docs if none found on Web client startup
* Add "Docs" link to local docs index
* Add working implementation of building docs in docker image
* Using project dir copies too much to _site directory and causes TS issues
* In order to keep "home" page in generated docs need to duplicate project README (unfortunately)
* Fix ignoring of filters when they are plain objects
* Fix default filter merging behavior to account for "new" filter structure (named criteria)
* Add tests for building/merging filters
Same issues as a949a4ed10 -- memory provider stores objects in memory (no serialization) so need to check for object instance before trying to reconstruct
* Extract cache related functions into own class and encapsulate pruning/key search there as well
* Refactor SubredditStates to be "re-init"-able if state frequency changes
* Simplify and SubredditSource init and configuration functionality so that class is only created once and then reconfigured if major settings change
* Only reinstantiates stats or cache class based on setting changes
* Remove obsolete maxActionedEvents from operator/subreddit config (from pre-db code)
Same as modActions...
* Add `referencesCurrentActivity` boolean to filter by notes associated with current activity
* Add `note` string property to filter by note content (string or regular expression)
* Replace `allowDuplicates` with `existingNoteCheck` on UserNoteAction to allow for more granular note control on action
* Use keep-alive http agent to reuse open connections
* Decrease max batch size and flush interval to payload sent is smaller
* Add debug logging for fail/success/retry events on flush
* Use "modActions" authorIs filtering to check note prior to adding note using "existingNoteCheck" property
* Refactors concept of "allowDuplicates" to allow any arbitrary modActions test to be used
* Provide convenience ModLogCriteria generation for "existingNoteCheck" based on boolean (emulates allowDuplicates functionality)
* Implement filtering activityType by "false" in order to return actions/notes not added to a specific activity
* Implement "referencesCurrentActivity" property to allow filtering by actions/notes that are associated with the activity being processed
* Implement using "count" for "current" search to enable criteria condition based on presence or non-presence of current action/note passing
* Tie loading indicator to live stream status and display error if one occurs
* Add manual restart action to end of error
* Restart stream automatically if reader ends, up to 3 retries
* Implement a DTO class for activity source to make parts usage (type, identifier) and matching easier
* Implement regex to parse type and identifier from activity source string
* Refactor activity source interface/types to better distinguish as string, data, and class
* Add mhs rule type and MHS credentials interface
* Implement MHS rule with similar criteria options to sentiment
* Allow testing against author history content
* Refactor write/read into separate functions
* Improve error hinting for wiki read/write/permissions WRT mod/oauth permissions
* Remove superfluous error wrapping to reduce logging length for wiki errors
* More debug logging for onboarding process
* Don't return error if manager fails to parse after all onboarding complete (not critical)
* run initHeartbeat on any GET route that render a page so that user doesn't get access denied on initial app load
* Force client refresh if no invite found on initial check for onboarding landing
Reddit returns 403 if the subreddit exists but is private. Using this function wraps the error so we can just get boolean back along with subreddit object, if successful
* Add "default" hint to force val to a url or wiki key if neither is detected but know it should be one of them
* Refactor wiki/url fetching into own functions for better reuseability
* Implement mod permission getter function to check for valid permissions on wiki page error
* Improve error hints on wiki page read failure
* May be a string or array of strings. Passes if any expression matches
* Value may be a convience day-of-week value (mon, tues, wed...)
* or a cron expression
* Refactor some manager/bot methods to be more accessible for invites/wiki
* Add routes and authentication for getting invite information and checking user is moderator of subreddit
* Add route for accept invite and completing onboarding guest/config
* Refactor to use db instead of cache for persisting invites
* Implement subreddit invite helper page
* Add initial config and guests as optional data for invite
* Refactor bot to use db subreddit invite and auto-accept when no config/guests
TODO: subreddit accept page, mod authorization, initial config usage, and documentation
* Refactor/remove websockets functionality for relaying opstats from server with direct polling by client
* Implement delta responses initially introduced in #91 to reduce bandwidth
since 'to' be now be templated a user can configure a message to send to the subreddit the action is being processed from using `to: 'r/{{item.subreddit}}'`
* Add top-level 'actionSummary' template variable that renders a markdown list of action results
* Add individual action result/data, in the same structure as rules, under the top-level 'actions' template variable
* Implement interfaces for template parts
* Refactor Action constructor to take an object for runtime options (cleaner, more extensible)
* Refactor subreddit resource and snoowraputils content rendering and organization to use objects of optional data rather than required arguments
* Make almost all data optional and only parse/render if included
* Move rule results parsing/formatting into own function
* Refactor footer render to use renderContent (DRY)
* Add some requested template data #104
* {{item.subreddit}} #87
* {{check}} #87
* Updated templating documentation
* Create self/link submissions, no reddit media (upload) yet
* Submission can use user modifiers (nsfw, spoiler) and mod modifiers (sticky, distinguish, lock)
* Can create submission in current subreddit (default) or arbitrary subreddit defined in configuration (targets)
* Can create multiple submissions by defining multiple targets
* Check for no/null guest data during invite creation/usage for both client and server side
* Add instances tab on invite page
* Replace instance select on invite page with query string usage
* Redirect from status page to auth helper when instance has no bots
* Rename to Guest since this is more accurate of a description -- "mod" is confusing since user doesn't have any actual mod power
* Add guests as an option to invite creation and display on invite page
It makes more sense for the CM instance that will actually have a bot added to it to own the invite for that bot.
* (BC) Move Invite into server entity mappings and rename to BotInvite
* Add guests and initialConfig (future use)
* BC -- Existing invites need to have instance defined to be used
* BC -- Cache-based invite storage has been REMOVED
* BC -- Removed inviteMaxAge config property (for now)
* Add SubredditInvite entity for future use
* Force/implement server (api) having a defined friendly name. This is used to determine which invites in a DB belong to which server instance
* If not defined in config it is generated at random and then written to config
* Refactor how invites are retrieved and parsed client-side -- CRUD using server api
* BC -- Invite URL structure has changed from ?invite=id to /invite/id...
* Added better UI for migration redirect
* Show all reachable instances in redirect page header
* Error page also shows reachable instances in page header
* Cache subreddit removal reasons and display in popup helper in authenticated config editor
* Add fields for removal reason note/reason in remove action
* Update remove action documentation to include new fields
Using undocumented endpoints pulled from praw documentation/code. Snoowrap can now:
* add removal reason/mod note on a removed activity
* get subreddit removal reasons (used for getting ids for use with removal reason endpoint)
* Trim image to remove arbitrary borders
* Convert image to greyscale in order to reduce effect of saturation differences
* Implement "mirrored" (y-axis flip) hash calculations
* Implement local file fetch and image processing test suite
* Use BotInstance class on client side for easier state tracking and simplifying ACL functions (moved from user to instance)
* Implement and use same interface for client/server Bot representations
* Implement mod/guest context for status and live stats data
* Restrict guest mod CRUD to mods only
Stop or restart real-time data (logs, stats) based on page visibility API so that client does continue to consume data when page is in the background/in a non-visible tab
* Document memory management approaches
* Change node args for docker to a better name (NODE_ARGS)
* Implement default node arg for docker `--max_old_space_size=512`
* Include reddit thing id as 'id'
* Include 'title' -- for submission this is submission title. For comment this is the first 50 characters of the comment truncated with '...'
* Include 'shortTitle' -- same as above but truncated to 15 characters
* More granular delta for nested objects
* Custom delta structure for delayedItems
* Fix abort controller overwrite
* Enforce maximum of two unfocused log streams before cancelling immediately on visible change to reduce concurrent number of requests from browser
* Refactor time parsing and comparison utils to consolidate logic and be more flexible
* Tests for the above
* Fix indexes for ActivityReport (should be unique)
* Fix missing eager load for Activity-Report relationship
* Implement time filtering for reports in itemIs
Snoowrap doesn't add fetch property to objects from listings so even though subreddit may already be fetched it always re-fetches, using an api call. Do a dirty fetch check here to prevent wasting calls when we are sure the subreddit is fetched.
* Potentially fix#30
* Break out api sampling and output into own function
* Add influx metrics for used calls and sample faster
* Fix default tags inheritance
* Refactor config fragment parsing function to always return an array for simpler usage in hydrate function
* Add error handling to all fragment calls so user can get a clear description of which fragment failed, contextually
* Fix incorrect variable used for cache key that prevented cache from working at all (oops)
* Don't add subreddit context to ext url cache key so it can be re-used from any subreddit
* Refactor manager build/init from bot to be independent of bot init process
* Remove/destroy managers for subreddits no longer moderated by bot account or not in operator list
* Add/create managers for subreddits that should be moderated
* Run manager build function during heartbeat to sync managers
* Improve action criteria to FullCriteria function
* Don't include undefined properties
* Iterate entries with switch to simplify property transformations
* Fix mod action test switch case matching to be case-insensitive (same as key)
* Fix missing/bad assignments for filtering mod actions
* Fix typo usage of foundNoteResult in modActions case block
* Throw error if mod action criteria isn't recognized as note/log instead of silently falling back to log
* Test props in order of least likely to use an API call
* Enables simplifying shadowbanned test and allows testing for more properties on shadowbanned user
* Fix existing bot removal
* Return response to client after testing client rather than after managers build to avoid long response time if bot has many subreddits
* Normalize (depopulate from snoowrap) mod note raw data so it can be constructed agnostic of source (cache or api)
* Implement cache GET for modnotes with default TTL of 60 seconds
* Refactor mod note action and implement cache PUT when new notes are added
I was under the impression primary keys were always indexed but that is not the case, at least for postgres. This migration explicitly creates uniques indexes for all tables that use random ids and adds other indexes to filter/premise/result tables on other FK strings. Improves event retrieval timing dramatically.
* Implement separate language detection functionality
* Clearer/simpler sentiment processing
* Add languageHints to help coerce low confidence language detection
* Add test cases for lang detection, sentiment detection, and sentiment tests
* Fix neutral range -- was not using normalized score range
* Build NLP container ad-hoc so only supported languages are included from npm
* Use vader/wink as heuristics for detecting language when content is very short
* Add languageHint option for sentiment config to make coercing a confident sentiment easier
* Refactor lang processing to fail the sentiment test rather than throwing an error when language is not support/not confident -- provides more insight into outcome
typeorm depends on ts-node as an *optional peer* dependency -- ts-node can be used in the typeorm cli to parse entities and run migrations.
However CM doesn't use typeorm CLI for running production so its not needed. And ts-node depends on typescript so npm install --production always installs both at about ~30MB.
I couldn't find a good way to remove peer deps ONLY for typeorm so instead just manually remove the folders from the prod install in the final layer of the docker.
* Refactor to use interval in browser to call to api proxy endpoint and get live stats directly instead of using websockets. Generally simplifies things.
* Remove empty/superfluous data from cache stats returned for live data
Prevents CM from iterating through n+1 pages of polling sources (mostly unmoderated) due to a source-of-truth change
See comments for scenario this helps avoid
* Get streamed logs directly from api (through proxy) in browser using streaming apis instead of through client websockets
* Use observer visibility to determine which logs to stream
* Timeout and abort any streaming logs if tab hasn't been visible for more than 15 seconds
TODO streaming system logs
* Use browserify to include logform functions and triple-beam symbols in client js
* Implement default log transform function in client js
* Remove formatted message and transport data from non-streaming log data sent to client
* Fix missing assignment for filtered activities after removing activity
* Remove 'processing' state from delayed activity lifecycle
* Allows delayed queue to immediately remove activity after pushing to firehose -- simplifies lifecycle since another function (queue) doesn't have to handle this
* Removing delayed activity function call from queue logic reduces calls to database
* Likelihood an activity is cancelled while also processing is small (i hope...)
* Re-order item refresh logic in activity handling so delayed items are fetched before any proxy properties are accessed
* Catch errors on adding delayed activities from DB and just log -- not essential function
Use 'name' since this should *always* be present on a submission/comment -- when snoowrap creates an empty proxy it only includes 'name'.
* Potentially fixes facet of #64
* Fixes shouldRefresh undefined when activity is from delayed activities in database (empty proxy)
* Refactor from async while depending on queue state into interval that always runs and just checks paused status of queue -- eliminates need to restart while loop if queue state is not running
* Add canary debug output for delayed activities to be able to know if it is actually running
* Use duration field as SECONDS and remove additional time-based field (not necessary)
* Refactor dayjs usage in UI to parse action delay duration and correctly display "time until dispatch" to show if duration is negative
* Implement english-only scoring with wink https://github.com/winkjs/wink-sentiment (AFINN, emojis)
* Implement english-only scoring with NLP.js https://github.com/axa-group/nlp.js/blob/master/docs/v3/sentiment-analysis.md (AFINN, Senticon, Pattern)
* Refactor language processing into standalone functions for future use
* Add limited multi-langauge support
* Can run sentiment with NLP.js on english, german, spanish, and french
* Normalize all scores to range between -1 and +1
* Improve score accuracy by averaging all scores
* Add deprecation warnings to rules when building if properties should be migrated to window
* Add `debug` option to window to increase verbosity of filter logging. Default to false.
* Fix object assigment when building hash key for window filters
* Further cleanup for circular dependencies by moving some filter and logging functions into respective files
* Fix list function passed from author activities convenience method
* Move author history caching into main activity fetching function
* Do a better job at rehydrating snoowrap objects from cache data -- set as fetched, substitute relationships for non-fetching objects, and remove listing related objects
* Cache key for results based on window and pre-filter only -- post filter can be done after fetching cached results (Should save api calls!)
* Move some interfaces and types into own files to breakup huge interfaces file
* Refactor window shape for config and "full" usage in app to support subreddit/item filtering
* Refactor author activities into resources class so we can take advantage of caching on subreddit/item filter results
* subreddit filtering on window can now use include or exclude
* subreddit resources uses batch/cache retrieval
* temp fix to keep string subreddit name parity check in recent activity self inclusion logic
Fixes scenario where a dispatched activity does not inherit DR state from currently processing activity
* Add dryrun state to dispatch activity data in app and database
* Use general DR state for dispatched activity rather than DR for dispatched action
* Defer to explicitly defined DR in Task data when manager/queue handles task
* Implement a "not serious" property for these errors so we don't report as an error to manager since they not an actual problem with the api or CM
* Check for [deleted] user name before trying to fetch an author
* Break out documentation into more standalone docs and reorganize into an operator folder
* Remove outdated information on adding bot
* Add additional information on docker install
* Make configuration more opinionated for "recommend" approach
* Add docs on database and caching
* Rewrite operator getting started guide to be more concise
* user-configurable retention period (number of events OR duration) at operator, bot, subreddit override, and subreddit config level
* run database cleanup using retention policy on startup and every 30 minutes
* show retention policy in UI on manager overview
* Extends postBehavior interface to allow specifying different record output options (database, influx)
* Can specify for *either* post behavior which enables storing events that were not triggered
Instead of "skipping" the rule will now fail. This aligns Rule behavior with how filters work through the rest of CM which should reduce cognitive load and development effort.
If the skipping behavior is still desired a user can use a RuleSet with OR condition to achieve the same effect.
* Collect same stats as all time but on at a specified frequency
* Frequency is configurable at operator, bot, and subreddit level
* Operator and bot level frequency can have an enforced minimum
* Query for non-hydrated events to get ids then get fully hydrated objects using only ids -- dramatically improves performance
* Remove typeorm-pagination due to non-optimized count/select approach (should use typeorm getManyAndCount)
* Also removes dup typeorm dependency
* Make title a link to "default" events view
* Fix event link
* Always return first page when fetching events by permalink to reset any existing pagination state
When reference submission is a self post identifier may vary slightly since it is considering both title and body. Use string matching on identifies to find "close" matches for reference submission
* Separate rule results from check result ownership since results can be re-used
* Implement ruleset data structure for better representation of results
* Add ruleset assoication to check result
* Partially refactor check caching (TODO)
* Same behavior/structure as authorIs
* Refactor interfaces to use a generic FilterOption type
* Allow simplified data for item/author filters -- when prop data is an array of criteria treat as "include OR"
* Switching to lsio as base means inclusion of s6-overlay for improved init, handling of default permissions, and a maintained base for security/bug fixes
* Also means we can use PUID and PGID as variables instead of requiring --user in run command (more friendly to unraid, portainer, etc...)
* BREAKING: mount directory for configuration,etc. has been changed to /config by default to align with lsio
* Clear npm cache after final production install to reclaim 100MB on image size
* More comprehensive dockerignore to prevent accidentally including development files
* Use correct property from entity metadata which has all real db prefixes added (tablePath)
* Look for specified metadata table name instead of generic 'migrations'
* Simplify conditions (DRY)
Don't rely on default sort chosen by reddit as comment/overview are sorted by new but submitted is sorted by hot (why??). Make sure requests always sort by new to prevent inconsistencies.
To prevent any accidental referenced object property re-assignment only use subreddit name to construct subreddit for usernotes. Want to make sure notes don't get read/written to different subs by mistake...this shouldn't be happening but just to be double extra sure.
Since web DB needs to be built separately from server to allow migrations anyway might as well provide *all* the flexibility
* Reverts breaking change removing caching from web
* Add database config override in web config
To make persistence align with client-server architecture
* Create separate datasource instances for client and server
* Refactor migration logic into service so code can be reused between client/server
* Implement endpoints for backup/migrate on client
* Move migrations into separate folders and move client sessions, invites, and web settings into own migration
* Defer session secret generation until client instantiation and save to database when possible
* BREAKING: Replace web.caching with web.storage and restrict options to specifying either top-level database or cache
* Simplifies init but provides options for different use cases
* Default to database
* Implement generic storage provider for web persistence. Uses either cache or database
* Refactor session/invites to use storage provider
* Implement invite entity for database
* Allow specifying different storage provider for sessions
* Remove action/rule as standalone entities because they depend too much on premise data to make sense (not normalized). Instead, consolidate into premise entities.
* Breakout premise config into config/itemIs/authorIs data to make future querying easier
* Improve errors when checking if file is readable or directory/file is writable
* Implement custom winston-typeorm logging mappings so migrations are output at INFO level
* Add typeorm logging options to database config and default to displaying error, warning, and migrations
* Ensure directory is writeable before using log rotation transport for winston
* Move init logger into index so its always available
* Simplify printing stack for SimpleError when it's a cause
* Better handling for access permissions when reading operator config file
* Add hint to errors relating to permissions access if app detects its running in docker
* Change project location to /home/node to avoid root permissions when creating WORKDIR
* Create default config directory with correct permissions in the event no volume is mounted
* Set default db driver to better-sqlite3 since we control environment -- it has better performance and we know we can use a pre-built binary b/c of base docker image
* Add DATA_DIR env to allow specifying a "base directory" for all other config paths. Defaults to project root (same as existing behavior)
* OPERATOR_ENV OPERATOR_CONFIG LOG_DIR and database locations can now be specified as absolute OR relative (from DATA_DIR)
* Look for config.yaml and config.json if no OPERATOR_CONFIG is present
* millisecond timestamp using bigint doesn't translate well between databases typeorm/typeorm#2400 so avoid this
* regular int/unix epoch doesn't include milliseconds which we need for ordering results and required resolution
* sqlite and mysql/mariadb support datetime with millisecond precision so settle for that, for now
* Persist events, queue, and manager states to database on change
* Use invokee/state from database on startup to determine if manager should be run after building
* Refactor database init from backup and migration attempt so they can be invoked separately
* Implement more generic db backup function to make additional backup strategies possible in the future
* Add operator config option to try migrations if database backup succeeds
* Implement client web migration runner and database backup flows
* Determine what actual crit object is based on object shape (could be from app or api)
* Initialize filters are undefined so they overwrite null values from api when transforming for ui
* Simplify/refactor stats to only track all-time in database...
* remove "last reload" stats from app and ui
* simplify tracked stats to counts only (no maps-of-types counts)
* Add migration function in stat init to convert any all-time stats in cache to database, when database is empty
* Implement a generic activity source to track where an event was retrieved from and some initial parameters
* Consolidate dispatch action data into generic source
* Refactor delayUtil logic for event handling to prevent blocking worker for a non-trivial amount of time by dispatching event
* Implement own createdAt functionality using hooks and dayjs so we can use unix timestamp to eliminate TZ ambiguity and increase granularity to milliseconds
* hook insert/load to convert timestamp between dayjs and int
* Set value on class instantiation rather than object save to preserve actual creation time
* CM data is highly relational so it doesn't make sense to support both rdbms and document dbs
* repository usage is different for mongo, not supporting both
* Use setter/getter to parse id/name to make it easier to use fullname or short id interchangeably
* rename title to content to reflect property usage better
* Reworked many mappings to fix cascades and relationships
* Add constructors to entities where useful
* Renamed many db entities classes to be different than interface names
* Re-generated initial migration and added convenience npm script for running it
* Added ormconfig for typeorm 3 cli
* Create all run/check/rule/action premises (static info) on manager creation
Makes CM less sensitive to random blips in reddit API by enabling snoowrap to retry some network-issue related error codes alongside status codes
* Add timeoutCodes as an operator configurable array of error codes that is passed to extended snoowrap
* Patch snoowrap to check for timeout codes on request error and use retry logic if found
* Additionally, add retryErrorCodes (status codes) to operator configuration
* Refactor main author filter logic into subredditresources to take advantage of cache provider
* Implement methods to retrieve and cache subreddit moderators and author information
* Refactor live stats to work for "All Subreddits" as well as individual subs
* Refactor live stats to take place of opStats and update almost all bot stats live now (only cache breakdown TODO)
* Refactor opstats to return status of bots/subreddits only for ui indicators in tabs
* Move activity source normalization and verification into own function (thrown on invalid source string)
* Correct source-filter comparison by comparing source to filter rather than other way around to make sure inclusive filter is passed
* Also rename item filter from 'dispatch' to 'dispatched' to match verb tense of other state properties
* Simplify identifier property name in config to just 'identifier' -- there's enough context for what it is already
* Correctly render system logs to html
* Simplify websocket logging so it matches how logs are received o browser from server
* Fix instance redirect name when no friendly is set for api config
Using mocha, chai, and nyc
* tests for parsing string for numeric value comparison
* tests for parsing string for durations and duration comparisons
* tests for parsing reddit entity (subreddit/user) from string
* tests for parsing submission/comment id from reddit permalink string
* tests for initial config parsing/merging
Still can't get nyc to get coverage for everything in src using "all" -- causes reporting to show 0 for everything??
* Add Runs to main docs readme and concepts
* Add high level diagram in main docs readme to show CM lifecycle
* Refactor subreddit/rule examples to use runs syntax
* Was causing uncaught promise rejection in userflairaction because it should have been accessing name instead of id
* Wrap all as/is utility functions where value may be from cache (plain object) or proxy (from snoowrap) with try-catch to prevent any more uncaught promise rejections -- would rather swallow silently (for now) than crash the entire application
* Implement rerun configuration that satisfies requirements from #72
* rerun as action
* optional, user-defined identifier
* cancel rerun as action
* cancel based on re-queued sources
* on existing behavior
* can specify initial goto
* filter item by source (where item was retrieved from for non-cached items)
* filter item by rerun state/identifier
* Add rerun label to event logging
* Add rerun data to actioned event data
* Add id and activity type to event activity data
* Include parent submission activity data if activity is a comment
* Refactor event page ui to simplify headers and move content into collapsible
* Add context to content by including submission context for comments
* Allow FilterResult as a property result
* Remove pre-item testing cache optimization for submissionState to simplify flow
* Helped reduce key count but not worth the cost of overly complex code for returning filter results
* Remove expected prop from results and instead use criteria in filter results to generate this for logs/events
* Refactor log/event generation to handle FilterResult in filter property result
Fixes edge case where a usernote was created by a moderator that no longer mods the sub
* Store mod index so we can recreate note even if moderator is missing
* Refactor moderator hydration on usernote from raw data to just warn if moderator cannot be found
* Only return logs for "default viewed" subreddit/bot when fetching instance status, when specified from QS
* Greatly reduces amount of data fetched and response time
* Return logs with formatted property for non-streaming response
* Implement server live stats endpoint to return subreddit/all stats based on QS
* Use client websocket connection to return stats for currently viewed subreddit
* Move "sorting" log objects into lists for retrieval from server and into bot/managers for each log object type
* Refactor log filtering and aggregration under status/log endpoints to use logs from each entity rather than pulling from server
Reduces complexity in historical log data structures at the expense of slightly more runtime data crunching. The trade-off is well worth it and paves the way for easier retrieval of single/subsets of logs
* Fixes an issue where the cached notes for a user only contain the last added note instead of all notes + new
* Also reduced api calls by caching moderator adding new note instead of calling each time
If the item is not actually removed (it's hard to tell from reddit api) we don't want to prematurely end remove action. Just warn and try to remove anyway
* Refactor events view to show checks within runs
* Build cohesive runs server-side before rendering so user can see all checks in a run together
* Add collapse/expand behavior for activity/run/check with ability to toggle based on triggered state
* Default to collapsing all non-triggered states
* Build check summary on-the-fly instead of storing in event result data
* Store migration state in cache instance
* Migrate on default cache init or private cache init
* Implement first migration to deal with run structure in actioned events
* Add Run and postCheckBehavior config structures to schema and interfaces
* Implement parsing from config and initial flow logic for running on activities in manager
* Implement glob pattern or regex as argument
* Implement scan search for redis for efficiency otherwise iterate keys using generic function
* Implement cache reset based on passed item from action -- reset item crit for activities, author crit for users, and overwrite any cached activity
* Makes error cause easier to see in stack and fixes error now logging during action failure
* Use error with cause for logging action error for clearer stack
* Add Run and postCheckBehavior config structures to schema and interfaces
* Implement parsing from config and initial flow logic for running on activities in manager
* Use same technique as repost rule which has high accuracy and let false-positives
* Implement ability to see similarity score, case sensitivity, and text transformations
* Set running to false when error is caught. Was not caught on last stream refactor which changed polling behavior to end if any error is caught rather than waiting for external source to clear interval
* Add debugging/error messages on polling start/stop
* Use ErrorWithCause so we can get and print a chain of error causes
* Make reddit error response in stack trace more readable by replacing them with a "translated" parent response and add them as the cause
* Properly handle error formatting for winston by looking at shape of log object for error rather than testing instanceof (see comments in errorAwareFormat)
* Fix formatting in web interface for log lines with white-space pre css and properly splitting timestamp from rest of the message
* Implement declaration file for snoowrap errors so they can be imported directly
* Implement logging function to handle boilerplate for known error responses (reddit HTTP response, rate limit, etc.)
* Add properties for file, console, and stream in logging object of operator config
* Each property inherits a (useful) subset of winston transport options
Since snoowrap's WikiPage isn't a "real" object setting it as a property on the class means if it rejects the whole application crashes. Fix this by building wiki proxy every time we need it before awaiting promise for edit/retrieval so that promise scope is bound to the function we are in (that has try-catch)
* Use interface for comparison results at both criteria property level and criteria level
* Implement summary functions to build string results of comparisons
* Output all comparisons to debug and provide summaries to verbose (when applicable)
* Don't just overwrite (duh)
* Drop any default filters that include object keys that are also present in user-defined filters -- this way user-defined always takes precedence on merge
* Add options for /logs endpoint to stream objects instead of strings
* Always return log objects from /status endpoint -- fixes bug where all bots/subreddits got lines from logs that had newlines
* Return context-aware, formatted log lines to client to reduce line length IE if returning to botA -> subA then do not need to include labels for botA,subA #40
* Shorten timestamp to just time and wrap full timestamp in tooltip #40
* Emit log objects to client to reduce parsing complexity (don't have to regex for bot/subreddit name)
* write to config when bot is added
* replace/add based on existing bot
* implement specify instance from instances user is operator of
* implement specify subreddits to run on using comma-separated list
* rewrite invite flow ending to be more clear on results and next steps
* use node-comment and yaml@next to keep comment information intact
* store ast/source version of parsed config for operator
* implement generic yaml/json operator config classes to keep everything organized and simplify marshalling source to js/string
* refactor file parsing and json/yaml parsing to have better single responsibility
* Add excludeCondition to control how exclude sets are tested (and/or)
* Refactor authorIs logic from check/rule/action into standalone function (DRY)
* Simplify filter defaults -- don't need to specify automoderator since it is always a mod
* Add default behavior config to operator and manager config
* Implement configurable behavior when filter is present on check
* Add defaults to exclude mods and automoderator from checks
* Remove unused clearProcessing code
* Use same data structures (Map) for storing polling objects in both Manager and Bot to reduce cognitive load and re-use some logic
* Rename "mod" streams to "shared" streams
* Implement detection and updating of polling when manager config changes
* Implement detection and updating of shared streams on manager config update
* Use shared retry handler for manager polling to better handle general reddit api issues (all polling stops faster)
* Move initial polling buffer into polling object (instead of in manager) for better logic encapsulation and add debug logging for it
* Add more debug logging for manager/bot poll building
* Refactor polling config to use new 'shared' string list of polling sources and deprecate 'sharedMod' property
* Refactor how shared sources are built to look for shared intention in manager polling options before creating
* Implement continuity check for comment/submission polling to ensure no activities are missed
* Add debug logging to polling
* Add abstract user class with auth methods with implementations for client/server
* Refactor client/server logic to use class methods instead of inline auth checks
Closes#71
* Support fetching from reddit wiki
* Support fetching from raw URL
* Support parsing and fetching from gist, github blob, and regexr (very experimental)
* Change manager acquisition so all managers belong to a bot before they start logging so all logs are captured correctly
* Fix log capture logic that prevented all subreddits from being populated
Cache reported items or new comments made by bot for a short time (default to twice polling interval, 1 minute) to prevent bot from running on things it did itself
* Reduce retry for snoowrap to 2 since we do our own error handling in-app and 2 is enough for the occasional, non-systemic blip
* Reduce manager retries
* Fix polling timeout to actually stop on error by simplifying timeout and waiting until response is OK to recreate next timeout call
* Use "unexpected exception" retry count for all non well-known "reddit blip" responses in retry handler rather than failing immediately AND log this distinction
* Fix managers not emitting errors from checks
* Fix bot not awaiting retry handler on manager error emit
* Increase nanny loop delay on error to reduce api pressure when there are many bots running
* (unrelated) set bot as running before starting managers so UI is available earlier
* Implement all entities for actioned events
* Pass global database to subreddit resources
* Read/write actioned events using database, constraining to bot/subreddit
* Add typeorm dependency with backend drivers for sqljs and, optionally, postgres/mongodb/mysql/mariadb
* Add operator configuration structure for global database connection
* Implement config parsing and defaults for sqljs db location or in-memory fallback
#66
* Can flair user on comment/submission
* fix dryrun if-else block (maybe a debugging artifact?)
* allow all properties to be undefined/null/empty and use as intention to unflair user
**Context Mod** (CM) is an event-based, [reddit](https://reddit.com) moderation bot built on top of [snoowrap](https://github.com/not-an-aardvark/snoowrap) and written in [typescript](https://www.typescriptlang.org/).
<img src="/docs/logo.png" align="right"
alt="ContextMod logo" width="180" height="176">
[**Context Mod**](https://contextmod.dev/) (CM) is an event-based, [reddit](https://reddit.com) moderation bot built on top of [snoowrap](https://github.com/not-an-aardvark/snoowrap) and written in [typescript](https://www.typescriptlang.org/).
It is designed to help fill in the gaps for [automoderator](https://www.reddit.com/wiki/automoderator/full-documentation) in regard to more complex behavior with a focus on **user-history based moderation.**
An example of the above that Context Bot can do now:
An example of the above that Context Bot can do:
> * On a new submission, check if the user has also posted the same link in **N** number of other subreddits within a timeframe/# of posts
> * On a new submission or comment, check if the user has had any activity (sub/comment) in **N** set of subreddits within a timeframe/# of posts
>
>In either instance Context Bot can then perform any action a moderator can (comment, report, remove, lock, etc...) against that user, comment, or submission.
Some feature highlights:
* Simple rule-action behavior can be combined to create any level of complexity in behavior
*Server/client architecture
Feature Highlights for **Moderators:**
*Complete bot **autonomy**. YAML config is [stored in your subreddit's wiki](/docs/moderators/gettingStarted.md#setup-wiki-page) (like automoderator)
* Simple rule-action behavior can be combined to create **complex behavior detection**
* Support Activity filtering based on:
* [Author criteria](docs/subreddit-configuration/in-depth/filters/README.md#author-filter) (name, css flair/text, age, karma, moderator status, [Toolbox User Notes](https://www.reddit.com/r/toolbox/wiki/docs/usernotes), and more!)
* State of Subreddit Activity is in [Subreddit](docs/subreddit-configuration/in-depth/filters/README.md#subreddit-filter) (nsfw, name, profile, etc...)
* Rules and Actions support [named references](docs/subreddit-configuration/README.md#named-rules) -- **write once, reference anywhere**
* [Delay/re-process activities](docs/subreddit-configuration/README.md#dispatch) using arbitrary rules
* [**Image Comparisons**](docs/subreddit-configuration/imageComparison.md) via fingerprinting and/or pixel differences
* [**Repost detection**](docs/subreddit-configuration/in-depth/repost) with support for external services (youtube, etc...)
* Event notification via Discord
* [**Web interface**](#web-ui-and-screenshots) for monitoring, administration, and oauth bot authentication
* [**Placeholders**](docs/subreddit-configuration/actionTemplating.md) (like automoderator) can be configured via a wiki page or raw text and supports [mustache](https://mustache.github.io) templating
* [**Partial Configurations**](docs/subreddit-configuration/README.md#partial-configurations) -- offload parts of your configuration to shared locations to consolidate logic between multiple subreddits
* [Guest Access](docs/moderators/README.md#guest-access) enables collaboration and easier setup by allowing temporary access
* [Toxic content prediction](docs/subreddit-configuration/README.md#moderatehatespeechcom-predictions) using [moderatehatespeech.com](https://moderatehatespeech.com) machine learning model
Feature highlights for **Developers and Hosting (Operators):**
* Default/no configuration runs "All In One" behavior
* Additional configuration allows web interface to connect to multiple servers
* Each server instance can run multiple reddit accounts as bots
***Per-subreddit configuration** is handled by YAML (**like automoderator!**) or JSON stored in the subreddit wiki
*Any text-based actions (comment, submission, message, usernotes, ban, etc...) can be configured via a wiki page or raw text and supports [mustache](https://mustache.github.io) [templating](/docs/actionTemplating.md)
*History-based rules support multiple "valid window" types -- [ISO 8601 Durations](https://en.wikipedia.org/wiki/ISO_8601#Durations), [Day.js Durations](https://day.js.org/docs/en/durations/creating), and submission/comment count limits.
*Support Activity skipping based on:
*Author criteria (name, css flair/text, age, karma, moderator status, and [Toolbox User Notes](https://www.reddit.com/r/toolbox/wiki/docs/usernotes))
*Activity state (removed, locked, distinguished, etc.)
*Rules and Actions support named references (write once, reference anywhere)
* [**Image Comparisons**](/docs/imageComparison.md) via fingerprinting and/or pixel differences
* [**Repost detection**](/docs/examples/repost) with support for external services (youtube, etc...)
* Global/subreddit-level **API caching**
* Support for [Toolbox User Notes](https://www.reddit.com/r/toolbox/wiki/docs/usernotes) as criteria or Actions (writing notes)
* Docker container support
* Event notification via Discord
* **Web interface** for monitoring, administration, and oauth bot authentication
*Global/subreddit-level [**caching**](/docs/operator/caching.md) of Reddit APIs responses and CM results
*[Database Persistence](/docs/operator/database.md) using SQLite, MySql, or Postgres
*Audit trails for bot activity
*Historical statistics
*[Docker container](/docs/operator/installation.md#docker-recommended) and [docker-compose](/docs/operator/installation.md#docker-compose) support
*Easy, UI-based [OAuth authentication](/docs/operator/addingBot.md) for adding Bots and moderator dashboard
*Integration with [InfluxDB](https://www.influxdata.com) for detailed [time-series metrics](/docs/operator/database.md#influx) and a pre-built [Grafana](https://grafana.com) [dashboard](/docs/operator/database.md#grafana)
# Table of Contents
@@ -45,7 +63,7 @@ Some feature highlights:
Each subreddit using the RCB bot configures its behavior via their own wiki page.
When a monitored **Event** (new comment/submission, new modqueue item, etc.) is detected the bot runs through a list of **Checks** to determine what to do with the **Activity** from that Event. Each **Check** consists of :
When a monitored **Activity** (new comment/submission, new modqueue item, etc.) is detected the bot runs through a list of [**Checks**](docs/subreddit-configuration/README.md#checks) to determine what to do with the **Activity** from that Event. Each **Check** consists of :
#### Kind
@@ -53,11 +71,11 @@ Is this check for a submission or comment?
#### Rules
A list of **Rule** objects to run against the **Activity**. Triggered Rules can cause the whole Check to trigger and run its **Actions**
A list of [**Rules**](docs/subreddit-configuration/README.md#rules) to run against the **Activity**. Triggered Rules can cause the whole Check to trigger and run its **Actions**
#### Actions
A list of **Action** objects that describe what the bot should do with the **Activity** or **Author** of the activity (comment, remove, approve, etc.). The bot will run **all** Actions in this list.
A list of [**Actions**](docs/subreddit-configuration/README.md#actions) that describe what the bot should do with the **Activity** or **Author** of the activity (comment, remove, approve, etc.). The bot will run **all** Actions in this list.
___
@@ -67,7 +85,7 @@ When an Event occurs all Checks of that type are run in the order they were list
___
[Learn more about the RCB lifecycle and core concepts in the docs.](/docs#how-it-works)
[Learn more about the RCB lifecycle and core concepts in the docs.](/docs/README.md#how-it-works)
## Getting Started
@@ -75,20 +93,20 @@ ___
This guide is for users who want to **run their own bot on a ContextMod instance.**
See the [Operator's Getting Started Guide](/docs/gettingStartedOperator.md)
See the [Operator's Getting Started Guide](/docs/operator/gettingStarted.md)
#### Moderators
This guide is for **reddit moderators** who want to configure an existing CM bot to run on their subreddit.
See the [Moderator's Getting Started Guide](/docs/gettingStartedMod.md)
See the [Moderator's Getting Started Guide](/docs/moderators/gettingStarted.md)
## Configuration and Documentation
Context Bot's configuration can be written in YAML (like automoderator) or [JSON5](https://json5.org/). Its schema conforms to [JSON Schema Draft 7](https://json-schema.org/). Additionally, many **operator** settings can be passed via command line or environmental variables.
* For **operators** (running the bot instance) see the [Operator Configuration](/docs/operatorConfiguration.md) guide
* For **moderators** consult the [app schema and examples folder](/docs/#configuration-and-usage)
* For **operators** (running the bot instance) see the [Operator Configuration](/docs/operator/configuration.md) guide
* For **moderators** consult the [Subreddit Configuration Docs](/docs/subreddit-configuration/README.md)
[**Check the full docs for in-depth explanations of all concepts and examples**](/docs)
@@ -108,19 +126,23 @@ CM comes equipped with a dashboard designed for use by both moderators and bot o
* View **real-time logs** of what the bot is doing on your subreddit
A bot oauth helper allows operators to define oauth credentials/permissions and then generate unique, one-time invite links that allow moderators to authenticate their own bots without operator assistance. [Learn more about using the oauth helper.](docs/botAuthentication.md#cm-oauth-helper-recommended)
A bot oauth helper allows operators to define oauth credentials/permissions and then generate unique, one-time invite links that allow moderators to authenticate their own bots without operator assistance. [Learn more about using the oauth helper.](docs/operator/addingBot.md#cm-oauth-helper-recommended)
Operator view/invite link generation:


Moderator view/invite and authorization:


A similar helper and invitation experience is available for adding **subreddits to an existing bot.**
# Copy config.yaml to /(this directory)/data/config.yaml and then modify to match any changed settings below (see comments on config.yaml)
ports:
- "${CM_WEB-8085}:8085"
environment:
IS_DOCKER:true
# If using a linux host, uncomment these and set them accordingly https://github.com/FoxxMD/context-mod/blob/master/docs/operator/installation.md#linux-host
# PUID: 1000
# PGID: 1000
cache:
image:'redis:7-alpine'
volumes:
# on linux will need to make sure this directory has correct permissions for container to access
- './data/cache:/data'
database:
image:'mariadb:10.9.3'
environment:
MYSQL_ROOT_PASSWORD:CHANGE_THIS
MYSQL_USER:cmuser
# this should match the password set in config.yaml
Review **at least** the **How It Works** and **Concepts** below, then:
* For **Operators** (running a bot instance) refer to [**Operator Getting Started**](/docs/gettingStartedOperator.md) guide
* For **Moderators** (configuring an existing bot for your subreddit) refer to the [**Moderator Getting Started**](/docs/gettingStartedMod.md) guide
* For **Operators** (running a bot instance) refer to [**Operator Getting Started**](operator/gettingStarted.md) guide
* For **Moderators** (configuring an existing bot for your subreddit) refer to the [**Moderator Getting Started**](moderators/gettingStarted.md) guide
## How It Works
Where possible Context Mod (CM) uses the same terminology as, and emulates the behavior, of **automoderator** so if you are familiar with that much of this may seem familiar to you.
### Diagram
Expand the section below for a simplified flow diagram of how CM processes an incoming Activity. Then refer the text description of the diagram below as well as [Concepts](#Concepts) for descriptions of individual components.
<details markdown="block">
<summary>Diagram</summary>

</details>
CM's lifecycle looks like this:
#### 1) A new event in your subreddit is received by CM
#### 1) A new Activity in your subreddit is received by CM
The events CM watches for are configured by you. These can be new modqueue/unmoderated items, submissions, or comments.
The Activities CM watches for are configured by you. These can be new modqueue/unmoderated items, submissions, or comments.
#### 2) CM sequentially processes each Check in your configuration
#### 2) CM sequentially processes each Run in your configuration
A [**Run**](#Runs) is made up of a set of [**Checks**](#Checks)
#### 3) CM sequentially processes each Check in the current Run
A **Check** is a set of:
* One or more **Rules** that define what conditions should **trigger** this Check
* One or more **Actions** that define what the bot should do once the Check is **triggered**
* One or more [**Rules**](#Rule) that define what conditions should **trigger** this Check
* One or more [**Actions**](#Action) that define what the bot should do once the Check is **triggered**
#### 3) Each Check is processed, *in order*, until a Check is triggered
#### 4) Each Check is processed, *in order*, until a Check is **triggered**
Once a Check is **triggered** no more Checks will be processed. This means all subsequent Checks in your configuration (in the order you listed them) are basically skipped.
In CM's default configuration, once a Check is **triggered** no more Checks will be processed. This means all subsequent Checks in this Run (in the order you listed them) are skipped.
#### 4) All Actions from that Check are executed
#### 5) All Actions from the triggered Check are executed
After all Actions are executed CM returns to waiting for the next Event.
After all **Actions** from the triggered **Check** are executed CM begins processing the next **Run**
#### 6) Rinse and Repeat from #3
Until all Runs have been processed.
## Concepts
Core, high-level concepts regarding how CM works.
### Event
An **Event** refers to the [Activity](#activity) (Comment or Submission) CM receives to process as well as the results of processing that Activity.
### Activity
An Activity is a Comment or Submission from Reddit.
### Runs
A **Run** is made up of a set of **Checks** that represent a group of related behaviors the bot should check for or perform -- that are independent of any other behaviors the Bot should perform.
An example of Runs:
* A group of Checks that look for missing flairs on a user or a new submission and flair accordingly
* A group of Checks that detect spam or self-promotion and then remove those activities
Both group of Checks are independent of each other (don't have any patterns or actions in common).
[Full Documentation for Runs](subreddit-configuration/README.md#runs)
### Checks
A **Check** is the main logical unit of behavior for the bot. It is equivalent to "if X then Y". A Check is comprised of:
A **Check** is the main logical unit of behavior for the bot. It is equivalent to "if X then Y". A Check is composed of:
* One or more **Rules** that are tested against an **Activity**
* One of more **Actions** that are performed when the **Rules** are satisfied
The bot's configuration can be made up of one or more **Checks** that are processed **in the order they are listed in the configuration.**
A Run can be made up of one or more **Checks** that are processed **in the order they are listed in the Run.**
Once a Check is **triggered** (its Rules are satisfied and Actions performed) all subsequent Checks are skipped.
Some other important concepts regarding Checks:
* All Checks have a **kind** (defined in the configuration) that determine if they should run on **Submissions** or **Comments**
* Checks have a **condition** property that determines when they are considered **triggered**
* If the **condition** is `AND` then ALL of their **Rules** must be **triggered** for the Check to be **triggered**
* If the **condition** is `OR` then if ANY **Rules** is triggered **triggered** then the Check is **triggered**
Examples of different types of Checks can be found in the [subreddit-ready examples.](/docs/examples/subredditReady)
[Full Documentation for Checks](subreddit-configuration/README.md#checks)
### Rule
A **Rule** is some set of **criteria** (conditions) that are tested against an Activity (comment/submission), a User, or a User's history. A Rule is considered **triggered** when the **criteria** for that rule are found to be **true** for whatever is being tested against.
There are generally three main properties for a Rule:
* **Critiera** -- The conditions/values you want to test for.
* **Activities Window** -- If applicable, the range of activities that the **criteria** will be tested against.
* **Rule-specific options** -- Any number of options that modify how the **criteria** are tested.
CM has different **Rules** that can test against different types of behavior and aspects of a User, their history, and the Activity (submission/common) being checked.
#### Available Rules
Find detailed descriptions of all the Rules, with examples, below:
[Full Documentation for Rules](subreddit-configuration/README.md#rules)
@@ -147,207 +139,28 @@ An **Action** is some action the bot can take against the checked Activity (comm
#### Available Actions
* Remove (Comment/Submission)
* Flair (Submission)
* Ban (User)
* Approve (Comment/Submission)
* Comment (Reply to Comment/Submission)
* Lock (Comment/Submission)
* Report (Comment/Submission)
* [UserNote](/docs/examples/userNotes) (User, when /r/Toolbox is used)
For detailed explanation and options of what individual Actions can do [see the links in the `actions` property in the schema.](https://json-schema.app/view/%23/%23%2Fdefinitions%2FSubmissionCheckJson?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json)
**Checks, Rules, and Actions** all have two additional (optional) criteria "tests". These tests behave differently than rule/check triggers in that:
**Runs, Checks, Rules, and Actions** all have two additional (optional) criteria "pre-tests". These tests are different from rules/checks in these ways:
*When they**pass** the thing being tested continues to process as usual
*When they **fail** the thing being tested **is skipped, not failed.**
*Filters test against the **current state** of the Activity, the Author of the Activity, or the Subreddit of the Activity -- rather than looking at history/context/etc...
*Filter test results only determine if the Run, Check, Rule, or Action **should run** -- rather than triggering it
* When the filter test **passes** the thing being tested continues to process as usual
* When the filter test **fails** the thing being tested **fails**.
For **Checks** and **Actions** skipping means that the thing is not processed. The Action is not run, the Check is not triggered.
In the context of **Rules** (in a Check) skipping means the Rule does not get run BUT it does not fail. The Check will continue processing as if the Rule did not exist. However, if ALL Rules in a Check are skipped then the Check does "fail" (is not triggered).
#### Available Filters
##### Item Filter (`itemIs`)
This filter will test against the **state of the Activity currently being run.** Some criteria available to test against IE "Is the activity...":
* removed
* nsfw
* locked
* stickied
* deleted
* etc...
The `itemIs` filter is made up of an array (list) of `State` criteria objects. **All** criteria in the array must pass for this filter to pass.
There are two different State criteria depending on what type of Activity is being tested:
This filter will test against the **Author of the Activity currently being run.** Some criteria available to test against:
* account age
* comment, link, and total karma
* subreddit flair text/css
* name
* User Notes
* verified
* etc...
The `authorIs` filter is made up two (optional) lists of [`AuthorCriteria`](https://json-schema.app/view/%23/%23%2Fdefinitions%2FSubmissionCheckJson/%23%2Fdefinitions%2FAuthorOptions/%23%2Fdefinitions%2FAuthorCriteria?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) criteria objects that define how the test behaves:
*`include` list -- If **any**`AuthorCriteria` from this list passes then the `authorIs` test passes
*`exclude` list -- If **any**`AuthorCriteria` from this list **does not pass** then the `authorIs` test passes. **Note:** This property is ignored if `include` is also present IE you cannot use both properties at the same time.
Refer to the [app schema for `AuthorCriteria`](https://json-schema.app/view/%23/%23%2Fdefinitions%2FSubmissionCheckJson/%23%2Fdefinitions%2FAuthorOptions/%23%2Fdefinitions%2FAuthorCriteria?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) for all available properties to test against.
Some examples of using `authorIs` can be found in the [Author examples.](/docs/examples/author)
[Full Documentation for Filters](subreddit-configuration/README.md#filters)
## Configuration And Usage
* For **Operator/Bot maintainers** see **[Operation Configuration](/docs/operatorConfiguration.md)**
* For **Operator/Bot maintainers** see **[Operation Guide](operator/README.md)**
* For **Moderators**
*Refer to the [examples folder](/docs/examples) or the [subreddit-ready examples](/docs/examples/subredditReady)
*as well as the [schema editor](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) which has
*Start with the [Subreddit/Moderator docs](moderators/README.md) or [Moderator Getting Started guide](moderators/gettingStarted.md)
*Refer to the [Subreddit Components Documentation](subreddit-configuration) or the [subreddit-ready examples](subreddit-configuration/cookbook)
* as well as the [schema](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) which has
* fully annotated configuration data/structure
* generated examples in json/yaml
* built-in editor that automatically validates your config
## Common Resources
Technical information on recurring, common data/patterns used in CM.
### Activities `window`
Most **Rules** must define the **range of Activities (submissions and/or comments)** that will be used to check the criteria of the Rule. This range is defined wherever you see a `window` property in configuration.
Refer to the [Activities Window](/docs/activitiesWindow.md) documentation for a technical explanation with examples.
### Thresholds and Comparisons
Most rules/filters have criteria that require you to define a specific condition to test against. This can be anything from repeats of activities to account age.
In all of these scenarios the condition is defined using a subset of [comparison operators](https://www.codecademy.com/articles/fwd-js-comparison-logical) (very similar to how automoderator does things).
Available operators:
*`<` -- **less than** => `5 < 6` => 5 is less than 6
*`>` -- **greater than** => `6 > 5` => 6 is greater than 5
*`<=` -- **less than or equal to** => `5 <= 5` => 5 is less than **or equal to** 5
*`>=` -- **greater than or equal to** => `5 >= 5` => 5 is greater than **or equal to** 5
In the context of a rule/filter comparison you provide the comparison **omitting** the value that is being tested. An example...
The RepeatActivity rule has a `threshold` comparison to test against the number of repeat activities it finds
* You want the rule to trigger if it finds **4 or more repeat activities**
* The rule would be configured like this `"threshold": ">= 4"`
Essentially what this is telling the rule is `threshold: "x >= 4"` where `x` is the largest repeat of activities it finds.
#### Other Comparison Types
Other than comparison numeric values there are two other values that can be compared (depending on the criteria)
##### Percentages
Some criteria accept an optional **percentage** to compare against:
```
"threshold": "> 20%"
```
Refer to the individual rule/criteria schema to see what this percentage is comparing against.
##### Durations
Some criteria accept an optional **duration** to compare against:
```
"threshold": "< 1 month"
```
The duration value compares a time range from **now** to `duration value` time in the past.
Refer to [duration values in activity window documentation](/docs/activitiesWindow.md#duration-values) as well as the individual rule/criteria schema to see what this duration is comparing against.
### Image Comparisons
ContextMod implements two methods for comparing **image content**, perceptual hashing and pixel-to-pixel comparisons. Comparisons can be used to filter activities in some activities.
See [image comparison documentation](/docs/imageComparison.md) for a full reference.
## Best Practices
### Named Rules
All **Rules** in a subreddit's configuration can be assigned a **name** that can then be referenced from any other Check.
Create general-use rules so they can be reused and de-clutter your configuration. Additionally, CM will automatically cache the result of a rule so there is a performance and api usage benefit to re-using Rules.
See [ruleNameReuse.json5](/docs/examples/advancedConcepts/ruleNameReuse.json5) for a detailed configuration with annotations.
### Check Order
Checks are run in the order they appear in your configuration, therefore you should place your highest requirement/severe action checks at the top and lowest requirement/moderate actions at the bottom.
This is so that if an Activity warrants a more serious reaction that Check is triggered first rather than having a lower requirement check with less severe actions triggered and causing all subsequent Checks to be skipped.
* Attribution >50% AND Repeat Activity 8x AND Recent Activity in 2 subs => remove submission + ban
* Attribution >20% AND Repeat Activity 4x AND Recent Activity in 5 subs => remove submission + flair user restricted
* Attribution >20% AND Repeat Activity 2x => remove submission
* Attribution >20% AND History comments <30% => remove submission
* Attribution >15% => report
* Repeat Activity 2x => report
* Recent Activity in 3 subs => report
* Author not vetted => flair new user submission
### Rule Order
The ordering of your Rules within a Check/RuleSet can have an impact on Check performance (speed) as well as API usage.
Consider these three rules:
* Rule A -- Recent Activity => 3 subreddits => last 15 submissions
* Rule B -- Repeat Activity => last 3 days
* Rule C -- Attribution => >10% => last 90 days or 300 submissions
The first two rules are lightweight in their requirements -- Rule A can be completed in 1 API call, Rule B potentially completed in 1 Api call.
However, depending on how active the Author is, Rule C will take *at least* 3 API calls just to get all activities (Reddit limit 100 items per call).
If the Check is using `AND` condition for its rules (default) then if either Rule A or Rule B fail then Rule C will never run. This means 3 API calls never made plus the time waiting for each to return.
**It is therefore advantageous to list your lightweight Rules first in each Check.**
### Caching
ContextMod implements caching functionality for:
* author history (`window` criteria in rules)
*`authorIs` results
*`content` that uses wiki pages (on Comment/Report/Ban Actions)
* and User Notes
All of these use api requests so caching them reduces api usage.
Cached results can be re-used if the criteria in configuration is identical to a previously cached result. So...
* author history cache results are re-used if **`window` criteria on a Rule is identical to the `window` on another Rule** IE always use **7 Days** or always use **50 Items** for absolute counts.
*`authorIs` criteria is identical to another `authorIs` elsewhere in configuration..
* etc...
Re-use will result in less API calls and faster Check times.
PROTIP: You can monitor the re-use of cache in the `Cache` section of your subreddit on the web interface. See the tooltips in that section for a better breakdown of cache statistics.
Actions that can submit text (Report, Comment) will have their `content` values run through a [Mustache Template](https://mustache.github.io/). This means you can insert data generated by Rules into your text before the Action is performed.
See here for a [cheatsheet](https://gist.github.com/FoxxMD/d365707cf99fdb526a504b8b833a5b78) and [here](https://www.tsmean.com/articles/mustache/the-ultimate-mustache-tutorial/) for a more thorough tutorial.
All Actions with `content` have access to this data:
```json5
{
item: {
kind: 'string', // the type of item (comment/submission)
author: 'string', // name of the item author (reddit user)
permalink: 'string', // a url to the item
url: 'string', // if the item is a Submission then its URL (external for link type submission, reddit link for self-posts)
title: 'string', // if the item is a Submission, then the title of the Submission,
botLink: 'string' // a link to the bot's FAQ
},
rules: {
// contains all rules that were run and are accessible using the name, lowercased, with all spaces/dashes/underscores removed
}
}
```
The properties of `rules` are accessible using the name, lower-cased, with all spaces/dashes/underscores. If no name is given `kind` is used as `name` Example:
```
"rules": [
{
"name": "My Custom-Recent Activity Rule", // mycustomrecentactivityrule
"kind": "recentActivity"
},
{
// name = repeatsubmission
"kind": "repeatActivity",
}
]
```
**To see what data is available for individual Rules [consult the schema](#configuration) for each Rule.**
#### Quick Templating Tutorial
As a quick example for how you will most likely be using templating -- wrapping a variable in curly brackets, `{{variable}}`, will cause the variable value to be rendered instead of the brackets:
```
myVariable = 50;
myOtherVariable = "a text fragment"
template = "This is my template, the variable is {{myVariable}}, my other variable is {{myOtherVariable}}, and that's it!";
"This is my template, the variable is 50, my other variable is a text fragment, and that's it!";
```
**Note: When accessing an object or its properties you must use dot notation**
```
const item = {
aProperty: 'something',
anotherObject: {
bProperty: 'something else'
}
}
const content = "My content will render the property {{item.aProperty}} like this, and another nested property {{item.anotherObject.bProperty}} like this."
Most **Rules** have a `window` property somewhere within their configuration. This property defines the range of **Activities** (submission and/or comments) that should be retrieved for checking the criteria of the Rule.
As an example if you want to run an **Recent Activity Rule** to check if a user has had activity in /r/mealtimevideos you also need to define what range of activities you want to look at from that user's history.
## `window` property overview (tldr)
The value of `window` can be any of these types:
*`number` count of activities
*`string` [duration](#duration-string-recommended) or [iso 8601](#an-iso-8601-duration-string)
# ActivityWindowCriteria, last 100 activities or 6 weeks of activities (whichever is found first)
window:
count:100
duration:6weeks
```
```json5
// count, last 100 activities
{
"window": 100
}
// duration string, last 10 days
{
"window": "10 days"
}
// duration object, last 2 months and 5 days
{
"window": {
"months": 2,
"days": 5,
}
}
// iso 8601 string, last 15 minutes
{
"window": "PT15M"
}
// ActivityWindowCriteria, last 100 activities or 6 weeks of activities (whichever is found first)
{
"window": {
"count": 100,
"duration": "6 weeks"
}
}
```
</details>
## Types of Ranges
There are two types of values that can be used when defining a range:
### Count
This is the **number** of activities you want to retrieve. It's straightforward -- if you want to look at the last 100 activities for a user you can use `100` as the value.
### Duration
A **duration of time** between which all activities will be retrieved. This is a **relative value** that calculates the actual range based on **the duration of time subtracted from when the rule is run.**
For example:
* Today is **July 15th**
* You define a duration of **10 days**
Then the range of activities to be retrieved will be between **July 5th and July 15th** (10 days).
#### Duration Values
The value used to define the duration can be **any of these three types**:
##### Duration String (recommended)
A string consisting of
* A [Dayjs unit of time](https://day.js.org/docs/en/durations/creating#list-of-all-available-units)
* The value of that unit of time
Examples:
*`9 days`
*`14 hours`
*`80 seconds`
You can ensure your string is valid by testing it [here.](https://regexr.com/61em3)
##### Duration Object
If you need to specify multiple units of time for your duration you can instead provide a [Dayjs duration **object**](https://day.js.org/docs/en/durations/creating#list-of-all-available-units) consisting of Dayjs unit-values.
Example
JSON
```json
{
"days":4,
"hours":6,
"minutes":20
}
```
YAML
```yaml
window:
days:4
hours:6
minutes:20
```
##### An ISO 8601 duration string
If you're a real nerd you can also use a [standard duration](https://en.wikipedia.org/wiki/ISO_8601#Durations)) string.
Examples
*`PT15M` (15 minutes)
Ensure your string is valid by testing it [here.](https://regexr.com/61em9)
## ActivityWindowCriteria
This is an object that lets you specify more granular conditions for your range.
The full object looks like this:
JSON
```json
{
"count":100,
"duration":"10 days",
"satisfyOn":"any",
"subreddits":{
"include":["mealtimevideos","pooptimevideos"],
"exclude":["videos"]
}
}
```
YAML
```yaml
window:
count:100
duration:10days
satisfyOn:any
subreddits:
include:
- mealtimevideos
- pooptimevideos
exclude:
- videos
```
### Specifying Range
You may use **one or both range properties.**
If both range properties are specified then the value `satisfyOn` determines how the final range is determined
#### Using `"satisfyOn": "any"` (default)
If **any** then Activities will be retrieved until one of the range properties is met, **whichever occurs first.**
Example
JSON
```json
{
"count":80,
"duration":"90 days",
"satisfyOn":"any"
}
```
YAML
```yaml
window:
count:80
duration:90days
satisfyOn:any
```
Activities are retrieved in chunks of 100 (or `count`, whichever is smaller)
* If 90 days of activities returns only 40 activities => returns 40 activities
* If 80 activities is only 20 days of range => 80 activities
#### Using `"satisfyOn": "all"`
If **all** then both ranges must be satisfied. Effectively, whichever range produces the most Activities will be the one that is used.
Example
JSON
```json
{
"count":100,
"duration":"90 days",
"satisfyOn":"all"
}
```
YAML
```yaml
window:
count:100
duration:90days
satisfyOn:all
```
Activities are retrieved in chunks of 100 (or `count`, whichever is smaller)
* If at 90 days of activities => 40 activities retrieved
* continue retrieving results until 100 activities
* so range is >90 days of activities
* If at 100 activities => 20 days of activities retrieved
* continue retrieving results until 90 days of range
* so results in >100 activities
### Filtering Activities
You may filter retrieved Activities using an array of subreddits.
**Note:** Activities are filtered **before** range check is made so you will always end up with specified range (but may require more api calls if many activities are filtered out)
#### Include
Use **include** to specify which subreddits should be included from results
Example where only activities from /r/mealtimevideos and /r/modsupport will be returned
JSON
```json
{
"count":100,
"duration":"90 days",
"satisfyOn":"any",
"subreddits":{
"include":["mealtimevideos","modsupport"]
}
}
```
YAML
```yaml
window:
count:100
duruation:90days
satisfyOn:any
subreddits:
include:
- mealtimevideos
- modsupport
```
#### Exclude
Use **exclude** to specify which subreddits should NOT be in the results
Example where activities from /r/mealtimevideos and /r/modsupport will not be returned in results
JSON
```json
{
"count":100,
"duration":"90 days",
"satisfyOn":"any",
"subreddits":{
"exclude":["mealtimevideos","modsupport"]
}
}
```
YAML
```yaml
window:
count:100
duruation:90days
satisfyOn:any
subreddits:
exclude:
- mealtimevideos
- modsupport
```
**Note:**`exclude` will be ignored if `include` is also present.
**Note:** This is for **bot operators.** If you are a subreddit moderator check out the **[Getting Started Guide](/docs/gettingStartedMod.md)**
Before you can start using your bot on reddit there are a few steps you must take:
* Create your bot account IE the reddit account that will be the "bot"
* Create a Reddit application
* Authenticate your bot account with the application
At the end of this process you will have this info:
* clientId
* clientSecret
* refreshToken
* accessToken
* redirectUri
**Note:** If you already have this information you can skip this guide **but make sure your redirect uri is correct if you plan on using the web interface.**
# Table Of Contents
* [Creating an Application](#create-application)
* [Authenticate Your Bot](#authenticate-your-bot-account)
* [Using CM OAuth Helper](#cm-oauth-helper-recommended)
Then open the CM web interface (default is [http://localhost:8085](http://localhost:8085)) and login.
After logging in you should be automatically redirected the auth page. If you are not then visit [http://localhost:8085/auth/helper](http://localhost:8085/auth/helper))
Follow the directions in the helper to create an **auth invite link.** Open this link and then follow the directions to authenticate your bot. At the end of the process you will receive an **Access Token** and **Refresh Token**
## Aardvark OAuth Helper
This method should only be used if you cannot use the [CM OAuth Helper method](#cm-oauth-helper-recommended) because you cannot access the CM web interface.
* Visit [https://not-an-aardvark.github.io/reddit-oauth-helper/](https://not-an-aardvark.github.io/reddit-oauth-helper/) and follow the instructions given.
* **Note:** You will need to update your **redirect uri.**
* Input your **Client ID** and **Client Secret** in the text boxes with those names.
* Choose scopes. **It is very important you check everything on this list or CM may not work correctly**
* edit
* flair
* history
* identity
* modcontributors
* modflair
* modposts
* modself
* mysubreddits
* read
* report
* submit
* wikiread
* wikiedit (if you are using Toolbox User Notes)
* Click **Generate tokens**, you will get a popup asking you to approve access (or login) -- **the account you approve access with is the account that Bot will control.**
* After approving an **Access Token** and **Refresh Token** will be shown at the bottom of the page. Save these to use with CM.
# Provide Credentials to CM
At the end of the last step you chose you should now have this information saved somewhere:
* clientId
* clientSecret
* refreshToken
* accessToken
* redirectUri
This is all the information you need to run your bot with CM.
Using these credentials follow the [operator config guide](/docs/operatorConfiguration.md) to finish setting up your CM instance.
* Ruby 2.5.0 or higher and [RubyGems](https://rubygems.org/pages/download) (usually bundled)
* [Bundler](https://bundler.io/) installed
## Install Doc Dependencies
```bash
npm run docs-install
```
## Serve Docs
```bash
npm run docs-start
```
# Developing/Testing Github Actions
Use [act](https://github.com/nektos/act) to run Github actions locally.
An example secrets file can be found in the project working directory at [act.env.example](../act.env.example)
Modify [push-hook-sample.json](../.github/push-hook-sample.json) to point to the local branch you want to run a `push` event trigger on, then run this command from the project working directory:
Easiest way is to install the [docker container](https://www.mock-server.com/mock_server/running_mock_server.html#pull_docker_image) ([from here](https://hub.docker.com/r/mockserver/mockserver))
Map port `1080:1080` -- acts as both the proxy port and the UI endpoint with the below URL:
```
http(s)://localhost:1080/mockserver/dashboard
```
In your [operator configuration](operator/configuration.md) define a proxy for snoowrap at the top-level:
```yaml
snoowrap:
proxy:'http://localhost:8010'
#debug: true # optionally set debug to true to make snoowrap requests output to log
```
## Usage
### Forwarding Requests (Monitoring Behavior)
This is what will make MockServer act as an actual **proxy server**. In this state CM will operate normally. In the MockServer UI you will be able to monitor all requests/responses made.
```HTTP
PUT/mockserver/expectationHTTP/1.1
Host:localhost:8010
Content-Type:application/json
Content-Length:155
```
<details markdown="block">
<summary>CURL</summary>
```bash
curl --location --request PUT 'http://localhost:8010/mockserver/expectation'\
--header 'Content-Type: application/json'\
--data-raw '{
"httpRequest": {},
"priority": 0,
"httpForward": {
"host": "oauth.reddit.com",
"port": 443,
"scheme": "HTTPS"
}
}'
```
</details>
### Mocking Network Issues
MockServer is a bit confusing and regex'ing for specific paths don't work well (for me??)
The lifecycle of a mock call I do:
* Make sure [forwarding](#forwarding-requests-monitoring-behavior) is set, to begin with
* Breakpoint before the code you want to test with mocking
* [Mock the network issue](#create-network-issue-behavior)
* Once the mock behavior should be "done" then
* [Clear all exceptions](#clearing-behavior)
* Set [forwarding behavior](#forwarding-requests-monitoring-behavior) again
### Create Network Issue Behavior
#### All Responses return 403
<details markdown="block">
<summary>HTTP</summary>
```HTTP
PUT/mockserver/expectationHTTP/1.1
Host:localhost:8010
Content-Type:application/json
Content-Length:1757
```
</details>
<details markdown="block">
<summary>CURL</summary>
```bash
curl --location --request PUT 'http://localhost:8010/mockserver/expectation'\
PROTIP: You can edit/build on examples by using the [schema editor.](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json)
See **Rule Name Reuse Examples [YAML](/docs/examples/advancedConcepts/ruleNameReuse.yaml) | [JSON](/docs/examples/advancedConcepts/ruleNameReuse.json5)**
### Check Order
Checks are run in the order they appear in your configuration, therefore you should place your highest requirement/severe action checks at the top and lowest requirement/moderate actions at the bottom.
This is so that if an Activity warrants a more serious reaction that Check is triggered first rather than having a lower requirement check with less severe actions triggered and causing all subsequent Checks to be skipped.
* Attribution >50% AND Repeat Activity 8x AND Recent Activity in 2 subs => remove submission + ban
* Attribution >20% AND Repeat Activity 4x AND Recent Activity in 5 subs => remove submission + flair user restricted
* Attribution >20% AND Repeat Activity 2x => remove submission
* Attribution >20% AND History comments <30% => remove submission
* Attribution >15% => report
* Repeat Activity 2x => report
* Recent Activity in 3 subs => report
* Author not vetted => flair new user submission
### Rule Sets
The `rules` array on a `Checks` can contain both `Rule` objects and `RuleSet` objects.
A **Rule Set** is a "nested" set of `Rule` objects with a passing condition specified. These allow you to create more complex trigger behavior by combining multiple rules.
See **ruleSets [YAML](/docs/examples/advancedConcepts/ruleSets.yaml) | [JSON](/docs/examples/advancedConcepts/ruleSets.json5)** for a complete example as well as consulting the [schema](https://json-schema.app/view/%23%2Fdefinitions%2FRuleSetJson?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json).
### Rule Order
The ordering of your Rules within a Check/RuleSet can have an impact on Check performance (speed) as well as API usage.
Consider these three rules:
* Rule A -- Recent Activity => 3 subreddits => last 15 submissions
* Rule B -- Repeat Activity => last 3 days
* Rule C -- Attribution => >10% => last 90 days or 300 submissions
The first two rules are lightweight in their requirements -- Rule A can be completed in 1 API call, Rule B potentially completed in 1 Api call.
However, depending on how active the Author is, Rule C will take *at least* 3 API calls just to get all activities (Reddit limit 100 items per call).
If the Check is using `AND` condition for its rules (default) then if either Rule A or Rule B fail then Rule C will never run. This means 3 API calls never made plus the time waiting for each to return.
**It is therefore advantageous to list your lightweight Rules first in each Check.**
### API Caching
Context Mod implements some basic caching functionality for **Author Activities** and wiki pages (on Comment/Report Actions).
**Author Activities** are cached for a subreddit-configurable amount of time (10 seconds by default). A cached activities set can be re-used if the **window on a Rule is identical to the window on another Rule**.
This means that when possible you should re-use window values.
IE If you want to check an Author's Activities for a time range try to always use **7 Days** or always use **50 Items** for absolute counts.
Re-use will result in less API calls and faster Check times.
"description": "Remove submission because author has self-promo >10% and posted in karma subs recently",
"kind": "submission",
"rules": [
// named rules can be referenced at any point in the configuration (where they occur does not matter)
// and can be used in any Check
// Note: rules do not transfer between subreddit configurations
"freekarmasub",
{
"name": "attr10all",
"kind": "attribution",
"criteria": [
{
"threshold": "> 10%",
"window": "90 days"
},
{
"threshold": "> 10%",
"window": 100
}
],
}
],
"actions": [
{
"kind": "remove"
},
{
"kind": "comment",
"content": "Your submission was removed because you are over reddit's threshold for self-promotion and recently posted this content in a karma sub"
}
]
},
{
"name": "Free Karma On Submission Alert",
"description": "Check if author has posted this submission in 'freekarma' subreddits",
"kind": "submission",
"rules": [
{
// rules can be re-used throughout a configuration by referencing them by name
//
// The rule name itself can only contain spaces, hyphens and underscores
// The value used to reference it will have all of these removed, and lower-cased
//
// so to reference this rule use the value 'freekarmasub'
"name": "Free_Karma-SUB",
"kind": "recentActivity",
"lookAt": "submissions",
"useSubmissionAsReference":true,
"thresholds": [
{
"threshold": ">= 1",
"subreddits": [
"DeFreeKarma",
"FreeKarma4U",
"FreeKarma4You",
"upvote"
]
}
],
"window": "7 days"
}
],
"actions": [
{
"kind": "report",
"content": "Submission posted {{rules.freekarmasub.totalCount}} times in karma {{rules.freekarmasub.subCount}} subs over {{rules.freekarmasub.window}}: {{rules.freekarmasub.subSummary}}"
The **Attribution** rule will aggregate an Author's content Attribution (youtube channels, twitter, website domains, etc.) and can check on their totals or percentages of all Activities over a time period:
* Total # of attributions
* As percentage of all Activity or only Submissions
* Look at all domains or only media (youtube, vimeo, etc.)
* Include self posts (by reddit domain) or not
Consult the [schema](https://json-schema.app/view/%23/%23%2Fdefinitions%2FCheckJson/%23%2Fdefinitions%2FAttributionJSONConfig?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) for a complete reference of the rule's properties.
### Examples
* Self Promotion as percentage of all Activities [YAML](/docs/examples/attribution/redditSelfPromoAll.yaml) | [JSON](/docs/examples/attribution/redditSelfPromoAll.json5) - Check if Author is submitting much more than they comment.
* Self Promotion as percentage of Submissions [YAML](/docs/examples/attribution/redditSelfPromoSubmissionsOnly.yaml) | [JSON](/docs/examplesm/attribution/redditSelfPromoSubmissionsOnly.json5) - Check if any of Author's aggregated submission origins are >10% of their submissions
The **History** rule can check an Author's submission/comment statistics over a time period:
* Submission total or percentage of All Activity
* Comment total or percentage of all Activity
* Comments made as OP (commented in their own Submission) total or percentage of all Comments
Consult the [schema](https://json-schema.app/view/%23%2Fdefinitions%2FHistoryJSONConfig?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) for a complete reference of the rule's properties.
### Examples
* Low Comment Engagement [YAML](/docs/examples/history/lowEngagement.yaml) | [JSON](/docs/examples/history/lowEngagement.json5) - Check if Author is submitting much more than they comment.
* OP Comment Engagement [YAML](/docs/examples/history/opOnlyEngagement.yaml) | [JSON](/docs/examples/history/opOnlyEngagement.json5) - Check if Author is mostly engaging only in their own content
The **Recent Activity** rule can check if an Author has made any Submissions/Comments in a list of defined Subreddits.
Consult the [schema](https://json-schema.app/view/%23%2Fdefinitions%2FRecentActivityRuleJSONConfig?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) for a complete reference of the rule's properties.
### Examples
* Free Karma Subreddits [YAML](/docs/examples/recentActivity/freeKarma.yaml) | [JSON](/docs/examples/recentActivity/freeKarma.json5) - Check if the Author has recently posted in any "free karma" subreddits
* Submission in Free Karma Subreddits [YAML](/docs/examples/recentActivity/freeKarmaOnSubmission.yaml) | [JSON](/docs/examples/recentActivity/freeKarmaOnSubmission.json5) - Check if the Author has posted the Submission this check is running on in any "free karma" subreddits recently
"description": "Check if author has posted in 'freekarma' subreddits",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "freekarma",
"kind": "recentActivity",
"useSubmissionAsReference": false,
// when `lookAt` is not present this rule will look for submissions and comments
// lookAt: "submissions"
// lookAt: "comments"
"thresholds": [
{
// for all subreddits, if the number of activities (sub/comment) is equal to or greater than 1 then the rule is triggered
"threshold": ">= 1",
"subreddits": [
"DeFreeKarma",
"FreeKarma4U",
"FreeKarma4You",
"upvote"
]
}
],
// will look at all of the Author's activities in the last 7 days
"window": "7 days"
}
],
"actions": [
{
"kind": "report",
"content": "{{rules.freekarma.totalCount}} activities in karma {{rules.freekarma.subCount}} subs over {{rules.freekarma.window}}: {{rules.freekarma.subSummary}}"
"description": "Check if author has posted this submission in 'freekarma' subreddits",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "freekarmasub",
"kind": "recentActivity",
// rule will only look at Author's submissions in these subreddits
"lookAt": "submissions",
// rule will only look at Author's submissions in these subreddits that have the same content (link) as the submission this event was made on
// In simpler terms -- rule will only check to see if the same link the author just posted is also posted in these subreddits
"useSubmissionAsReference":true,
"thresholds": [
{
// for all subreddits, if the number of activities (sub/comment) is equal to or greater than 1 then the rule is triggered
"threshold": ">= 1",
"subreddits": [
"DeFreeKarma",
"FreeKarma4U",
"FreeKarma4You",
"upvote"
]
}
],
// look at all of the Author's submissions in the last 7 days
"window": "7 days"
}
],
"actions": [
{
"kind": "report",
"content": "Submission posted {{rules.freekarmasub.totalCount}} times in karma {{rules.freekarmasub.subCount}} subs over {{rules.freekarmasub.window}}: {{rules.freekarmasub.subSummary}}"
The **Regex** rule matches on text content from a comment or submission in the same way automod uses regex. The rule, however, provides additional functionality automod does not:
* Can set the **number** of matches that trigger the rule (`matchThreshold`)
Which can then be used in conjunction with a [`window`](https://github.com/FoxxMD/context-mod/blob/master/docs/activitiesWindow.md) to match against activities from the history of the Author of the Activity being checked (including the Activity being checked):
* Can set the **number of Activities** that meet the `matchThreshold` to trigger the rule (`activityMatchThreshold`)
* Can set the **number of total matches** across all Activities to trigger the rule (`totalMatchThreshold`)
* Can set the **type of Activities** to check (`lookAt`)
* When an Activity is a Submission can **specify which parts of the Submission to match against** IE title, body, and/or url (`testOn`)
### Examples
* Trigger if regex matches against the current activity - [YAML](/docs/examples/regex/matchAnyCurrentActivity.yaml) | [JSON](/docs/examples/regex/matchAnyCurrentActivity.json5)
* Trigger if regex matches 5 times against the current activity - [YAML](/docs/examples/regex/matchThresholdCurrentActivity.yaml) | [JSON](/docs/examples/regex/matchThresholdCurrentActivity.json5)
* Trigger if regex matches against any part of a Submission - [YAML](/docs/examples/regex/matchSubmissionParts.yaml) | [JSON](/docs/examples/regex/matchSubmissionParts.json5)
* Trigger if regex matches any of Author's last 10 activities - [YAML](/docs/examples/regex/matchHistoryActivity.yaml) | [JSON](/docs/examples/regex/matchHistoryActivity.json5)
* Trigger if regex matches at least 3 of Author's last 10 activities - [YAML](/docs/examples/regex/matchActivityThresholdHistory.json5) | [JSON](/docs/examples/regex/matchActivityThresholdHistory.json5)
* Trigger if there are 5 regex matches in the Author's last 10 activities - [YAML](/docs/examples/regex/matchTotalHistoryActivity.yaml) | [JSON](/docs/examples/regex/matchTotalHistoryActivity.json5)
* Trigger if there are 5 regex matches in the Author's last 10 comments - [YAML](/docs/examples/regex/matchSubsetHistoryActivity.yaml) | [JSON](/docs/examples/regex/matchSubsetHistoryActivity.json5)
* Remove comments that are spamming discord links - [YAML](/docs/examples/regex/removeDiscordSpam.yaml) | [JSON](/docs/examples/regex/removeDiscordSpam.json5)
* Differs from just using automod because this config can allow one-off/organic links from users who DO NOT spam discord links but will still remove the comment if the user is spamming them
Provided here are **complete, ready-to-go configuration** that can copy-pasted straight into your configuration wiki page to get going with ContextMod immediately.
These configurations attempt to provide sensible, non-destructive, default behavior for some common scenarios and subreddit types.
In most cases these will perform decently out-of-the-box but they are not perfect. You should still monitor bot behavior to see how it performs and will most likely still need to tweak these configurations to get your desired behavior.
All actions for these configurations are non-destructive in that:
* All instances where an activity would be modified (remove/ban/approve) will have `dryRun: true` set to prevent the action from actually being performed
* These instances will also have a `report` action detailing the action would have been performed
**You will have to remove the `report` action and `dryRun` settings yourself.** This is to ensure that you understand the behavior the bot will be performing. If you are unsure of this you should leave them in place until you are certain the behavior the bot is performing is acceptable.
**YAML** is the same format as **automoderator**
## Submission-based Behavior
### Remove submissions from users who have used 'freekarma' subs to bypass karma checks
If the link origin (youtube author, twitter author, etc. or regular domain for non-media links)
* comprises 10% or more of the users **entire** history in the past (100 activities or 6 months)
* or comprises 10% or more of the users **submission** history in the past (100 activities or 6 months) and the user has low engagement (<50% of history is comments or 40%> of comment are as OP)
then remove the submission
## Comment-based behavior
### Remove comment if the user has posted the same comment 4 or more times in a row
This rule goes a step further than automod can by being more discretionary about how it handles this type of spam.
* Remove the comment and **ban a user** if:
* Comment being checked contains **only** a discord link (no other text) AND
* Discord links appear **anywhere** in three or more of the last 10 comments the Author has made
otherwise...
* Remove the comment if:
* Comment being checked contains **only** a discord link (no other text) OR
* Comment contains a discord link **anywhere** AND
* Discord links appear **anywhere** in three or more of the last 10 comments the Author has made
Using these checks ContextMod can more easily distinguish between these use cases for a user commenting with a discord link:
* actual spammers who only spam a discord link
* users who may comment with a link but have context for it either in the current comment or in their history
* users who many comment with a link but it's a one-off event (no other links historically)
Additionally, you could modify both/either of these checks to not remove one-off discord link comments but still remove if the user has a historical trend for spamming links
// remove the line below after confirming behavior is acceptable
"dryRun": true
},
// optionally remove "dryRun" from below if you want to leave a comment on removal
// PROTIP: the comment is bland, you should make it better
{
"kind": "comment",
"content": "Your submission has been removed because you cross-posted it {{rules.xpostlow.largestRepeat}} times and you have very low engagement outside of making submissions",
// remove the line below after confirming behavior is acceptable
"dryRun": true
},
// optionally remove "dryRun" from below if you want to leave a comment on removal
// PROTIP: the comment is bland, you should make it better
{
"kind": "comment",
"content": "Your submission has been removed because you cross-posted it {{rules.xpostlow.largestRepeat}} times and you have very low engagement outside of making submissions",
"distinguish": true,
"dryRun": true
}
]
},
{
//
// Remove submissions from users who have recent activity in freekarma subs within the last 50 activities or 6 months (whichever is less)
//
"name": "freekarma removal",
"description": "Remove submission if user has used freekarma sub recently",
"kind": "submission",
"itemIs": [
{
"removed": false
}
],
"condition": "AND",
"rules": [
{
"name": "freekarma",
"kind": "recentActivity",
"window": {
"count": 50,
"duration": "6 months"
},
"useSubmissionAsReference": false,
"thresholds": [
{
"subreddits": [
"FreeKarma4U",
"FreeKarma4You",
"KarmaStore",
"promote",
"shamelessplug",
"upvote"
]
}
]
}
],
"actions": [
// remove this after confirming behavior is acceptable
{
"kind": "report",
"content": "Remove=> {{rules.newtube.totalCount}} activities in freekarma subs"
},
//
//
{
"kind": "remove",
// remove the line below after confirming behavior is acceptable
"dryRun": true
},
// optionally remove "dryRun" from below if you want to leave a comment on removal
// PROTIP: the comment is bland, you should make it better
{
"kind": "comment",
"content": "Your submission has been removed because you have recent activity in 'freekarma' subs",
// Remove a SUBMISSION if the link comprises more than or equal to 10% of users history (100 activities or 6 months) OR
//
// if link comprises 10% of submission history (100 activities or 6 months)
// AND less than 50% of their activity is comments OR more than 40% of those comments are as OP (in the own submissions)
//
"name": "Self-promo all AND low engagement",
"description": "Self-promo is >10% for all or just sub and low comment engagement",
"kind": "submission",
"condition": "OR",
"rules": [
{
"name": "attr",
"kind": "attribution",
"criteria": [
{
"threshold": ">= 10%",
"window": {
"count": 100,
"duration": "6 months"
},
"domains": [
"AGG:SELF"
]
}
],
},
{
"condition": "AND",
"rules": [
{
"name": "attrsub",
"kind": "attribution",
"criteria": [
{
"threshold": ">= 10%",
"thresholdOn": "submissions",
"window": {
"count": 100,
"duration": "6 months"
},
"domains": [
"AGG:SELF"
]
}
]
},
{
"name": "lowOrOpComm",
"kind": "history",
"criteriaJoin": "OR",
"criteria": [
{
"window": {
"count": 100,
"duration": "6 months"
},
"comment": "< 50%"
},
{
"window": {
"count": 100,
"duration": "6 months"
},
"comment": "> 40% OP"
}
]
}
]
}
],
"actions": [
{
"kind": "report",
"content": "{{rules.attr.largestPercent}}{{rules.attrsub.largestPercent}} of {{rules.attr.activityTotal}}{{rules.attrsub.activityTotal}} items ({{rules.attr.window}}{{rules.attrsub.window}}){{#rules.loworopcomm.thresholdSummary}} => {{rules.loworopcomm.thresholdSummary}}{{/rules.loworopcomm.thresholdSummary}}"
},
//
//
{
"kind": "remove",
// remove the line below after confirming behavior is acceptable
"dryRun": true
},
// optionally remove "dryRun" from below if you want to leave a comment on removal
// PROTIP: the comment is bland, you should make it better
{
"kind": "comment",
"content": "Your submission has been removed it comprises 10% or more of your recent history ({{rules.attr.largestPercent}}{{rules.attrsub.largestPercent}}). This is against [reddit's self promotional guidelines.](https://www.reddit.com/wiki/selfpromotion#wiki_guidelines_for_self-promotion_on_reddit)",
"description": "Tag SP only if user does not have good contributor user note",
// check will run on a new submission in your subreddit and look at the Author of that submission
"kind": "submission",
"rules": [
{
"name": "attr10all",
"kind": "attribution",
"author": {
"exclude": [
{
// the key of the usernote type to look for https://github.com/toolbox-team/reddit-moderator-toolbox/wiki/Subreddit-Wikis%3A-usernotes#working-with-note-types
// rule will not run if current usernote on Author is of type 'gooduser'
description:Tag SP only if user does not have good contributor user note
# check will run on a new submission in your subreddit and look at the Author of that submission
kind:submission
rules:
- name:attr10all
kind:attribution
author:
exclude:
# the key of the usernote type to look for https://github.com/toolbox-team/reddit-moderator-toolbox/wiki/Subreddit-Wikis%3A-usernotes#working-with-note-types
# rule will not run if current usernote on Author is of type 'gooduser'
This getting started guide is for **Operators** -- that is, someone who wants to run the actual software for a ContentMod bot. If you are a **Moderator** check out the [moderator getting started](/docs/gettingStartedMod.md) guide instead.
* [Run Your Bot and Start Moderating](#run-your-bot-and-start-moderating)
# Installation
In order to run a ContextMod instance you must first you must install it somewhere.
ContextMod can be run on almost any operating system but it is recommended to use Docker due to ease of deployment.
## Docker (Recommended)
PROTIP: Using a container management tool like [Portainer.io CE](https://www.portainer.io/products/community-edition) will help with setup/configuration tremendously.
This template provides a **web** and **worker** dyno for heroku.
* **Web** -- Will run the bot **and** the web interface for ContextMod.
* **Worker** -- Will run **just** the bot.
Be aware that Heroku's [free dyno plan](https://devcenter.heroku.com/articles/free-dyno-hours#dyno-sleeping) enacts some limits:
* A **Web** dyno will go to sleep (pause) after 30 minutes without web activity -- so your bot will ALSO go to sleep at this time
* The **Worker** dyno **will not** go to sleep but you will NOT be able to access the web interface. You can, however, still see how Cm is running by reading the logs for the dyno.
If you want to use a free dyno it is recommended you perform first-time setup (bot authentication and configuration, testing, etc...) with the **Web** dyno, then SWITCH to a **Worker** dyno so it can run 24/7.
# Bot Authentication
Next you need to create a bot and authenticate it with Reddit. Follow the [bot authentication guide](/docs/botAuthentication.md) to complete this step.
# Instance Configuration
Finally, you must provide the credentials you received from the **Bot Authentication** step to the ContextMod instance you installed earlier. Refer to the [Operator Configuration](/docs/operatorConfiguration.md) guide to learn how this can be done as there are multiple approaches depending on how you installed the software.
Additionally, at this step you can also tweak many more settings and behavior concerning how your CM bot will operate.
# Run Your Bot and Start Moderating
Congratulations! You should now have a fully authenticated bot running on ContextMod software.
In order for your Bot to operate on reddit though it **must be a moderator in the subreddit you want it to run in.** This may be your own subreddit or someone else's.
**Note: ContextMod does not currently handle moderation invites automatically** and may never have this functionality. Due to the fact that many of its behaviors are api-heavy and that subreddits can control their own configuration the api and resource (cpu/memory) usage of a ContextMod instance can be highly variable. It therefore does not make sense to allow any/all subreddits to automatically have access to an instance through automatically accepting moderator invites. So...if you are planning to run a ContextMod instance for subreddits other than those you moderate you should establish solid trust with moderators of that subreddit as well as a solid line of communication in order to ensure their configurations can be tailored to best fit their needs and your resources.
Once you have logged in as your bot and manually accepted the moderator invite you will need to restart your ContextMod instance in order for these changes to take effect.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.