Merge branch 'edge'

* Support filename for non-raw gist
* Fix raw gist URL usage by fetching from CDN

.dockerignore

@@ -8,6 +8,7 @@ coverage
 .idea
 *.bak
 *.sqlite
+*.sqlite*
 *.json
 *.json5
 *.yaml

.gitignore

@@ -383,7 +383,7 @@ dist
 **/src/**/*.js
 **/tests/**/*.js
 **/tests/**/*.map
-!src/Web/assets/public/yaml/*
+!src/Web/assets/**
 **/src/**/*.map
 /**/*.sqlite
 /**/*.bak

Dockerfile

@@ -111,7 +111,9 @@ COPY --from=build --chown=abc:abc /app /app
 
 RUN npm install --production \
     && npm cache clean --force \
-    && chown abc:abc node_modules
+    && chown abc:abc node_modules \
+    && rm -rf node_modules/ts-node \
+    && rm -rf node_modules/typescript
 
 ENV NPM_CONFIG_LOGLEVEL debug
 

README.md

@@ -30,6 +30,7 @@ Feature Highlights for **Moderators:**
 * Event notification via Discord
 * [**Web interface**](#web-ui-and-screenshots) for monitoring, administration, and oauth bot authentication
 * [**Placeholders**](/docs/subreddit/actionTemplating.md) (like automoderator) can be configured via a wiki page or raw text and supports [mustache](https://mustache.github.io) templating
+* [**Partial Configurations**](/docs/subreddit/components/README.md#partial-configurations) -- offload parts of your configuration to shared locations to consolidate logic between multiple subreddits
 
 Feature highlights for **Developers and Hosting (Operators):**
 
@@ -43,6 +44,7 @@ Feature highlights for **Developers and Hosting (Operators):**
   * Historical statistics
 * [Docker container support](/docs/operator/installation.md#docker-recommended)
 * Easy, UI-based [OAuth authentication](/docs/operator/addingBot.md) for adding Bots and moderator dashboard
+* Integration with [InfluxDB](https://www.influxdata.com) for detailed [time-series metrics](/docs/operator/database.md#influx) and a pre-built [Grafana](https://grafana.com) [dashboard](/docs/operator/database.md#grafana)
 
 # Table of Contents
 
@@ -145,6 +147,13 @@ A built-in editor using [monaco-editor](https://microsoft.github.io/monaco-edito
 
 ![Configuration View](docs/images/editor.jpg)
 
+### [Grafana Dashboard](/docs/operator/database.md#grafana)
+
+* Overall stats (active bots/subreddits, api calls, per second/hour/minute activity ingest)
+* Over time graphs for events, per subreddit, and for individual rules/check/actions
+
+![Grafana Dashboard](/docs/images/grafana.jpg)
+
 ## License
 
 [MIT](/LICENSE)

docs/operator/database.md

@@ -130,3 +130,59 @@ retention: 2000
 runs:
   ...
 ```
+
+# Influx
+
+ContextMod supports writing detailed time-series data to [InfluxDB](https://www.influxdata.com/). 
+
+This data can be used to monitor the overall health, performance, and metrics for a ContextMod server. Currently, this data can **only be used by an Operator** as it requires access to the operator configuration and CM instance.
+
+CM supports InfluxDB OSS > 2.3 or InfluxDB Cloud.
+
+**Note:** This is an **advanced feature** and assumes you have enough technical knowledge to follow the documentation provided by each application to deploy and configure them. No support is guaranteed for installation, configuration, or use of Influx and Grafana.
+
+## Supported Metrics
+
+TBA
+
+## Setup
+
+### InfluxDB OSS
+
+* Install [InfluxDB](https://docs.influxdata.com/influxdb/v2.3/install/)
+* [Configure InfluxDB using the UI](https://docs.influxdata.com/influxdb/v2.3/install/#set-up-influxdb-through-the-ui)
+  * You will need **Username**, **Password**, **Organization Name**, and **Bucket Name** later for Grafana setup so make sure to record them somewhere
+* [Create a Token](https://docs.influxdata.com/influxdb/v2.3/security/tokens/create-token/) with enough permissions to write/read to the bucket you configured
+  * After the token is created **view/copy the token** to clipboard by clicking the token name. You will need this for Grafana setup.
+
+### ContextMod
+
+Add the following block to the top-level of your operator configuration:
+
+```yaml
+influxConfig:
+  credentials:
+    url: 'http://localhost:8086' # URL to your influx DB instance
+    token: '9RtZ5YZ6bfEXAMPLENJsTSKg==' # token created in the previous step
+    org: MyOrg # organization created in the previous step
+    bucket: contextmod # name of the bucket created in the previous step
+```
+
+## Grafana
+
+A pre-built dashboard for [Grafana](https://grafana.com) can be imported to display overall metrics/stats using InfluxDB data.
+
+![Grafana Dashboard](/docs/images/grafana.jpg)
+
+* Create a new Data Source using **InfluxDB** type
+  * Choose **Flux** for the **Query Language**
+  * Fill in the details for **URL**, **Basic Auth Details** and **InfluxDB Details** using the data you created in the [Influx Setup step](#influxdb-oss)
+  * Set **Min time interval** to `60s`
+  * Click **Save and test**
+* Import Dashboard
+  * **Browse** the Dashboard pane
+  * Click **Import** and **upload** the [grafana dashboard json file](/docs/operator/grafana.json)
+    * Chose the data source you created from the **InfluxDB CM** dropdown
+    * Click **Import**
+
+The dashboard can be filtered by **Bots** and **Subreddits** dropdowns at the top of the page to get more specific details.

docs/subreddit/actionTemplating.md

@@ -2,44 +2,92 @@ Actions that can submit text (Report, Comment, UserNote) will have their `conten
 
 See here for a [cheatsheet](https://gist.github.com/FoxxMD/d365707cf99fdb526a504b8b833a5b78) and [here](https://www.tsmean.com/articles/mustache/the-ultimate-mustache-tutorial/) for a more thorough tutorial.
 
+# Template Data
+
+## Activity Data
+
+Activity data can be accessed using the `item` variable. Example
+
+```
+This activity is a {{item.kind}} with {{item.votes}} votes, created {{item.age}} ago.
+```
+Produces:
+
+> This activity is a submission with 10 votes created 5 minutes ago.
+
+### Common
+
 All Actions with `content` have access to this data:
 
-```json5
+| Name        | Description                                                                                         | Example                                                                              |
+|-------------|-----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
+| `kind`      | The Activity type (submission or comment)                                                           | submission                                                                           |
+| `author`    | Name of the Author of the Activity being processed                                                  | FoxxMD                                                                               |
+| `permalink` | URL to the Activity                                                                                 | https://reddit.com/r/mySuibreddit/comments/ab23f/my_post                             |
+| `votes`     | Number of upvotes                                                                                   | 69                                                                                   |
+| `age`       | The age of the Activity in a [human friendly format](https://day.js.org/docs/en/durations/humanize) | 5 minutes                                                                            |
+| `botLink`   | A URL to CM's introduction thread                                                                   | https://www.reddit.com/r/ContextModBot/comments/otz396/introduction_to_contextmodbot |
 
-{
-    item: {
-        kind: 'string', // the type of item (comment/submission)
-        author: 'string', // name of the item author (reddit user)
-        permalink: 'string', // a url to the item
-        url: 'string', // if the item is a Submission then its URL (external for link type submission, reddit link for self-posts)
-        title: 'string', // if the item is a Submission, then the title of the Submission,
-        botLink: 'string' // a link to the bot's FAQ
-    },
-    rules: {
-        // contains all rules that were run and are accessible using the name, lowercased, with all spaces/dashes/underscores removed
-    }
-}
+### Submissions
+
+If the **Activity** is a Submission these additional properties are accessible:
+
+| Name          | Description                                                     | Example                 |
+|---------------|-----------------------------------------------------------------|-------------------------|
+| `upvoteRatio` | The upvote ratio                                                | 100%                    |
+| `nsfw`        | If the submission is marked as NSFW                             | true                    |
+| `spoiler`     | If the submission is marked as a spoiler                        | true                    |
+| `url`         | If the submission was a link then this is the URL for that link | http://example.com      |
+| `title`       | The title of the submission                                     | Test post please ignore |
+
+### Comments
+
+If the **Activity** is a Comment these additional properties are accessible:
+
+| Name | Description                                                  | Example |
+|------|--------------------------------------------------------------|---------|
+| `op` | If the Author is the OP of the Submission this comment is in | true    |
+
+### Moderator
+
+If the **Activity** occurred in a Subreddit the Bot moderates these properties are accessible:
+
+| Name          | Description                         | Example |
+|---------------|-------------------------------------|---------|
+| `reports`     | The number of reports recieved      | 1       |
+| `modReports`  | The number of reports by moderators | 1       |
+| `userReports` | The number of reports by users      | 1       |
+
+## Rule Data
+
+### Summary
+
+A summary of what rules were processed and which were triggered, with results, is available using the `ruleSummary` variable. Example:
 
 ```
+A summary of rules processed for this activity:
 
-The properties of `rules` are accessible using the name, lower-cased, with all spaces/dashes/underscores. If no name is given `kind` is used as `name` Example:
-
+{{ruleSummary}}
 ```
 
-"rules": [
-  {
-    "name": "My Custom-Recent Activity Rule", // mycustomrecentactivityrule
-    "kind": "recentActivity"
-  },
-  {
-    // name = repeatsubmission
-    "kind": "repeatActivity",
-  }
-]
+Would produce:
+> A summary of rules processed for this activity:
+> 
+> * namedRegexRule - ✘
+> * nameAttributionRule - ✓ - 1 Attribution(s) met the threshold of < 20%, with 1 (3%) of 32 Total -- window: 6 months
+> * noXPost ✓ - ✓ 1 of 1 unique items repeated <= 3 times, largest repeat: 1
+
+
+### Individual
+
+Individual **Rules** can be accessed using the name of the rule, **lower-cased, with all spaces/dashes/underscores.** Example:
 
 ```
+Submission was repeated {{rules.noxpost.largestRepeat}} times
+```
+Produces
 
-**To see what data is available for individual Rules [consult the schema](#configuration) for each Rule.**
+> Submission was repeated 7 times
 
 #### Quick Templating Tutorial
 

docs/subreddit/components/README.md

@@ -21,6 +21,7 @@ This list is not exhaustive. [For complete documentation on a subreddit's config
     * [Author](#author)
     * [Regex](#regex)
     * [Repost](#repost)
+    * [Sentiment Analysis](#sentiment-analysis)
 * [Rule Sets](#rule-sets)
 * [Actions](#actions)
   * [Named Actions](#named-actions)
@@ -39,10 +40,13 @@ This list is not exhaustive. [For complete documentation on a subreddit's config
     * [Message](#message)
     * [Remove](#remove)
     * [Report](#report)
-    * [UserNote](#usernote)
+    * [Toolbox UserNote](#usernote)
+    * [Mod Note](#mod-note)
 * [Filters](#filters)
   * [Filter Types](#filter-types)
     * [Author Filter](#author-filter)
+      * [Mod Notes/Actions](#mod-actionsnotes-filter)
+      * [Toolbox UserNotes](#toolbox-usernotes-filter)
     * [Item Filter](#item-filter)
     * [Subreddit Filter](#subreddit-filter)
   * [Named Filters](#named-filters)
@@ -63,6 +67,7 @@ This list is not exhaustive. [For complete documentation on a subreddit's config
     * [Check Order](#check-order)
     * [Rule Order](#rule-order)
   * [Configuration Re-use and Caching](#configuration-re-use-and-caching)
+  * [Partial Configurations](#partial-configurations)
 * [Subreddit-ready examples](#subreddit-ready-examples)
 
 # Runs
@@ -365,6 +370,12 @@ The **Repost** rule is used to find reposts for both **Submissions** and **Comme
 
 This rule is for searching **all of Reddit** for reposts, as opposed to just the history of the Author of the Activity being checked. If you only want to check for reposts by the Author of the Activity being checked you should use the [Repeat Activity](/docs/subreddit/components/repeatActivity) rule.
 
+### Sentiment Analysis
+
+[**Full Documentation**](/docs/subreddit/components/sentiment)
+
+The **Sentiment Rule** is used to determine the overall emotional intent (negative, neutral, positive) of a Submission or Comment by analyzing the actual text content of the Activity.
+
 # Rule Sets
 
 The `rules` list on a `Check` can contain both `Rule` objects and `RuleSet` objects.
@@ -651,6 +662,30 @@ actions:
     allowDuplicate: boolean # if false then the usernote will not be added if the same note appears for this activity
 ```
 
+### Mod Note
+
+[**Full Documentation**](/docs/subreddit/components/modActions/README.md#mod-note-action)
+
+Add a [Mod Note](https://www.reddit.com/r/modnews/comments/t8vafc/announcing_mod_notes/) for the Author of the Activity.
+
+* `type` must be one of the [valid note labels](https://www.reddit.com/dev/api#POST_api_mod_notes): 
+  * BOT_BAN
+  * PERMA_BAN
+  * BAN
+  * ABUSE_WARNING
+  * SPAM_WARNING
+  * SPAM_WATCH
+  * SOLID_CONTRIBUTOR
+  * HELPFUL_USER
+
+```yaml
+actions:
+  - kind: modnote
+    type: SPAM_WATCH
+    content: 'a note only mods can see message' # optional
+    referenceActivity: boolean # if true the Note will be linked to the Activity being processed
+```
+
 # Filters
 
 **Filters** are an additional channel for determining if an Event should be processed by ContextMod. They differ from **Rules** in several key ways:
@@ -732,6 +767,14 @@ There are two types of Filter. Both types have the same "shape" in the configura
 
 Test the Author of an Activity. See [Schema documentation](https://json-schema.app/view/%23%2Fdefinitions%2FAuthorCriteria?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Freddit-context-bot%2Fedge%2Fsrc%2FSchema%2FApp.json) for all possible Author Criteria
 
+#### Mod Actions/Notes Filter
+
+See [Mod Actions/Notes](/docs/subreddit/components/modActions/README.md#mod-action-filter) documentation.
+
+#### Toolbox UserNotes Filter
+
+See [UserNotes](/docs/subreddit/components/userNotes/README.md) documentation
+
 ### Item Filter
 
 Test for properties of an Activity:
@@ -1031,7 +1074,7 @@ If the Check is using `AND` condition for its rules (default) then if either Rul
 
 **It is therefore advantageous to list your lightweight Rules first in each Check.**
 
-### Configuration Re-use and Caching
+## Configuration Re-use and Caching
 
 ContextMod implements caching functionality for:
 
@@ -1055,6 +1098,116 @@ PROTIP: You can monitor the re-use of cache in the `Cache` section of your subre
 
 [Learn more about how Caching works](/docs/operator/caching.md)
 
+## Partial Configurations
+
+ContextMod supports fetching parts of a configuration (a **Fragment**) from an external source. Fragments are an advanced feature and should only be used by users who are familiar with CM's configuration syntax and understand the risks/downsides associates with fragmenting a configuration.
+
+**Fragments** are supported for:
+
+* [Runs](#runs)
+* [Checks](#checks)
+* [Rules](#rules)
+* [Actions](#actions)
+
+### Should You Use Partial Configurations?
+
+* **PROS**
+  * Consolidate shared configuration for many subreddits into one location
+  * Shared configuration can be updated independently of subreddits
+  * Allows sharing access to configuration outside of moderators of a specific subreddit or even reddit
+* **CONS**
+  * Editor does not currently support viewing, editing, or updating Fragments. Only the Fragment URL is visible in a Subreddit's configuration
+    * No editor support for viewing obscures "complete view" of configuration and makes editor less useful for validation
+  * Currently, editor cannot validate individual Fragments. They must be copy-pasted "in place" within a normal configuration.
+  * Using external (non-wiki) sources means **you** are responsible for the security/access to the fragment
+
+In general, Fragments should only be used to offload small, well-tested pieces of a configuration that can be shared between many subreddits. Examples:
+
+* A regex Rule for spam links
+* A Recent Activity Rule for reporting users from freekarma subreddits
+
+### Usage
+
+A Fragment may be either a special string or a Fragment object. The fetched Fragment can be either an object or an array of objects of the type of Fragment being replaced.
+
+**String**
+
+If value starts with `wiki:` then the proceeding value will be used to get a wiki page from the current subreddit
+
+* EX `wiki:botconfig/myFragment` tries to get `https://reddit.com/r/currentSubreddit/wiki/botconfig/myFragment`
+
+If the value starts with `wiki:` and ends with `|someValue` then `someValue` will be used as the base subreddit for the wiki page
+
+* EX `wiki:myFragment/test|ContextModBot` tries to get `https://reddit.com/r/ContextModBot/wiki/myFragment/test`
+
+If the value starts with `url:` then the value is fetched as an external url and expects raw text returned
+
+* EX `url:https://pastebin.com/raw/38qfL7mL` tries to get the text response of `https://pastebin.com/raw/38qfL7mL`
+
+**Object**
+
+The object contains:
+
+* `path` -- REQUIRED string following rules above
+* `ttl` -- OPTIONAL, number of seconds to cache the URL result. Defaults to `WikiTTL`
+
+#### Examples
+
+**Replacing A Rule with a URL Fragment**
+
+```yaml
+runs:
+- checks:
+  - name: Free Karma Alert
+    description: Check if author has posted in 'freekarma' subreddits
+    kind: submission
+    rules:
+      - 'url:https://gist.githubusercontent.com/FoxxMD/0e1ee1ab950ff4d1f0cd26172bae7f8f/raw/0ebfaca903e4a651827effac5775c8718fb6e1f2/fragmentRule.yaml'
+      - name: badSub
+        kind: recentActivity
+        useSubmissionAsReference: false
+        thresholds:
+          # if the number of activities (sub/comment) found CUMULATIVELY in the subreddits listed is
+          # equal to or greater than 1 then the rule is triggered
+          - threshold: '>= 1'
+            subreddits:
+              - MyBadSubreddit
+        window: 7 days
+      actions:
+      - kind: report
+        content: 'uses freekarma subreddits and bad subreddits'
+```
+
+**Replacing A Rule with a URL Fragment (Multiple)**
+
+```yaml
+runs:
+- checks:
+  - name: Free Karma Alert
+    description: Check if author has posted in 'freekarma' subreddits
+    kind: submission
+    rules:
+      - 'url:https://gist.githubusercontent.com/FoxxMD/0e1ee1ab950ff4d1f0cd26172bae7f8f/raw/0ebfaca903e4a651827effac5775c8718fb6e1f2/fragmentRuleArray.yaml'
+    actions:
+      - kind: report
+        content: 'uses freekarma subreddits and bad subreddits'
+```
+
+**Replacing A Rule with a Wiki Fragment**
+
+```yaml
+runs:
+- checks:
+  - name: Free Karma Alert
+    description: Check if author has posted in 'freekarma' subreddits
+    kind: submission
+    rules:
+      - 'wiki:freeKarmaFrag'
+    actions:
+      - kind: report
+        content: 'uses freekarma subreddits'
+```
+
 # Subreddit-Ready Examples
 
 Refer to the [Subreddit-Ready Examples](/docs/subreddit/components/subredditReady) section to find ready-to-use configurations for common scenarios (spam, freekarma blocking, etc...). This is also a good place to familiarize yourself with what complete configurations look like.

docs/subreddit/components/modActions/README.md

@@ -0,0 +1,152 @@
+# Table of Contents
+
+* [Overview](#overview)
+* [Mod Note Action](#mod-note-action)
+* [Mod Action Filter](#mod-action-filter)
+  * [API Usage](#api-usage)
+  * [When To Use?](#when-to-use)
+  * [Examples](#examples)
+
+# Overview
+
+[Mod Notes](https://www.reddit.com/r/modnews/comments/t8vafc/announcing_mod_notes/) is a feature for New Reddit that allow moderators to add short, categorizable notes to Users of their subreddit, optionally associating te note with a submission/comment the User made. They are inspired by [Toolbox User Notes](https://www.reddit.com/r/toolbox/wiki/docs/usernotes) which are also [supported by ContextMod.](/docs/subreddit/components/userNotes) Reddit's **Mod Notes** also combine [Moderation Log](https://mods.reddithelp.com/hc/en-us/articles/360022402312-Moderation-Log) actions (**Mod Actions**) for the selected User alongside moderator notes, enabling a full "overview" of moderator interactions with a User in their subreddit.
+
+ContextMod supports adding **Mod Notes** to an Author using an [Action](/docs/subreddit/components/README.md#mod-note) and using **Mod Actions/Mod Notes** as a criteria in an [Author Filter](/docs/subreddit/components/README.md#author-filter)
+
+# Mod Note Action
+
+[**Schema Reference**](https://json-schema.app/view/%23%2Fdefinitions%2FModNoteActionJson?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Freddit-context-bot%2Fedge%2Fsrc%2FSchema%2FApp.json)
+
+* `type` must be one of the [valid note labels](https://www.reddit.com/dev/api#POST_api_mod_notes):
+    * BOT_BAN
+    * PERMA_BAN
+    * BAN
+    * ABUSE_WARNING
+    * SPAM_WARNING
+    * SPAM_WATCH
+    * SOLID_CONTRIBUTOR
+    * HELPFUL_USER
+
+```yaml
+actions:
+  - kind: modnote
+    type: SPAM_WATCH
+    content: 'a note only mods can see message' # optional
+    referenceActivity: boolean # if true the Note will be linked to the Activity being processed
+```
+
+# Mod Action Filter
+
+ContextMod can use **Mod Actions** (from moderation log) and **Mod Notes** in an [Author Filter](/docs/subreddit/components/README.md#author-filter).
+
+## API Usage
+
+Notes/Actions are **not** included in the data Reddit returns for either an Author or an Activity. This means that, in most cases, ContextMod is required to make **one additional API call to Reddit during Activity processing** if Notes/Actions as used as part of an **Author Filter**.
+
+The impact of this additional call is greatest when the Author Filter is used as part of a **Comment Check** or running for **every Activity** such as part of a Run. Take this example:
+
+No Mod Action filtering
+
+* CM makes 1 api call to return new comments, find 10 new comments across 6 users
+* Processing each comment, with no other filters, requires 0 additional calls
+* At the end of processing 10 comments, CM has used a total of 1 api call.
+
+Mod Action Filtering Used
+
+* CM makes 1 api call to return new comments, find 10 new comments across 6 users
+* Processing each comment, with a mod action filter, requires 1 additional api call per user
+* At the end of processing 10 comments, CM has used a total of **7 api calls**
+
+### When To Use?
+
+In general,**do not** use Mod Actions in a Filter if:
+
+* The filter is on a [**Comment** Check](/docs/subreddit/components/README.md#checks) and your subreddit has a high volume of Comments
+* The filter is on a [Run](/docs/subreddit/components/README.md#runs) and your subreddit has a high volume of Activities
+
+If you need Mod Notes-like functionality for a high volume subreddit consider using [Toolbox UserNotes](/docs/subreddit/components/userNotes) instead.
+
+In general, **do** use Mod Actions in a Filter if:
+
+* The filter is on a [**Submission** Check](/docs/subreddit/components/README.md#checks)
+* The filter is part of an [Author **Rule**](/docs/subreddit/components/README.md#author) that is processed as **late as possible in the rule order for a Check**
+* Your subreddit has a low volume of Activities (less than 100 combined submissions/comments in a 10 minute period, for example)
+* The filter is on an Action
+
+## Usage and Examples
+
+Filter by Mod Actions/Notes on an Author Filter are done using the `modActions` property:
+
+```yaml
+age: '> 1 month'
+# ...
+modActions:
+  - ...
+```
+
+There two valid shapes for the Mod Action criteria: [ModLogCriteria](https://json-schema.app/view/%23%2Fdefinitions%2FModLogCriteria?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Freddit-context-bot%2Fedge%2Fsrc%2FSchema%2FApp.json) and [ModNoteCriteria](https://json-schema.app/view/%23%2Fdefinitions%2FModNoteCriteria?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Freddit-context-bot%2Fedge%2Fsrc%2FSchema%2FApp.json).
+
+### ModLogCriteria
+
+Used for filtering by **Moderation Log** actions *and/or general notes*.
+
+* `activityType` -- Optional. If Mod Action is associated with an activity specify it here. A list or one of:
+  * `submission`
+  * `comment`
+* `type` -- Optional. The type of Mod Log Action. A list or one of:
+  * `INVITE`
+  * `NOTE`
+  * `REMOVAL`
+  * `SPAM`
+  * `APPROVAL`
+* `description` -- additional mod log details (string) to filter by -- not documented by reddit. Can be string or regex string-like `/.* test/i`
+* `details` -- additional mod log details (string) to filter by -- not documented by reddit. Can be string or regex string-like `/.* test/i`
+
+```yaml
+activityType: submission
+type:
+  - REMOVAL
+  - SPAM
+search: total
+count: '> 3 in 1 week'
+```
+### ModNoteCriteria
+
+Inherits `activityType` from ModLogCriteria. If either of the below properties in included on the criteria then any other ModLogCriteria-specific properties are **ignored**.
+
+* `note` -- the contents of the note to match against. Can be one of or a list of strings/regex string-like `/.* test/i`
+* `noteType` -- If specified by the note, the note type (see [Mod Note Action](#mod-note-action) type). Can be one of or a list of strings/regex string-like `/.* test/i`
+
+```yaml
+noteType: SOLID_CONTRIBUTOR
+search: total
+count: '> 3 in 1 week'
+```
+
+### Examples
+
+Author has more than 2 submission approvals in the last month
+
+```yaml
+type: APPROVAL
+activityType: submission
+search: total
+count: '> 2 in 1 month'
+```
+
+Author has at least 1 BAN note
+
+```yaml
+noteType: BAN
+search: total
+count: '>= 1'
+```
+
+Author has at least 3 notes which include the words "self" and "promotion" in the last month
+
+```yaml
+note: '/self.*promo/i'
+activityType: submission
+search: total
+count: '>= 3 in 1 month'
+```

docs/subreddit/components/sentiment/README.md

@@ -0,0 +1,183 @@
+# Table of Contents
+
+* [Overview](#overview)
+  * [Pros And Cons](#pros-and-cons)
+  * [Technical Overview](#technical-overview)
+* [Sentiment Values](#sentiment-values)
+* [Usage](#usage)
+  * [Testing Sentiment Value](#testing-sentiment-value)
+    * [Numerical](#numerical)
+    * [Text](#text)
+  * [Sentiment Rule](#sentiment-rule)
+    * [Historical](#historical)
+* [Examples](#examples)
+
+# Overview
+
+[Sentiment Analysis](https://monkeylearn.com/sentiment-analysis/) (SA) is a form of [Natural Language Processing](https://monkeylearn.com/natural-language-processing/) (NLP) used to extract the overall [sentiment](https://www.merriam-webster.com/dictionary/sentiment) (emotional intent) from a piece of text. Simply, SA is used to determine how positive or negative the emotion of a sentence is.
+
+Examples:
+
+* "I love how curly your hair is" -- very positive
+* "The United States is over 200 years old" -- neutral
+* "Frankly, your face is disgusting and I would hate to meet you" -- very negative
+
+SA can be a powerful signal for determining the intent of a user's comment/submission. However, it should not be the **only** tool as it comes with both strengths and weaknesses.
+
+## Pros and Cons
+
+Pros
+
+* In terms of Reddit API usage, SA is **free**. It requires no API calls and is computationally trivial.
+* Extremely powerful signal for intent since it analyzes the actual text content of an activity
+* Requires almost no setup to use
+  * Can be used as a substitute for regex/keyword matching when looking for hateful/toxic comments
+* English language comprehension is very thorough
+  * Uses 3 independent algorithms to evaluate sentiment
+  * Understands common english slang, internet slang, and emojis
+
+Cons
+
+* Language limited -- only supported for English (most thorough), French, German, and Spanish
+* Less accurate for small word count content (less than 4 words)
+* Does not understand sarcasm/jokes
+* Accuracy depends on use of common words
+* Accuracy depends on clear intent
+  * Heavy nuance, obscure word choice, and hidden meanings are not understood
+
+## Technical Overview
+
+ContextMod attempts to identify the language of the content it is processing. Based on its confidence of the language it will use up to three different NLP libraries to extract sentiment:
+
+* [NLP.js](https://github.com/axa-group/nlp.js/blob/master/docs/v3/sentiment-analysis.md) (english, french, german, and spanish)
+* [vaderSentiment-js](https://github.com/vaderSentiment/vaderSentiment-js/) (english only)
+* [wink-sentiment](https://github.com/winkjs/wink-sentiment) (english only)
+
+The above libraries make use of these Sentiment Analysis algorithms:
+
+* VADER https://github.com/cjhutto/vaderSentiment
+* AFINN http://corpustext.com/reference/sentiment_afinn.html
+* Senticon https://ieeexplore.ieee.org/document/8721408
+* Pattern https://github.com/clips/pattern
+* wink https://github.com/winkjs/wink-sentiment (modified AFINN with emojis)
+
+Each library produces a normalized score: the sum of all the valence values for each recognized token in its lexicon, divided by the number of words/tokens.
+
+ContextMod takes each normalized score and adjusts it to be between -1 and +1. It then adds finds the average of all normalized score to produce a final sentiment between -1 and +1.
+
+# Sentiment Values
+
+Each piece of content ContextMod analyses produces a score from -1 to +1 to represent the sentiment of that content
+
+| Score | Sentiment          |
+|-------|--------------------|
+| -1    |                    |
+| -0.6  | Extremely Negative |
+| -0.3  | Very Negative      |
+| -0.1  | Negative           |
+| 0     | Neutral            |
+| 0.1   | Positive           |
+| 0.3   | Very Positive      |
+| 0.6   | Extremely Positive |
+| 1     |                    |
+
+# Usage
+
+## Testing Sentiment Value
+
+Testing for sentiment in the Sentiment Rule is done using either a **text** or **numerical** comparison.
+
+### Numerical
+
+Similar to other numerical comparisons in CM -- use an equality operator and the number to test for:
+
+* `> 0.1` -- sentiment is at least positive
+* `<= -0.1` -- sentiment is not negative
+
+Testing for *only* neutral sentiment should be done use a text comparison (below).
+
+### Text
+
+Use any of the **Sentiment** text values from the above table to form a test:
+
+* `is very positive`
+* `is neutral`
+* `is extremely negative`
+
+You may also use the `not` operator:
+
+* `is not negative`
+* `is not very negative`
+* `is not neutral`
+
+## Sentiment Rule
+
+An example rule that tests the current comment/submission to see if it has negative sentiment:
+
+```yaml
+sentiment: 'is negative'
+```
+
+It's very simple :)
+
+### Historical
+
+You may also test the Sentiment of Activities from the user's history. (Note: this may use an API call to get history)
+
+```yaml
+sentiment: 'is negative'
+historical:
+  window:
+    count: 50
+  mustMatchCurrent: true # optional, the initial activity being tested must test true ("is positive" must be true) before historical tests are run
+  sentimentVal: 'is very negative' # optional, if the sentiment test to use for historical content is different than the initial test
+  totalMatching: '> 3' # optional, a comparison for how many historical activities must match sentimentVal
+```
+
+# Examples
+
+#### Check with Rules for recent problem subreddit activity and negative sentiment in comment
+
+```yaml
+name: Probably Toxic Comment
+kind: comment
+rules:
+  - kind: recentActivity
+    thresholds:
+      - aProblemSubreddit
+  - kind: sentiment
+    name: negsentiment
+    sentiment: 'is very negative'
+actions:
+  - kind: report
+    content: 'Sentiment of {{rules.negsentiment.averageScore}} {{rules.negsentiment.sentimentTest}}'
+```
+
+#### Check with Rules for recent problem subreddit activity and negative sentiment in comment history from problem subreddits
+
+```yaml
+name: Toxic Comment With History
+kind: comment
+rules:
+  - kind: recentActivity
+    thresholds:
+      - aProblemSubreddit
+      - aSecondProblemSubreddit
+  - kind: sentiment
+    sentiment: 'is very negative'
+    historical:
+      sentimentVal: 'is negative'
+      mustMatchCurrent: true
+      totalMatching: '> 1'
+      window:
+        count: 100
+        filterOn:
+          post:
+            subreddits:
+              include:
+                - name:
+                  - aProblemSubreddit
+                  - aSecondProblemSubreddit
+actions:
+  - kind: remove
+```

docs/subreddit/components/userNotes/README.md

@@ -6,6 +6,16 @@ Context Mod supports reading and writing [User Notes](https://www.reddit.com/r/t
 
 [Click here for the Toolbox Quickstart Guide](https://www.reddit.com/r/toolbox/wiki/docs/quick_start)
 
+Valid Note Types:
+
+* `gooduser`
+* `spamwatch`
+* `spamwarn`
+* `abusewarn`
+* `ban`
+* `permban`
+* `botban`
+
 ## Filter
 
 User Notes are an additional criteria on [AuthorCriteria](https://json-schema.app/view/%23%2Fdefinitions%2FAuthorCriteria?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FApp.json) that can be used alongside other Author properties for both [filtering rules and in the AuthorRule.](/docs/subreddit/components/author/)

package.json

@@ -1,19 +1,22 @@
 {
   "name": "redditcontextbot",
-  "version": "0.5.1",
+  "version": "0.11.4",
   "description": "",
   "main": "index.js",
   "scripts": {
     "test": "nyc ./node_modules/.bin/_mocha 'tests/**/*.test.ts'",
-    "build": "tsc",
+    "build": "tsc && npm run bundle-front",
+    "bundle-front": "browserify src/Web/assets/browser.js | terser --compress --mangle > src/Web/assets/public/browserBundle.js",
     "start": "node src/index.js run",
-    "schema": "npm run -s schema-app & npm run -s schema-ruleset & npm run -s schema-rule & npm run -s schema-action & npm run -s schema-config",
-    "schema-app": "typescript-json-schema tsconfig.json JSONConfig --out src/Schema/App.json --required --tsNodeRegister --refs --validationKeywords deprecationMessage",
-    "schema-ruleset": "typescript-json-schema tsconfig.json RuleSetJson --out src/Schema/RuleSet.json --required --tsNodeRegister --refs --validationKeywords deprecationMessage",
-    "schema-rule": "typescript-json-schema tsconfig.json RuleJson --out src/Schema/Rule.json --required --tsNodeRegister --refs --validationKeywords deprecationMessage",
-    "schema-action": "typescript-json-schema tsconfig.json ActionJson --out src/Schema/Action.json --required --tsNodeRegister --refs --validationKeywords deprecationMessage",
+    "schema": "npm run -s schema-app & npm run -s schema-ruleset & npm run -s schema-rule & npm run -s schema-action & npm run -s schema-check & npm run -s schema-run & npm run -s schema-config",
+    "schema-app": "typescript-json-schema tsconfig.json SubredditConfigData --out src/Schema/App.json --required --tsNodeRegister --refs --validationKeywords deprecationMessage",
+    "schema-ruleset": "typescript-json-schema tsconfig.json RuleSetConfigData --out src/Schema/RuleSet.json --required --tsNodeRegister --refs --validationKeywords deprecationMessage",
+    "schema-rule": "typescript-json-schema tsconfig.json RuleConfigData --out src/Schema/Rule.json --required --tsNodeRegister --refs --validationKeywords deprecationMessage",
+    "schema-check": "typescript-json-schema tsconfig.json ActivityCheckConfigValue --out src/Schema/Check.json --required --tsNodeRegister --refs --validationKeywords deprecationMessage",
+    "schema-run": "typescript-json-schema tsconfig.json RunConfigValue --out src/Schema/Run.json --required --tsNodeRegister --refs --validationKeywords deprecationMessage",
+    "schema-action": "typescript-json-schema tsconfig.json ActionConfigData --out src/Schema/Action.json --required --tsNodeRegister --refs --validationKeywords deprecationMessage",
     "schema-config": "typescript-json-schema tsconfig.json OperatorJsonConfig --out src/Schema/OperatorConfig.json --required --tsNodeRegister --refs --validationKeywords deprecationMessage",
-    "schemaNotWorking": "./node_modules/.bin/ts-json-schema-generator -f tsconfig.json -p src/JsonConfig.ts -t JSONConfig --out src/Schema/vegaSchema.json",
+    "schemaNotWorking": "./node_modules/.bin/ts-json-schema-generator -f tsconfig.json -p src/SubredditConfigData.ts -t JSONConfig --out src/Schema/vegaSchema.json",
     "circular": "madge --circular --extensions ts src/index.ts",
     "circular-graph": "madge --image graph.svg --circular --extensions ts src/index.ts",
     "postinstall": "patch-package",
@@ -29,6 +32,15 @@
   "dependencies": {
     "@awaitjs/express": "^0.8.0",
     "@googleapis/youtube": "^2.0.0",
+    "@influxdata/influxdb-client": "^1.27.0",
+    "@influxdata/influxdb-client-apis": "^1.27.0",
+    "@nlpjs/core": "^4.23.4",
+    "@nlpjs/lang-de": "^4.23.4",
+    "@nlpjs/lang-en": "^4.23.4",
+    "@nlpjs/lang-es": "^4.23.4",
+    "@nlpjs/lang-fr": "^4.23.4",
+    "@nlpjs/language": "^4.22.7",
+    "@nlpjs/nlp": "^4.23.5",
     "@stdlib/regexp-regexp": "^0.0.6",
     "ajv": "^7.2.4",
     "ansi-regex": ">=5.0.1",
@@ -39,11 +51,10 @@
     "cache-manager-redis-store": "^2.0.0",
     "commander": "^8.0.0",
     "comment-json": "^4.1.1",
-    "connect-typeorm": "github:FoxxMD/connect-typeorm#typeormBump",
+    "connect-typeorm": "^2.0.0",
     "cookie-parser": "^1.3.5",
     "dayjs": "^1.10.5",
     "deepmerge": "^4.2.2",
-    "delimiter-stream": "^3.0.1",
     "ejs": "^3.1.6",
     "env-cmd": "^10.1.0",
     "es6-error": "^4.1.1",
@@ -52,16 +63,15 @@
     "express-session-cache-manager": "^1.0.2",
     "express-socket.io-session": "^1.3.5",
     "fast-deep-equal": "^3.1.3",
-    "fuse.js": "^6.4.6",
     "globrex": "^0.1.2",
     "got": "^11.8.2",
     "he": "^1.2.0",
     "http-proxy": "^1.18.1",
-    "image-size": "^1.0.0",
     "json5": "^2.2.0",
     "jsonwebtoken": "^8.5.1",
     "leven": "^3.1.0",
     "lodash": "^4.17.21",
+    "logform": "^2.4.0",
     "lru-cache": "^6.0.0",
     "migrate": "github:johsunds/node-migrate#49b0054de0a9295857aa8b8eea9a3cdeb2643913",
     "mustache": "^4.2.0",
@@ -77,9 +87,7 @@
     "patch-package": "^6.4.7",
     "pixelmatch": "^5.2.1",
     "pony-cause": "^1.1.1",
-    "pretty-print-json": "^1.0.3",
     "reflect-metadata": "^0.1.13",
-    "safe-stable-stringify": "^1.1.1",
     "snoostorm": "^1.5.2",
     "snoowrap": "^1.23.0",
     "socket.io": "^4.1.3",
@@ -87,10 +95,11 @@
     "string-similarity": "^4.0.4",
     "tcp-port-used": "^1.0.2",
     "triple-beam": "^1.3.0",
-    "typeorm": "^0.3.4",
+    "typeorm": "^0.3.7",
     "typeorm-logger-adaptor": "^1.1.0",
-    "typescript": "^4.3.4",
+    "vader-sentiment": "^1.1.3",
     "webhook-discord": "^3.7.7",
+    "wink-sentiment": "^5.0.2",
     "winston": "github:FoxxMD/winston#fbab8de969ecee578981c77846156c7f43b5f01e",
     "winston-daily-rotate-file": "^4.5.5",
     "winston-duplex": "^0.1.1",
@@ -106,6 +115,7 @@
     "@types/cache-manager": "^3.4.2",
     "@types/cache-manager-redis-store": "^2.0.0",
     "@types/chai": "^4.3.0",
+    "@types/chai-as-promised": "^7.1.5",
     "@types/cookie-parser": "^1.4.2",
     "@types/express": "^4.17.13",
     "@types/express-session": "^1.17.4",
@@ -130,16 +140,19 @@
     "@types/string-similarity": "^4.0.0",
     "@types/tcp-port-used": "^1.0.0",
     "@types/triple-beam": "^1.3.2",
+    "browserify": "^17.0.0",
     "chai": "^4.3.6",
+    "chai-as-promised": "^7.1.1",
     "mocha": "^9.2.1",
     "nyc": "^15.1.0",
     "source-map-support": "^0.5.21",
+    "terser": "^5.13.1",
     "ts-essentials": "^9.1.2",
     "ts-json-schema-generator": "^0.93.0",
     "ts-mockito": "^2.6.1",
     "ts-node": "^10.7.0",
     "tsconfig-paths": "^3.13.0",
-    "typescript": "^4.3.4",
+    "typescript": "^4.6.4",
     "typescript-json-schema": "~0.53"
   },
   "optionalDependencies": {

src/Action/ActionFactory.ts

@@ -17,6 +17,7 @@ import {DispatchAction, DispatchActionJson} from "./DispatchAction";
 import {CancelDispatchAction, CancelDispatchActionJson} from "./CancelDispatchAction";
 import ContributorAction, {ContributorActionJson} from "./ContributorAction";
 import {StructuredFilter} from "../Common/Infrastructure/Filters/FilterShapes";
+import {ModNoteAction, ModNoteActionJson} from "./ModNoteAction";
 
 export function actionFactory
 (config: StructuredActionJson, logger: Logger, subredditName: string, resources: SubredditResources, client: ExtendedSnoowrap, emitter: EventEmitter): Action {
@@ -47,6 +48,8 @@ export function actionFactory
             return new CancelDispatchAction({...config as StructuredFilter<CancelDispatchActionJson>, logger, subredditName, resources, client, emitter})
         case 'contributor':
             return new ContributorAction({...config as StructuredFilter<ContributorActionJson>, logger, subredditName, resources, client, emitter})
+        case 'modnote':
+            return new ModNoteAction({...config as StructuredFilter<ModNoteActionJson>, logger, subredditName, resources, client, emitter})
         default:
             throw new Error('rule "kind" was not recognized.');
     }

src/Action/ApproveAction.ts

@@ -7,6 +7,7 @@ import Comment from "snoowrap/dist/objects/Comment";
 import {RuleResultEntity} from "../Common/Entities/RuleResultEntity";
 import {runCheckOptions} from "../Subreddit/Manager";
 import {ActionTarget, ActionTypes} from "../Common/Infrastructure/Atomic";
+import {asComment, asSubmission} from "../util";
 
 export class ApproveAction extends Action {
 
@@ -29,22 +30,24 @@ export class ApproveAction extends Action {
         const dryRun = this.getRuntimeAwareDryrun(options);
         const touchedEntities = [];
 
-        const realTargets = item instanceof Submission ? ['self'] : this.targets;
+        const realTargets = asSubmission(item) ? ['self'] : this.targets;
+
+        let msg: string[] = [];
 
         for(const target of realTargets) {
             let targetItem = item;
-            if(target !== 'self' && item instanceof Comment) {
+            if(target !== 'self' && asComment(item)) {
                 targetItem = await this.resources.getActivity(this.client.getSubmission(item.link_id));
             }
 
             // @ts-ignore
             if (targetItem.approved) {
-                const msg = `${target === 'self' ? 'Item' : 'Comment\'s parent Submission'} is already approved`;
+                msg.push(`${target === 'self' ? 'Item' : 'Comment\'s parent Submission'} is already approved??`);
                 this.logger.warn(msg);
                 return {
                     dryRun,
                     success: false,
-                    result: msg
+                    result: msg.join('|')
                 }
             }
 
@@ -53,6 +56,9 @@ export class ApproveAction extends Action {
                 if(target !== 'self' && !(targetItem instanceof Submission)) {
                     // @ts-ignore
                     targetItem = await this.client.getSubmission((item as Comment).link_id).fetch();
+                    msg.push(`Approving parent Submission ${targetItem.name}`);
+                } else {
+                    msg.push(`Approving self ${targetItem.name}`);
                 }
                 // @ts-ignore
                 touchedEntities.push(await targetItem.approve());
@@ -70,6 +76,7 @@ export class ApproveAction extends Action {
         }
 
         return {
+            result: msg.join(' | '),
             dryRun,
             success: true,
             touchedEntities

src/Action/BanAction.ts

@@ -6,6 +6,10 @@ import {ActionProcessResult, Footer, RuleResult} from "../Common/interfaces";
 import {RuleResultEntity} from "../Common/Entities/RuleResultEntity";
 import {runCheckOptions} from "../Subreddit/Manager";
 import {ActionTypes} from "../Common/Infrastructure/Atomic";
+import {truncateStringToLength} from "../util";
+
+const truncate = truncateStringToLength(100);
+const truncateLongMessage = truncateStringToLength(200);
 
 export class BanAction extends Action {
 
@@ -41,11 +45,14 @@ export class BanAction extends Action {
         const renderedBody = content === undefined ? undefined : await renderContent(content, item, ruleResults, this.resources.userNotes);
         const renderedContent = renderedBody === undefined ? undefined : `${renderedBody}${await this.resources.generateFooter(item, this.footer)}`;
 
+        const renderedReason = this.reason === undefined ? undefined : truncate(await renderContent(this.reason, item, ruleResults, this.resources.userNotes));
+        const renderedNote = this.note === undefined ? undefined : truncate(await renderContent(this.note, item, ruleResults, this.resources.userNotes));
+
         const touchedEntities = [];
         let banPieces = [];
-        banPieces.push(`Message: ${renderedContent === undefined ? 'None' : `${renderedContent.length > 100 ? `\r\n${renderedContent}` : renderedContent}`}`);
-        banPieces.push(`Reason:  ${this.reason || 'None'}`);
-        banPieces.push(`Note:    ${this.note || 'None'}`);
+        banPieces.push(`Message: ${renderedContent === undefined ? 'None' : `${renderedContent.length > 100 ? `\r\n${truncateLongMessage(renderedContent)}` : renderedContent}`}`);
+        banPieces.push(`Reason:  ${renderedReason || 'None'}`);
+        banPieces.push(`Note:    ${renderedNote || 'None'}`);
         const durText = this.duration === undefined ? 'permanently' : `for ${this.duration} days`;
         this.logger.info(`Banning ${item.author.name} ${durText}${this.reason !== undefined ? ` (${this.reason})` : ''}`);
         this.logger.verbose(`\r\n${banPieces.join('\r\n')}`);
@@ -56,8 +63,8 @@ export class BanAction extends Action {
             const bannedUser = await fetchedSub.banUser({
                 name: fetchedName,
                 banMessage: renderedContent === undefined ? undefined : renderedContent,
-                banReason: this.reason,
-                banNote: this.note,
+                banReason: renderedReason,
+                banNote: renderedNote,
                 duration: this.duration
             });
             touchedEntities.push(bannedUser);
@@ -65,7 +72,7 @@ export class BanAction extends Action {
         return {
             dryRun,
             success: true,
-            result: `Banned ${item.author.name} ${durText}${this.reason !== undefined ? ` (${this.reason})` : ''}`,
+            result: `Banned ${item.author.name} ${durText}${renderedReason !== undefined ? ` (${renderedReason})` : ''}`,
             touchedEntities
         };
     }
@@ -97,7 +104,10 @@ export interface BanActionConfig extends ActionConfig, Footer {
      * */
     message?: string
     /**
-     * Reason for ban.
+     * Reason for ban. Can use Templating.
+     *
+     * If the length expands to more than 100 characters it will truncated with "..."
+     *
      * @maxLength 100
      * @examples ["repeat spam"]
      * */
@@ -110,7 +120,10 @@ export interface BanActionConfig extends ActionConfig, Footer {
      * */
     duration?: number
     /**
-     * A mod note for this ban
+     * A mod note for this ban. Can use Templating.
+     *
+     * If the length expands to more than 100 characters it will truncated with "..."
+     *
      * @maxLength 100
      * @examples ["Sock puppet for u/AnotherUser"]
      * */

src/Action/CancelDispatchAction.ts

@@ -71,12 +71,7 @@ export class CancelDispatchAction extends Action {
                 } else {
                     matchedDispatchIdentifier = this.identifiers.filter(x => x !== null).includes(x.identifier);
                 }
-                const matched = matchedId && matchedDispatchIdentifier;
-                if(matched && x.processing) {
-                    this.logger.debug(`Cannot remove ${isSubmission(x.activity) ? 'Submission' : 'Comment'} ${x.activity.name} because it is currently processing`);
-                    return false;
-                }
-                return matched;
+                return matchedId && matchedDispatchIdentifier;
             });
             let cancelCrit;
             if (this.identifiers === undefined) {

src/Action/ModNoteAction.ts

@@ -0,0 +1,108 @@
+import {ActionJson, ActionConfig, ActionOptions} from "./index";
+import Action from "./index";
+import {Comment} from "snoowrap";
+import {renderContent} from "../Utils/SnoowrapUtils";
+import Submission from "snoowrap/dist/objects/Submission";
+import {ActionProcessResult, RichContent} from "../Common/interfaces";
+import {toModNoteLabel} from "../util";
+import {RuleResultEntity} from "../Common/Entities/RuleResultEntity";
+import {runCheckOptions} from "../Subreddit/Manager";
+import {ActionTypes, ModUserNoteLabel} from "../Common/Infrastructure/Atomic";
+import {ModNote} from "../Subreddit/ModNotes/ModNote";
+
+
+export class ModNoteAction extends Action {
+    content: string;
+    type?: string;
+    allowDuplicate: boolean;
+    referenceActivity: boolean
+
+    constructor(options: ModNoteActionOptions) {
+        super(options);
+        const {type, content = '', allowDuplicate = false, referenceActivity = true} = options;
+        this.type = type;
+        this.content = content;
+        this.allowDuplicate = allowDuplicate;
+        this.referenceActivity = referenceActivity;
+    }
+
+    getKind(): ActionTypes {
+        return 'modnote';
+    }
+
+    protected getSpecificPremise(): object {
+        return {
+            content: this.content,
+            type: this.type,
+            allowDuplicate: this.allowDuplicate,
+            referenceActivity: this.referenceActivity,
+        }
+    }
+
+    async process(item: Comment | Submission, ruleResults: RuleResultEntity[], options: runCheckOptions): Promise<ActionProcessResult> {
+        const dryRun = this.getRuntimeAwareDryrun(options);
+
+        const modLabel = this.type !== undefined ? toModNoteLabel(this.type) : undefined;
+
+        const content = await this.resources.getContent(this.content, item.subreddit);
+        const renderedContent = await renderContent(content, item, ruleResults, this.resources.userNotes);
+        this.logger.verbose(`Note:\r\n(${this.type}) ${renderedContent}`);
+
+        // TODO see what changes are made for bulk fetch of notes before implementing this
+        // https://www.reddit.com/r/redditdev/comments/t8w861/new_mod_notes_api/
+        // if (!this.allowDuplicate) {
+        //     const notes = await this.resources.userNotes.getUserNotes(item.author);
+        //     let existingNote = notes.find((x) => x.link !== null && x.link.includes(item.id));
+        //     if(existingNote === undefined && notes.length > 0) {
+        //         const lastNote = notes[notes.length - 1];
+        //         // possibly notes don't have a reference link so check if last one has same text
+        //         if(lastNote.link === null && lastNote.text === renderedContent) {
+        //             existingNote = lastNote;
+        //         }
+        //     }
+        //     if (existingNote !== undefined && existingNote.noteType === this.type) {
+        //         this.logger.info(`Will not add note because one already exists for this Activity (${existingNote.time.local().format()}) and allowDuplicate=false`);
+        //         return {
+        //             dryRun,
+        //             success: false,
+        //             result: `Will not add note because one already exists for this Activity (${existingNote.time.local().format()}) and allowDuplicate=false`
+        //         };
+        //     }
+        // }
+        if (!dryRun) {
+            await this.resources.addModNote({
+                label: modLabel,
+                note: renderedContent,
+                activity: this.referenceActivity ? item : undefined,
+                subreddit: this.resources.subreddit,
+                user: item.author
+            });
+        }
+        return {
+            success: true,
+            dryRun,
+            result: `${modLabel !== undefined ? `(${modLabel})` : ''} ${renderedContent}`
+        }
+    }
+}
+
+export interface ModNoteActionConfig extends ActionConfig, RichContent {
+    /**
+     * Add Note even if a Note already exists for this Activity
+     * @examples [false]
+     * @default false
+     * */
+    allowDuplicate?: boolean,
+    type?: ModUserNoteLabel
+    referenceActivity?: boolean
+}
+
+export interface ModNoteActionOptions extends Omit<ModNoteActionConfig, 'authorIs' | 'itemIs'>, ActionOptions {
+}
+
+/**
+ * Add a Toolbox User Note to the Author of this Activity
+ * */
+export interface ModNoteActionJson extends ModNoteActionConfig, ActionJson {
+    kind: 'modnote'
+}

src/Action/UserNoteAction.ts

@@ -7,12 +7,12 @@ import Submission from "snoowrap/dist/objects/Submission";
 import {ActionProcessResult, RuleResult} from "../Common/interfaces";
 import {RuleResultEntity} from "../Common/Entities/RuleResultEntity";
 import {runCheckOptions} from "../Subreddit/Manager";
-import {ActionTypes} from "../Common/Infrastructure/Atomic";
+import {ActionTypes, UserNoteType} from "../Common/Infrastructure/Atomic";
 
 
 export class UserNoteAction extends Action {
     content: string;
-    type: string;
+    type: UserNoteType;
     allowDuplicate: boolean;
 
     constructor(options: UserNoteActionOptions) {

src/App.ts

@@ -133,7 +133,7 @@ export class App {
         for (const b of this.bots) {
             if (b.error === undefined) {
                 try {
-                    await b.testClient();
+                    await b.init();
                     await b.buildManagers();
                     await sleep(2000);
                     b.runManagers(causedBy).catch((err) => {

src/Bot/index.ts

@@ -18,7 +18,7 @@ import {
     mergeArr,
     parseBool,
     parseDuration, parseMatchMessage, parseRedditEntity,
-    parseSubredditName, RetryOptions,
+    parseSubredditName, partition, RetryOptions,
     sleep
 } from "../util";
 import {Manager} from "../Subreddit/Manager";
@@ -41,6 +41,8 @@ import {ManagerRunState} from "../Common/Entities/EntityRunState/ManagerRunState
 import {Invokee, PollOn} from "../Common/Infrastructure/Atomic";
 import {FilterCriteriaDefaults} from "../Common/Infrastructure/Filters/FilterShapes";
 import {snooLogWrapper} from "../Utils/loggerFactory";
+import {InfluxClient} from "../Common/Influx/InfluxClient";
+import {Point} from "@influxdata/influxdb-client";
 
 class Bot {
 
@@ -49,6 +51,7 @@ class Bot {
     logs: LogInfo[] = [];
     wikiLocation: string;
     dryRun?: true | undefined;
+    inited: boolean = false;
     running: boolean = false;
     subreddits: string[];
     excludeSubreddits: string[];
@@ -71,6 +74,7 @@ class Bot {
     botName?: string;
     botLink?: string;
     botAccount?: string;
+    botUser?: RedditUser;
     maxWorkers: number;
     startedAt: Dayjs = dayjs();
     sharedStreams: PollOn[] = [];
@@ -90,6 +94,8 @@ class Bot {
 
     config: BotInstanceConfig;
 
+    influxClients: InfluxClient[] = [];
+
     database: DataSource
     invokeeRepo: Repository<InvokeeType>;
     runTypeRepo: Repository<RunStateType>;
@@ -177,8 +183,11 @@ class Bot {
 
         this.logger.stream().on('log', (log: LogInfo) => {
             if(log.bot !== undefined && log.bot === this.getBotName() && log.subreddit === undefined) {
-                const combinedLogs = [log, ...this.logs];
-                this.logs = combinedLogs.slice(0, 301);
+                this.logs.unshift(log);
+                if(this.logs.length > 300) {
+                    // remove all elements starting from the 300th index (301st item)
+                    this.logs.splice(300);
+                }
             }
         });
 
@@ -311,33 +320,22 @@ class Bot {
         }
     }
 
-    async testClient(initial = true) {
-        try {
-            // @ts-ignore
-            await this.client.getMe();
-            this.logger.info('Test API call successful');
-        } catch (err: any) {
-            if (initial) {
-                this.logger.error('An error occurred while trying to initialize the Reddit API Client which would prevent the entire application from running.');
-            }
-            const hint = getExceptionMessage(err, {
-                401: 'Likely a credential is missing or incorrect. Check clientId, clientSecret, refreshToken, and accessToken',
-                400: 'Credentials may have been invalidated manually or by reddit due to behavior',
-            });
-            let msg = `Error occurred while testing Reddit API client${hint !== undefined ? `: ${hint}` : ''}`;
-            this.error = msg;
-            const clientError = new CMError(msg, {cause: err});
-            clientError.logged = true;
-            this.logger.error(clientError);
-            throw clientError;
-        }
-    }
+    async init() {
+
+        if(this.inited) {
+            return;
+        }
+
+        let user: RedditUser;
+        try {
+            user = await this.testClient();
+        } catch(err: any) {
+            this.logger.error('An error occurred while trying to initialize the Reddit API Client which would prevent the Bot from running.');
+            throw err;
+        }
 
-    async buildManagers(subreddits: string[] = []) {
-        let availSubs = [];
-        // @ts-ignore
-        const user = await this.client.getMe().fetch();
         this.cacheManager.botName = user.name;
+        this.botUser = user;
         this.botLink = `https://reddit.com/user/${user.name}`;
         this.botAccount = `u/${user.name}`;
         this.logger.info(`Reddit API Limit Remaining: ${this.client.ratelimitRemaining}`);
@@ -364,35 +362,78 @@ class Bot {
             this.botEntity = b;
         }
 
+        if(this.config.opInflux !== undefined) {
+            this.influxClients.push(this.config.opInflux.childClient(this.logger, {bot: user.name}));
+            if(this.config.influxConfig !== undefined) {
+                const iClient = new InfluxClient(this.config.influxConfig, this.logger, {bot: user.name});
+                await iClient.isReady();
+                this.influxClients.push(iClient);
+            }
+        }
+
+        this.inited = true;
+    }
+
+    // @ts-ignore
+    async testClient(initial = true) {
+        try {
+            // @ts-ignore
+            const user = this.client.getMe().fetch();
+            this.logger.info('Test API call successful');
+            return user;
+        } catch (err: any) {
+            if (initial) {
+                this.logger.error('An error occurred while trying to initialize the Reddit API Client which would prevent the entire application from running.');
+            }
+            const hint = getExceptionMessage(err, {
+                401: 'Likely a credential is missing or incorrect. Check clientId, clientSecret, refreshToken, and accessToken',
+                400: 'Credentials may have been invalidated manually or by reddit due to behavior',
+            });
+            let msg = `Error occurred while testing Reddit API client${hint !== undefined ? `: ${hint}` : ''}`;
+            this.error = msg;
+            const clientError = new CMError(msg, {cause: err});
+            clientError.logged = true;
+            this.logger.error(clientError);
+            throw clientError;
+        }
+    }
+
+    async buildManagers(subreddits: string[] = []) {
+        await this.init();
+
+        this.logger.verbose('Syncing subreddits to moderate with managers...');
+
+        let availSubs: Subreddit[] = [];
+
         let subListing = await this.client.getModeratedSubreddits({count: 100});
         while(!subListing.isFinished) {
             subListing = await subListing.fetchMore({amount: 100});
         }
-        availSubs = subListing.filter(x => x.display_name !== `u_${user.name}`);
+        availSubs = subListing.filter(x => x.display_name !== `u_${this.botUser?.name}`);
 
-        this.logger.info(`u/${user.name} is a moderator of these subreddits: ${availSubs.map(x => x.display_name_prefixed).join(', ')}`);
+        this.logger.verbose(`${this.botAccount} is a moderator of these subreddits: ${availSubs.map(x => x.display_name_prefixed).join(', ')}`);
 
         let subsToRun: Subreddit[] = [];
         const subsToUse = subreddits.length > 0 ? subreddits.map(parseSubredditName) : this.subreddits;
         if (subsToUse.length > 0) {
-            this.logger.info(`Operator-defined subreddit constraints detected (CLI argument or environmental variable), will try to run on: ${subsToUse.join(', ')}`);
-            for (const sub of subsToUse) {
+            this.logger.info(`Operator-specified subreddit constraints detected, will only use these: ${subsToUse.join(', ')}`);
+            const availSubsCI = availSubs.map(x => x.display_name.toLowerCase());
+            const [foundSubs, notFoundSubs] = partition(subsToUse, (aSub) => availSubsCI.includes(aSub.toLowerCase()));
+            if(notFoundSubs.length > 0) {
+                this.logger.warn(`Will not run some operator-specified subreddits because they are not modded by, or do not have appropriate mod permissions for, this bot: ${notFoundSubs.join(', ')}`);
+            }
+
+            for (const sub of foundSubs) {
                 const asub = availSubs.find(x => x.display_name.toLowerCase() === sub.toLowerCase())
-                if (asub === undefined) {
-                    this.logger.warn(`Will not run on ${sub} because is not modded by, or does not have appropriate permissions to mod with, for this client.`);
-                } else {
-                    // @ts-ignore
-                    const fetchedSub = await asub.fetch();
-                    subsToRun.push(fetchedSub);
-                }
+                subsToRun.push(asub as Subreddit);
             }
         } else {
             if(this.excludeSubreddits.length > 0) {
-                this.logger.info(`Will run on all moderated subreddits but own profile and user-defined excluded: ${this.excludeSubreddits.join(', ')}`);
+                this.logger.info(`Will run on all moderated subreddits EXCEPT own profile and operator-defined excluded: ${this.excludeSubreddits.join(', ')}`);
                 const normalExcludes = this.excludeSubreddits.map(x => x.toLowerCase());
                 subsToRun = availSubs.filter(x => !normalExcludes.includes(x.display_name.toLowerCase()));
             } else {
-                this.logger.info(`No user-defined subreddit constraints detected, will run on all moderated subreddits EXCEPT own profile (${this.botAccount})`);
+                this.logger.info(`No operator-defined subreddit constraints detected, will run on all moderated subreddits EXCEPT own profile (${this.botAccount})`);
                 subsToRun = availSubs;
             }
         }
@@ -421,24 +462,60 @@ class Bot {
             }
         }
 
+        let subManagersChanged = false;
 
-        // get configs for subs we want to run on and build/validate them
+        const subsToRunNames = subsToRun.map(x => x.display_name.toLowerCase());
+
+        // first stop and remove any managers with subreddits not in subsToRun
+        // -- this covers scenario where bot is running and mods of a subreddit de-mod the bot
+        // -- or where the include/exclude subs list changed from operator (not yet implemented)
+        if(this.subManagers.length > 0) {
+            let index = 0;
+            for(const manager of this.subManagers) {
+                if(!subsToRunNames.includes(manager.subreddit.display_name.toLowerCase())) {
+                    subManagersChanged = true;
+                    // determine if bot was de-modded
+                    const deModded = !availSubs.some(x => x.display_name.toLowerCase() === manager.subreddit.display_name.toLowerCase());
+                    this.logger.warn(`Stopping and removing manager for ${manager.subreddit.display_name.toLowerCase()} because it is ${deModded ? 'no longer moderated by this bot' : 'not in the list of subreddits to moderate'}`);
+                    await manager.destroy('system', {reason: deModded ? 'No longer moderated by this bot' : 'Subreddit is not in moderated list'});
+                    this.subManagers.splice(index, 1);
+                }
+                index++;
+            }
+        }
+
+        // then create any managers that don't already exist
+        // -- covers init scenario
+        // -- and in-situ adding subreddits IE bot is modded to a new subreddit while CM is running
+        const subsToInit: string[] = [];
         for (const sub of subsToRun) {
-            try {
-                this.subManagers.push(await this.createManager(sub));
-            } catch (err: any) {
+            if(!this.subManagers.some(x => x.subreddit.display_name === sub.display_name)) {
+                subManagersChanged = true;
+                this.logger.info(`Manager for ${sub.display_name_prefixed} not found in existing managers. Creating now...`);
+                subsToInit.push(sub.display_name);
+                try {
+                    this.subManagers.push(await this.createManager(sub));
+                } catch (err: any) {
 
+                }
             }
         }
-        for(const m of this.subManagers) {
+        for(const subName of subsToInit) {
             try {
-                await this.initManager(m);
+                const m = this.subManagers.find(x => x.subreddit.display_name === subName);
+                await this.initManager(m as Manager);
             } catch (err: any) {
 
             }
         }
 
-        this.parseSharedStreams();
+        if(!subManagersChanged) {
+            this.logger.verbose('All managers were already synced!');
+        } else {
+            this.parseSharedStreams();
+        }
+
+        return subManagersChanged;
     }
 
     parseSharedStreams() {
@@ -559,7 +636,7 @@ class Bot {
         }
     }
 
-    async createManager(sub: Subreddit): Promise<Manager> {
+    async createManager(subVal: Subreddit): Promise<Manager> {
         const {
             flowControlDefaults: {
                 maxGotoDepth: botMaxDefault
@@ -570,6 +647,15 @@ class Bot {
             } = {}
         } = this.config;
 
+        let sub = subVal;
+        // make sure the subreddit is fully fetched
+        // @ts-ignore
+        if(subVal._hasFetched === false) {
+            // @ts-ignore
+            sub = await subVal.fetch();
+        }
+
+
         const override = overrides.find(x => {
             const configName = parseRedditEntity(x.name).name;
             if(configName !== undefined) {
@@ -630,6 +716,7 @@ class Bot {
             managerEntity: managerEntity as ManagerEntity,
             statDefaults: (statDefaultsFromOverride ?? databaseStatisticsDefaults) as DatabaseStatisticsOperatorConfig,
             retention,
+            influxClients: this.influxClients,
         });
         // all errors from managers will count towards bot-level retry count
         manager.on('error', async (err) => await this.panicOnRetries(err));
@@ -669,21 +756,6 @@ class Bot {
                 await this.client.getSubreddit(name).acceptModeratorInvite();
                 this.logger.info(`Accepted moderator invite for r/${name}!`);
                 await this.cacheManager.deletePendingSubredditInvite(name);
-                // @ts-ignore
-                const sub = await this.client.getSubreddit(name);
-                this.logger.info(`Attempting to add manager for r/${name}`);
-                try {
-                    const manager =  await this.createManager(sub);
-                    this.logger.info(`Starting manager for r/${name}`);
-                    this.subManagers.push(manager);
-                    await this.initManager(manager);
-                    await manager.start('system', {reason: 'Caused by creation due to moderator invite'});
-                    await this.runSharedStreams();
-                } catch (err: any) {
-                    if (!(err instanceof LoggedError)) {
-                        this.logger.error(err);
-                    }
-                }
             } catch (err: any) {
                 if (err.message.includes('NO_INVITE_FOUND')) {
                     this.logger.warn(`No pending moderation invite for r/${name} was found`);
@@ -742,6 +814,7 @@ class Bot {
     async healthLoop() {
         while (this.running) {
             await sleep(5000);
+            await this.apiHealthCheck();
             if (!this.running) {
                 break;
             }
@@ -757,8 +830,17 @@ class Bot {
             }
             if(now.isSameOrAfter(this.nextHeartbeat)) {
                 try {
-                    await this.heartbeat();
+
+                    // run sanity check to see if there is a service issue
+                    try {
+                        await this.testClient(false);
+                    } catch (err: any) {
+                        throw new SimpleError(`Something isn't right! This could be a Reddit API issue (service is down? buggy??) or an issue with the Bot account. Will not run heartbeat operations and will wait until next heartbeat (${dayjs.duration(this.nextHeartbeat.diff(dayjs())).humanize()}) to try again`);
+                    }
+
                     await this.checkModInvites();
+                    await this.buildManagers();
+                    await this.heartbeat();
                 } catch (err: any) {
                     this.logger.error(`Error occurred during heartbeat check: ${err.message}`);
                 }
@@ -770,6 +852,56 @@ class Bot {
         this.emitter.emit('healthStopped');
     }
 
+    getApiUsageSummary() {
+        const depletion = this.apiEstDepletion === undefined ? 'Not Calculated' : this.apiEstDepletion.humanize();
+        return`API Usage Rolling Avg: ${formatNumber(this.apiRollingAvg)}/s | Est Depletion: ${depletion} (${formatNumber(this.depletedInSecs, {toFixed: 0})} seconds)`;
+    }
+
+    async apiHealthCheck() {
+
+        const rollingSample = this.apiSample.slice(0, 7)
+        rollingSample.unshift(this.client.ratelimitRemaining);
+        this.apiSample = rollingSample;
+        const diff = this.apiSample.reduceRight((acc: number[], curr, index) => {
+            if (this.apiSample[index + 1] !== undefined) {
+                const d = Math.abs(curr - this.apiSample[index + 1]);
+                if (d === 0) {
+                    return [...acc, 0];
+                }
+                return [...acc, d / 10];
+            }
+            return acc;
+        }, []);
+        const diffTotal = diff.reduce((acc, curr) => acc + curr, 0);
+        if(diffTotal === 0 || diff.length === 0) {
+            this.apiRollingAvg = 0;
+        } else {
+            this.apiRollingAvg = diffTotal / diff.length; // api requests per second
+        }
+        this.depletedInSecs = this.apiRollingAvg === 0 ? Number.POSITIVE_INFINITY :  this.client.ratelimitRemaining / this.apiRollingAvg; // number of seconds until current remaining limit is 0
+        // if depletion/api usage is 0 we need a sane value to use here for both displaying in logs as well as for api nanny. 10 years seems reasonable
+        this.apiEstDepletion = dayjs.duration((this.depletedInSecs === Number.POSITIVE_INFINITY ? {years: 10} : {seconds: this.depletedInSecs}));
+
+        if(this.influxClients.length > 0) {
+            const apiMeasure = new Point('apiHealth')
+                .intField('remaining', this.client.ratelimitRemaining)
+                .stringField('nannyMod', this.nannyMode ?? 'none');
+
+            if(this.apiSample.length > 1) {
+                const curr = this.apiSample[0];
+                const last = this.apiSample[1];
+                if(curr <= last) {
+                    apiMeasure.intField('used', last - curr);
+                }
+            }
+
+            for(const iclient of this.influxClients) {
+                await iclient.writePoint(apiMeasure);
+            }
+        }
+
+    }
+
     async retentionCleanup() {
         const now = dayjs();
         if(now.isSameOrAfter(this.nextRetentionCheck)) {
@@ -783,15 +915,8 @@ class Bot {
     }
 
     async heartbeat() {
-        const heartbeat = `HEARTBEAT -- API Remaining: ${this.client.ratelimitRemaining} | Usage Rolling Avg: ~${formatNumber(this.apiRollingAvg)}/s | Est Depletion: ${this.apiEstDepletion === undefined ? 'N/A' : this.apiEstDepletion.humanize()} (${formatNumber(this.depletedInSecs, {toFixed: 0})} seconds)`
-        this.logger.info(heartbeat);
+        this.logger.info(`HEARTBEAT -- ${this.getApiUsageSummary()}`);
 
-        // run sanity check to see if there is a service issue
-        try {
-            await this.testClient(false);
-        } catch (err: any) {
-            throw new SimpleError(`Something isn't right! This could be a Reddit API issue (service is down? buggy??) or an issue with the Bot account. Will not run heartbeat operations and will wait until next heartbeat (${dayjs.duration(this.nextHeartbeat.diff(dayjs())).humanize()}) to try again`);
-        }
         let startedAny = false;
 
         for (const s of this.subManagers) {
@@ -844,6 +969,7 @@ class Bot {
 
     async runApiNanny() {
         try {
+            this.logger.debug(this.getApiUsageSummary());
             this.nextExpiration = dayjs(this.client.ratelimitExpiration);
             const nowish = dayjs().add(10, 'second');
             if (nowish.isAfter(this.nextExpiration)) {
@@ -867,30 +993,12 @@ class Bot {
                 }
                 this.nextExpiration = dayjs(this.client.ratelimitExpiration);
             }
-            const rollingSample = this.apiSample.slice(0, 7)
-            rollingSample.unshift(this.client.ratelimitRemaining);
-            this.apiSample = rollingSample;
-            const diff = this.apiSample.reduceRight((acc: number[], curr, index) => {
-                if (this.apiSample[index + 1] !== undefined) {
-                    const d = Math.abs(curr - this.apiSample[index + 1]);
-                    if (d === 0) {
-                        return [...acc, 0];
-                    }
-                    return [...acc, d / 10];
-                }
-                return acc;
-            }, []);
-            this.apiRollingAvg = diff.reduce((acc, curr) => acc + curr, 0) / diff.length; // api requests per second
-            this.depletedInSecs = this.client.ratelimitRemaining / this.apiRollingAvg; // number of seconds until current remaining limit is 0
-            this.apiEstDepletion = dayjs.duration({seconds: this.depletedInSecs});
-            this.logger.debug(`API Usage Rolling Avg: ${formatNumber(this.apiRollingAvg)}/s | Est Depletion: ${this.apiEstDepletion.humanize()} (${formatNumber(this.depletedInSecs, {toFixed: 0})} seconds)`);
-
 
             let hardLimitHit = false;
-            if (typeof this.hardLimit === 'string') {
+            if (typeof this.hardLimit === 'string' && this.apiEstDepletion !== undefined) {
                 const hardDur = parseDuration(this.hardLimit);
                 hardLimitHit = hardDur.asSeconds() > this.apiEstDepletion.asSeconds();
-            } else {
+            } else if(typeof this.hardLimit === 'number') {
                 hardLimitHit = this.hardLimit > this.client.ratelimitRemaining;
             }
 
@@ -899,7 +1007,6 @@ class Bot {
                     return;
                 }
                 this.logger.info(`Detected HARD LIMIT of ${this.hardLimit} remaining`, {leaf: 'Api Nanny'});
-                this.logger.info(`API Remaining: ${this.client.ratelimitRemaining} | Usage Rolling Avg: ${this.apiRollingAvg}/s | Est Depletion: ${this.apiEstDepletion.humanize()} (${formatNumber(this.depletedInSecs, {toFixed: 0})} seconds)`, {leaf: 'Api Nanny'});
                 this.logger.info(`All subreddit event polling has been paused`, {leaf: 'Api Nanny'});
 
                 for (const m of this.subManagers) {
@@ -916,10 +1023,10 @@ class Bot {
             }
 
             let softLimitHit = false;
-            if (typeof this.softLimit === 'string') {
+            if (typeof this.softLimit === 'string' && this.apiEstDepletion !== undefined) {
                 const softDur = parseDuration(this.softLimit);
                 softLimitHit = softDur.asSeconds() > this.apiEstDepletion.asSeconds();
-            } else {
+            } else if(typeof this.softLimit === 'number') {
                 softLimitHit = this.softLimit > this.client.ratelimitRemaining;
             }
 
@@ -928,7 +1035,6 @@ class Bot {
                     return;
                 }
                 this.logger.info(`Detected SOFT LIMIT of ${this.softLimit} remaining`, {leaf: 'Api Nanny'});
-                this.logger.info(`API Remaining: ${this.client.ratelimitRemaining} | Usage Rolling Avg: ${formatNumber(this.apiRollingAvg)}/s | Est Depletion: ${this.apiEstDepletion.humanize()} (${formatNumber(this.depletedInSecs, {toFixed: 0})} seconds)`, {leaf: 'Api Nanny'});
                 this.logger.info('Trying to detect heavy usage subreddits...', {leaf: 'Api Nanny'});
                 let threshold = 0.5;
                 let offenders = this.subManagers.filter(x => {

src/Check/index.ts

@@ -1,19 +1,18 @@
-import {RuleSet, IRuleSet, RuleSetJson, RuleSetObjectJson, isRuleSetJSON} from "../Rule/RuleSet";
-import {IRule, Rule, RuleJSONConfig} from "../Rule";
-import Action, {ActionConfig, ActionJson, StructuredActionJson} from "../Action";
+import {RuleSet, RuleSetConfigData, RuleSetConfigHydratedData, RuleSetConfigObject} from "../Rule/RuleSet";
+import {Rule} from "../Rule";
+import Action, {ActionConfig} from "../Action";
 import {Logger} from "winston";
-import Snoowrap, {Comment, Submission} from "snoowrap";
+import {Comment, Submission} from "snoowrap";
 import {actionFactory} from "../Action/ActionFactory";
 import {ruleFactory} from "../Rule/RuleFactory";
 import {
     asPostBehaviorOptionConfig,
-    boolToString,
-    createAjvFactory, determineNewResults,
-    FAIL, isRuleSetResult,
+    createAjvFactory,
+    FAIL,
+    isRuleSetResult,
     mergeArr,
     PASS,
     resultsSummary,
-    ruleNamesFromResults,
     truncateStringToLength
 } from "../util";
 import {
@@ -22,19 +21,17 @@ import {
     CheckSummary,
     JoinCondition,
     NotificationEventPayload,
-    PostBehavior, PostBehaviorOptionConfig, PostBehaviorOptionConfigStrong, PostBehaviorStrong,
-    RuleResult,
-    RuleSetResult, UserResultCache
+    PostBehavior,
+    PostBehaviorOptionConfigStrong,
+    PostBehaviorStrong,
+    RuleSetResult
 } from "../Common/interfaces";
 import * as RuleSchema from '../Schema/Rule.json';
 import * as RuleSetSchema from '../Schema/RuleSet.json';
 import * as ActionSchema from '../Schema/Action.json';
-import {
-    ActionJson as ActionTypeJson
-} from "../Common/types";
-import  {SubredditResources} from "../Subreddit/SubredditResources";
+import {SubredditResources} from "../Subreddit/SubredditResources";
 import {ExtendedSnoowrap} from '../Utils/SnoowrapClients';
-import {ActionProcessingError, CheckProcessingError, isRateLimitError} from "../Utils/Errors";
+import {ActionProcessingError, CheckProcessingError} from "../Utils/Errors";
 import {ErrorWithCause, stackWithCauses} from "pony-cause";
 import {runCheckOptions} from "../Subreddit/Manager";
 import EventEmitter from "events";
@@ -47,24 +44,28 @@ import {RunnableBase} from "../Common/RunnableBase";
 import {ActionResultEntity} from "../Common/Entities/ActionResultEntity";
 import {RuleSetResultEntity} from "../Common/Entities/RuleSetResultEntity";
 import {CheckToRuleResultEntity} from "../Common/Entities/RunnableAssociation/CheckToRuleResultEntity";
-import {
-    JoinOperands,
-    PostBehaviorType,
-    RecordOutputType,
-    recordOutputTypes
-} from "../Common/Infrastructure/Atomic";
-import {
-    MinimalOrFullFilter,
-    MinimalOrFullFilterJson
-} from "../Common/Infrastructure/Filters/FilterShapes";
-import {
-    CommentState,
-    SubmissionState,
-} from "../Common/Infrastructure/Filters/FilterCriteria";
+import {JoinOperands, PostBehaviorType, RecordOutputType, recordOutputTypes} from "../Common/Infrastructure/Atomic";
+import {CommentState, SubmissionState,} from "../Common/Infrastructure/Filters/FilterCriteria";
 import {ActivityType} from "../Common/Infrastructure/Reddit";
-import {RunnableBaseJson, RunnableBaseOptions, StructuredRunnableBase} from "../Common/Infrastructure/Runnable";
-import {RuleJson, StructuredRuleObjectJson, StructuredRuleSetObjectJson} from "../Common/Infrastructure/RuleShapes";
-import {ActionObjectJson, StructuredActionObjectJson} from "../Common/Infrastructure/ActionShapes";
+import {
+    RunnableBaseJson,
+    RunnableBaseOptions,
+    StructuredRunnableBase,
+    TypedRunnableBaseData, TypedStructuredRunnableBase
+} from "../Common/Infrastructure/Runnable";
+import {
+    RuleConfigData, RuleConfigHydratedData,
+    RuleConfigObject,
+    StructuredRuleConfigObject,
+    StructuredRuleSetConfigObject
+} from "../Common/Infrastructure/RuleShapes";
+import {
+    ActionConfigData,
+    ActionConfigHydratedData,
+    ActionConfigObject,
+    StructuredActionObjectJson
+} from "../Common/Infrastructure/ActionShapes";
+import {IncludesData} from "../Common/Infrastructure/Includes";
 
 const checkLogName = truncateStringToLength(25);
 
@@ -155,7 +156,7 @@ export abstract class Check extends RunnableBase implements Omit<ICheck, 'postTr
         if(asPostBehaviorOptionConfig(postFail)) {
             const {
                 behavior = 'next',
-                recordTo = false
+                recordTo = ['influx']
             } = postFail;
             let recordStrong: RecordOutputType[] = [];
             if(typeof recordTo === 'boolean') {
@@ -174,7 +175,7 @@ export abstract class Check extends RunnableBase implements Omit<ICheck, 'postTr
         } else {
             this.postFail = {
                 behavior: postFail,
-                recordTo: []
+                recordTo: ['influx']
             }
         }
 
@@ -192,12 +193,12 @@ export abstract class Check extends RunnableBase implements Omit<ICheck, 'postTr
                 let ruleErrors: any = [];
                 if (valid) {
                     const ruleConfig = r;
-                    this.rules.push(new RuleSet({...ruleConfig as StructuredRuleSetObjectJson, logger: this.logger, subredditName, resources: this.resources, client: this.client}));
+                    this.rules.push(new RuleSet({...ruleConfig as StructuredRuleSetConfigObject, logger: this.logger, subredditName, resources: this.resources, client: this.client}));
                 } else {
                     setErrors = ajv.errors;
                     valid = ajv.validate(RuleSchema, r);
                     if (valid) {
-                        this.rules.push(ruleFactory(r as StructuredRuleObjectJson, this.logger, subredditName, this.resources, this.client));
+                        this.rules.push(ruleFactory(r as StructuredRuleConfigObject, this.logger, subredditName, this.resources, this.client));
                     } else {
                         ruleErrors = ajv.errors;
                         const leastErrorType = setErrors.length < ruleErrors ? 'RuleSet' : 'Rule';
@@ -434,7 +435,7 @@ export abstract class Check extends RunnableBase implements Omit<ICheck, 'postTr
                 checkSum.postBehavior = this.postFail.behavior;
             }
 
-            behaviorT = checkSum.triggered ? 'Trigger' : 'Fail';
+            behaviorT = checkResult.triggered ? 'Trigger' : 'Fail';
 
             switch (checkSum.postBehavior.toLowerCase()) {
                 case 'next':
@@ -605,7 +606,7 @@ export interface ICheck extends JoinCondition, PostBehavior, RunnableBaseJson {
 }
 
 export interface CheckOptions extends Omit<ICheck, 'authorIs' | 'itemIs'>, RunnableBaseOptions {
-    rules: Array<StructuredRuleSetObjectJson | StructuredRuleObjectJson>;
+    rules: Array<RuleConfigObject | RuleSetConfigObject>;
     actions: ActionConfig[];
     logger: Logger;
     subredditName: string;
@@ -616,7 +617,15 @@ export interface CheckOptions extends Omit<ICheck, 'authorIs' | 'itemIs'>, Runna
     emitter: EventEmitter
 }
 
-export interface CheckJson extends ICheck {
+/*
+* Can contain actions/rules as:
+*  - full objects
+*  - string to hydrate IE "url:fsdfd"
+*  - named string IE "namedRule"
+*
+* Also can contain itemIs/authorIs as full object or named filter
+* */
+export interface CheckConfigData extends ICheck, RunnableBaseJson {
     /**
      * The type of event (new submission or new comment) this check should be run against
      * @examples ["submission", "comment"]
@@ -631,7 +640,7 @@ export interface CheckJson extends ICheck {
      *
      * **If `rules` is an empty array or not present then `actions` are performed immediately.**
      * */
-    rules?: Array<RuleSetJson | RuleJson>
+    rules?: (RuleSetConfigData | RuleConfigData | string | IncludesData)[]
     /**
      * The `Actions` to run after the check is successfully triggered. ALL `Actions` will run in the order they are listed
      *
@@ -639,7 +648,7 @@ export interface CheckJson extends ICheck {
      *
      * @examples [[{"kind": "comment", "content": "this is the content of the comment", "distinguish": true}, {"kind": "lock"}]]
      * */
-    actions?: Array<ActionTypeJson>
+    actions?: ActionConfigData[]
 
     /**
      * If notifications are configured and this is `true` then an `eventActioned` event will be sent when this check is triggered.
@@ -651,9 +660,49 @@ export interface CheckJson extends ICheck {
     cacheUserResult?: UserResultCacheOptions;
 }
 
-export interface SubmissionCheckJson extends CheckJson {
+export interface SubmissionCheckConfigData extends CheckConfigData, TypedRunnableBaseData<SubmissionState> {
     kind: 'submission'
-    itemIs?: MinimalOrFullFilterJson<SubmissionState>
+}
+
+export interface CommentCheckConfigData extends CheckConfigData, TypedRunnableBaseData<CommentState> {
+    kind: 'comment'
+}
+
+
+/*
+* Can contain actions/rules as:
+*  - full objects
+*  - named string IE "namedRule"
+*
+* Also can contain itemIs/authorIs as full object or named filter
+* */
+export interface CheckConfigHydratedData extends CheckConfigData {
+    rules?: (RuleSetConfigHydratedData | RuleConfigHydratedData)[]
+    actions?: ActionConfigHydratedData[]
+}
+
+export interface SubmissionCheckConfigHydratedData extends CheckConfigHydratedData, TypedRunnableBaseData<SubmissionState> {
+    kind: 'submission'
+}
+
+export interface CommentCheckConfigHydratedData extends CheckConfigHydratedData, TypedRunnableBaseData<CommentState> {
+    kind: 'comment'
+}
+
+/*
+* All actions/rules/filters should now be full objects
+* */
+export interface CheckConfigObject extends Omit<CheckConfigHydratedData, 'itemIs' | 'authorIs'>, StructuredRunnableBase {
+    rules: Array<RuleSetConfigObject | RuleConfigObject>
+    actions: Array<ActionConfigObject>
+}
+
+export interface SubmissionCheckConfigObject extends Omit<CheckConfigObject, 'itemIs' | 'author'>, TypedStructuredRunnableBase<SubmissionState> {
+    kind: 'submission'
+}
+
+export interface CommentCheckConfigObject extends Omit<CheckConfigObject, 'itemIs' | 'author'>, TypedStructuredRunnableBase<CommentState> {
+    kind: 'comment'
 }
 
 /**
@@ -691,33 +740,18 @@ export const userResultCacheDefault: Required<UserResultCacheOptions> = {
     runActions: true,
 }
 
-export interface CommentCheckJson extends CheckJson {
-    kind: 'comment'
-    itemIs?: MinimalOrFullFilterJson<CommentState>
-}
-
-export const asStructuredCommentCheckJson = (val: any): val is CommentCheckStructuredJson => {
+export const asStructuredCommentCheckJson = (val: any): val is CommentCheckConfigObject => {
     return val.kind === 'comment';
 }
 
-export const asStructuredSubmissionCheckJson = (val: any): val is SubmissionCheckStructuredJson => {
+export const asStructuredSubmissionCheckJson = (val: any): val is SubmissionCheckConfigObject => {
     return val.kind === 'submission';
 }
 
-export type CheckStructuredJson = SubmissionCheckStructuredJson | CommentCheckStructuredJson;
-// export interface CheckStructuredJson extends CheckJson {
-//     rules: Array<RuleSetObjectJson | RuleObjectJson>
-//     actions: Array<ActionObjectJson>
-// }
+export type ActivityCheckConfigValue = string | IncludesData | SubmissionCheckConfigData | CommentCheckConfigData;
 
-export interface SubmissionCheckStructuredJson extends Omit<SubmissionCheckJson, 'authorIs' | 'itemIs' | 'rules'>,  StructuredRunnableBase {
-    rules: Array<StructuredRuleSetObjectJson | StructuredRuleObjectJson>
-    actions: Array<ActionObjectJson>
-    itemIs?: MinimalOrFullFilter<SubmissionState>
-}
+export type ActivityCheckConfigData = Exclude<ActivityCheckConfigValue, IncludesData>;
 
-export interface CommentCheckStructuredJson extends Omit<CommentCheckJson, 'authorIs' | 'itemIs' | 'rules'>, StructuredRunnableBase {
-    rules: Array<StructuredRuleSetObjectJson | StructuredRuleObjectJson>
-    actions: Array<ActionObjectJson>
-    itemIs?: MinimalOrFullFilter<CommentState>
-}
+export type ActivityCheckConfigHydratedData = SubmissionCheckConfigHydratedData | CommentCheckConfigHydratedData;
+
+export type ActivityCheckObject = SubmissionCheckConfigObject | CommentCheckConfigObject;

src/Common/Config/ConfigToObjectOptions.ts

@@ -6,4 +6,5 @@ export interface ConfigToObjectOptions {
     location?: string,
     jsonDocFunc?: (content: string, location?: string) => AbstractConfigDocument<OperatorJsonConfig>,
     yamlDocFunc?: (content: string, location?: string) => AbstractConfigDocument<YamlDocument>
+    allowArrays?: boolean
 }

src/Common/Config/ConfigUtil.ts

@@ -19,6 +19,7 @@ export const parseFromJsonOrYamlToObject = (content: string, options?: ConfigToO
         location,
         jsonDocFunc = (content: string, location?: string) => new JsonConfigDocument(content, location),
         yamlDocFunc = (content: string, location?: string) => new YamlConfigDocument(content, location),
+        allowArrays = false,
     } = options || {};
 
     try {

src/Common/Config/Operator/index.ts

@@ -1,6 +1,6 @@
 import YamlConfigDocument from "../YamlConfigDocument";
 import JsonConfigDocument from "../JsonConfigDocument";
-import {YAMLMap, YAMLSeq} from "yaml";
+import {YAMLMap, YAMLSeq, Pair, Scalar} from "yaml";
 import {BotInstanceJsonConfig, OperatorJsonConfig} from "../../interfaces";
 import {assign} from 'comment-json';
 
@@ -15,10 +15,12 @@ export class YamlOperatorConfigDocument extends YamlConfigDocument implements Op
         if (bots === undefined) {
             this.parsed.add({key: 'bots', value: [botData]});
         } else if (botData.name !== undefined) {
-            // overwrite if we find an existing
+            // granularly overwrite (merge) if we find an existing
             const existingIndex = bots.items.findIndex(x => (x as YAMLMap).get('name') === botData.name);
             if (existingIndex !== -1) {
-                this.parsed.setIn(['bots', existingIndex], botData);
+                const botObj = this.parsed.getIn(['bots', existingIndex]) as YAMLMap;
+                const mergedVal = mergeObjectToYaml(botData, botObj);
+                this.parsed.setIn(['bots', existingIndex], mergedVal);
             } else {
                 this.parsed.addIn(['bots'], botData);
             }
@@ -32,6 +34,24 @@ export class YamlOperatorConfigDocument extends YamlConfigDocument implements Op
     }
 }
 
+export const mergeObjectToYaml = (source: object, target: YAMLMap) => {
+    for (const [k, v] of Object.entries(source)) {
+        if (target.has(k)) {
+            const targetProp = target.get(k);
+            if (targetProp instanceof YAMLMap && typeof v === 'object') {
+                const merged = mergeObjectToYaml(v, targetProp);
+                target.set(k, merged)
+            } else {
+                // since target prop and value are not both objects don't bother merging, just overwrite (primitive or array)
+                target.set(k, v);
+            }
+        } else {
+            target.add({key: k, value: v});
+        }
+    }
+    return target;
+}
+
 export class JsonOperatorConfigDocument extends JsonConfigDocument implements OperatorConfigDocumentInterface {
     addBot(botData: BotInstanceJsonConfig) {
         if (this.parsed.bots === undefined) {

src/Common/Entities/ActionPremise.ts

@@ -79,6 +79,7 @@ export class ActionPremise extends TimeAwareRandomBaseEntity  {
             this.active = data.active ?? true;
             this.configHash = objectHash.sha1(data.config);
             this.manager = data.manager;
+            this.managerId = data.manager.id;
             this.name = data.name;
 
             const {

src/Common/Entities/Activity.ts

@@ -1,11 +1,11 @@
-import {Entity, Column, PrimaryGeneratedColumn, ManyToOne, PrimaryColumn, OneToMany, OneToOne, Index} from "typeorm";
+import {Entity, Column, ManyToOne, PrimaryColumn, OneToMany, Index} from "typeorm";
 import {AuthorEntity} from "./AuthorEntity";
 import {Subreddit} from "./Subreddit";
 import {CMEvent} from "./CMEvent";
-import Submission from "snoowrap/dist/objects/Submission";
-import {Comment} from "snoowrap";
 import {asComment, getActivityAuthorName, parseRedditFullname, redditThingTypeToPrefix} from "../../util";
-import {ActivityType} from "../Infrastructure/Reddit";
+import {activityReports, ActivityType, Report, SnoowrapActivity} from "../Infrastructure/Reddit";
+import {ActivityReport} from "./ActivityReport";
+import dayjs, {Dayjs} from "dayjs";
 
 export interface ActivityEntityOptions {
     id: string
@@ -15,6 +15,7 @@ export interface ActivityEntityOptions {
     permalink: string
     author: AuthorEntity
     submission?: Activity
+    reports?: ActivityReport[]
 }
 
 @Entity()
@@ -69,6 +70,9 @@ export class Activity {
     @OneToMany(type => Activity, obj => obj.submission, {nullable: true})
     comments!: Activity[];
 
+    @OneToMany(type => ActivityReport, act => act.activity, {cascade: ['insert'], eager: true})
+    reports: ActivityReport[] | undefined
+
     constructor(data?: ActivityEntityOptions) {
         if(data !== undefined) {
             this.type = data.type;
@@ -78,10 +82,76 @@ export class Activity {
             this.permalink = data.permalink;
             this.author = data.author;
             this.submission = data.submission;
+            this.reports = data.reports !== undefined ? data.reports : undefined;
         }
     }
 
-    static fromSnoowrapActivity(subreddit: Subreddit, activity: (Submission | Comment)) {
+    /**
+     * @param {SnoowrapActivity} activity
+     * @param {Dayjs|undefined} lastKnownStateTimestamp Override the last good state (useful when tracked through polling)
+     * */
+    syncReports(activity: SnoowrapActivity, lastKnownStateTimestamp?: Dayjs) {
+        if(activity.num_reports > 0 && (this.reports === undefined || activity.num_reports !== this.reports.length)) {
+            if(this.reports === undefined) {
+                this.reports = [];
+            }
+            const reports = activityReports(activity);
+            // match up existing reports
+            const usedReportEntities: string[] = [];
+            const unsyncedReports: Report[] = [];
+            for(const r of reports) {
+                const matchedEntity = this.reports.find(x => !usedReportEntities.includes(x.id) && x.matchReport(r));
+                if(matchedEntity !== undefined) {
+                    usedReportEntities.push(matchedEntity.id);
+                } else {
+                    // found an unsynced report
+                    unsyncedReports.push(r);
+                }
+            }
+
+            // ideally we only have one report but it's possible (probable) there are more
+            //
+            // to simplify tracking over time we will spread out the "create time" for each report to be between NOW
+            // and the last recorded report, or if no reports then the create time of the activity
+
+            // -- the assumptions about tracking should be good enough for most users because:
+            // * default poll interval is 30 seconds so even if there are more than one reports in that time the resolution is high enough for accurate usage (most mods will use "> 1 report in 1 minute" or larger timescales)
+            // * for populating existing reports (CM has not been tracking since activity creation) we don't want to bunch up all reports at the timestamp which could create false positives,
+            //   it's more likely that reports would be spread out than all occurring at the same time.
+
+            // TODO additionally, will allow users to specify minimum required granularity to use when filtering by reports over time
+
+            let lastRecordedTime = lastKnownStateTimestamp;
+            if(lastKnownStateTimestamp === undefined) {
+                lastRecordedTime = this.reports.length > 0 ?
+                    // get the latest create date for existing reports
+                    this.reports.reduce((acc, curr) => curr.createdAt.isAfter(acc) ? curr.createdAt : acc, dayjs('2000-1-1'))
+                    // if no reports then use activity create date
+                    : dayjs(activity.created_utc * 1000);
+            }
+
+            // find the amount of time between now and last good timestamp
+            const missingTimespan = dayjs.duration(dayjs().diff(lastRecordedTime));
+            const granularity = Math.floor(missingTimespan.asSeconds());
+
+            // each report will have its create date spaced out (mostly) equally between now and the last good timestamp
+            //
+            // if only one report stick it in exact middle
+            // if more than one than decrease span by 1/4 so that we don't end up having reports dead-on the last timestamp
+            const increment = Math.floor(unsyncedReports.length === 1 ? (granularity / 2) : ((granularity / 1.25) / unsyncedReports.length));
+
+            for(let i = 0; i < unsyncedReports.length; i++) {
+               const r = new ActivityReport({...unsyncedReports[i], activity: this, granularity});
+               r.createdAt = dayjs().subtract(increment * (i + 1), 'seconds');
+               this.reports.push(r);
+            }
+
+            return true;
+        }
+        return false;
+    }
+
+    static fromSnoowrapActivity(subreddit: Subreddit, activity: SnoowrapActivity, lastKnownStateTimestamp?: dayjs.Dayjs | undefined) {
         let submission: Activity | undefined;
         let type: ActivityType = 'submission';
         let content: string;
@@ -99,7 +169,7 @@ export class Activity {
         const author = new AuthorEntity();
         author.name = getActivityAuthorName(activity.author);
 
-        return new Activity({
+        const entity = new Activity({
             id: activity.name,
             subreddit,
             type,
@@ -107,6 +177,10 @@ export class Activity {
             permalink: activity.permalink,
             author,
             submission
-        })
+        });
+
+        entity.syncReports(activity, lastKnownStateTimestamp);
+
+        return entity;
     }
 }

src/Common/Entities/ActivityReport.ts

@@ -0,0 +1,57 @@
+import {
+    Entity,
+    Column,
+    ManyToOne, JoinColumn, AfterLoad,
+} from "typeorm";
+import {Activity} from "./Activity";
+import {ManagerEntity} from "./ManagerEntity";
+import {TimeAwareRandomBaseEntity} from "./Base/TimeAwareRandomBaseEntity";
+import {Report, ReportType} from "../Infrastructure/Reddit";
+
+@Entity()
+export class ActivityReport extends TimeAwareRandomBaseEntity {
+
+    @Column({nullable: false, length: 500})
+    reason!: string
+
+    @Column({nullable: false, length: 20})
+    type!: ReportType
+
+    @Column({nullable: true, length: 100})
+    author?: string
+
+    @Column("int", {nullable: false})
+    granularity: number = 0;
+
+    @ManyToOne(type => Activity, act => act.reports, {cascade: ['update']})
+    @JoinColumn({name: 'activityId'})
+    activity!: Activity;
+
+    @Column({nullable: false, name: 'activityId'})
+    activityId!: string
+
+    constructor(data?: Report & { activity: Activity, granularity: number }) {
+        super();
+        if (data !== undefined) {
+            this.reason = data.reason;
+            this.type = data.type;
+            this.author = data.author;
+            this.activity = data.activity;
+            this.activityId = data.activity.id;
+            this.granularity = data.granularity
+        }
+    }
+
+    matchReport(report: Report): boolean {
+        return this.reason === report.reason
+            && this.type === report.type
+            && this.author === report.author;
+    }
+
+    @AfterLoad()
+    convertPrimitives() {
+        if(this.author === null) {
+            this.author = undefined;
+        }
+    }
+}

src/Common/Entities/DispatchedEntity.ts

@@ -152,6 +152,9 @@ export class DispatchedEntity extends TimeAwareRandomBaseEntity {
 
     async toActivityDispatch(client: ExtendedSnoowrap): Promise<ActivityDispatch> {
         const redditThing = parseRedditFullname(this.activityId);
+        if(redditThing === undefined) {
+            throw new Error(`Could not parse reddit ID from value '${this.activityId}'`);
+        }
         let activity: Comment | Submission;
         if (redditThing?.type === 'comment') {
             // @ts-ignore
@@ -161,12 +164,12 @@ export class DispatchedEntity extends TimeAwareRandomBaseEntity {
             activity = await client.getSubmission(redditThing.id);
         }
         activity.author = new RedditUser({name: this.author}, client, false);
+        activity.id  = redditThing.id;
         return {
             id: this.id,
             queuedAt: this.createdAt,
             activity,
             delay: this.delay,
-            processing: false,
             action: this.action,
             goto: this.goto,
             onExistingFound: this.onExistingFound,

src/Common/Entities/RulePremise.ts

@@ -83,6 +83,7 @@ export class RulePremise extends TimeAwareRandomBaseEntity {
             this.active = data.active ?? true;
             this.configHash = objectHash.sha1(data.config);
             this.manager = data.manager;
+            this.managerId = data.manager.id;
             this.name = data.name;
 
             const {

src/Common/ImageData.ts

@@ -2,7 +2,6 @@ import fetch from "node-fetch";
 import {Submission} from "snoowrap/dist/objects";
 import {URL} from "url";
 import {absPercentDifference, getSharpAsync, isValidImageURL} from "../util";
-import sizeOf from "image-size";
 import {Sharp} from "sharp";
 import {blockhash} from "./blockhash/blockhash";
 import {SimpleError} from "../Utils/Errors";

src/Common/Influx/InfluxClient.ts

@@ -0,0 +1,115 @@
+import {InfluxConfig} from "./interfaces";
+import {InfluxDB, Point, WriteApi, setLogger} from "@influxdata/influxdb-client";
+import {HealthAPI} from "@influxdata/influxdb-client-apis";
+import dayjs, {Dayjs} from "dayjs";
+import {Logger} from "winston";
+import {mergeArr} from "../../util";
+import {CMError} from "../../Utils/Errors";
+
+export interface InfluxClientConfig extends InfluxConfig {
+    client?: InfluxDB
+    ready?: boolean
+}
+
+export class InfluxClient {
+    config: InfluxConfig;
+    client: InfluxDB;
+    write: WriteApi;
+    health: HealthAPI;
+
+    tags: Record<string, string>;
+
+    logger: Logger;
+
+    ready: boolean;
+    lastReadyAttempt: Dayjs | undefined;
+
+    constructor(config: InfluxClientConfig, logger: Logger, tags: Record<string, string> = {}) {
+
+        const {client, ready = false, ...rest} = config;
+
+        this.logger = logger.child({
+            labels: ['Influx']
+        }, mergeArr);
+
+        this.config = rest;
+        this.ready = ready;
+        if(client !== undefined) {
+            this.client = client;
+        } else {
+           this.client = InfluxClient.createClient(this.config);
+           setLogger(this.logger);
+        }
+        this.write = this.client.getWriteApi(config.credentials.org, config.credentials.bucket, 'ms');
+        this.tags = tags;
+        this.write.useDefaultTags(tags);
+        this.health = new HealthAPI(this.client);
+    }
+
+    async isReady() {
+        if (this.ready) {
+            return true;
+        }
+        if (this.lastReadyAttempt === undefined || dayjs().diff(this.lastReadyAttempt, 's') >= 10) {
+            if (!(await this.testConnection())) {
+                this.logger.warn('Influx endpoint is not ready');
+            } else {
+                this.ready = true;
+            }
+        } else {
+            this.logger.debug(`Influx endpoint testing throttled. Waiting another ${10 - dayjs().diff(this.lastReadyAttempt, 's')} seconds`);
+        }
+        return this.ready;
+    }
+
+    async testConnection() {
+        try {
+            const result = await this.health.getHealth();
+            if (result.status === 'fail') {
+                return false;
+            }
+            return true;
+        } catch (e: any) {
+            this.logger.error(new CMError(`Testing health of Influx endpoint failed`, {cause: e, isSerious: false}));
+            return false;
+        }
+    }
+
+    async writePoint(data: Point | Point[]) {
+        if (await this.isReady()) {
+            if (Array.isArray(data)) {
+                this.write.writePoints(data);
+            } else {
+                this.write.writePoint(data);
+            }
+        }
+    }
+
+    async flush() {
+        if (await this.isReady()) {
+            try {
+                await this.write.flush(true);
+            } catch (e: any) {
+                this.logger.error(new CMError('Failed to flush data to Influx', {cause: e}));
+            }
+        }
+    }
+
+    static createClient(config: InfluxConfig): InfluxDB {
+        return new InfluxDB({
+            url: config.credentials.url,
+            token: config.credentials.token,
+            writeOptions: {
+                defaultTags: config.defaultTags
+            }
+        });
+    }
+
+    childClient(logger: Logger, tags: Record<string, string> = {}) {
+        return new InfluxClient({
+            ...this.config,
+            client: this.client,
+            ready: this.ready
+        }, logger, {...this.tags, ...tags});
+    }
+}

src/Common/Influx/interfaces.ts

@@ -0,0 +1,13 @@
+import {InfluxDB, WriteApi} from "@influxdata/influxdb-client/dist";
+
+export interface InfluxConfig {
+    credentials: InfluxCredentials
+    defaultTags?: Record<string, string>
+}
+
+export interface InfluxCredentials {
+    url: string
+    token: string
+    org: string
+    bucket: string
+}

src/Common/Infrastructure/ActionShapes.ts

@@ -1,5 +1,8 @@
 import {StructuredRunnableBase} from "./Runnable";
 import {ActionJson} from "../types";
+import {IncludesData} from "./Includes";
 
-export type ActionObjectJson = Exclude<ActionJson, string>;
-export type StructuredActionObjectJson = Omit<ActionObjectJson, 'authorIs' | 'itemIs'> & StructuredRunnableBase
+export type ActionConfigData = ActionJson;
+export type ActionConfigHydratedData = Exclude<ActionConfigData, IncludesData>;
+export type ActionConfigObject = Exclude<ActionConfigHydratedData, string>;
+export type StructuredActionObjectJson = Omit<ActionConfigObject, 'authorIs' | 'itemIs'> & StructuredRunnableBase

src/Common/Infrastructure/Atomic.ts

@@ -185,4 +185,95 @@ export type ActionTypes =
     | 'userflair'
     | 'dispatch'
     | 'cancelDispatch'
-    | 'contributor';
+    | 'contributor'
+    | 'modnote';
+
+/**
+ * Test the calculated VADER sentiment (compound) score for an Activity using this comparison. Can be either a numerical or natural language
+ *
+ * Sentiment values range from extremely negative to extremely positive in a numerical range of -1 to +1:
+ *
+ * * -0.6 => extremely negative
+ * * -0.3 => very negative
+ * * -0.1 => negative
+ * *    0 => neutral
+ * *  0.1 => positive
+ * *  0.3 => very positive
+ * *  0.6 => extremely positive
+ *
+ * The below examples are all equivocal. You can use either set of values as the value for `sentiment` (numerical comparisons or natural langauge)
+ *
+ * * `>= 0.1` = `is positive`
+ * * `<= 0.3` = `is very negative`
+ * * `< 0.1` = `is not positive`
+ * * `> -0.3` = `is not very negative`
+ *
+ * Special case:
+ *
+ * * `is neutral` equates to `> -0.1 and < 0.1`
+ * * `is not neutral` equates to `< -0.1 or > 0.1`
+ *
+ * ContextMod uses a normalized, weighted average from these sentiment tools:
+ *
+ * * NLP.js (english, french, german, and spanish) https://github.com/axa-group/nlp.js/blob/master/docs/v3/sentiment-analysis.md
+ * * (english only) vaderSentiment-js https://github.com/vaderSentiment/vaderSentiment-js/
+ * * (english only) wink-sentiment https://github.com/winkjs/wink-sentiment
+ *
+ * More about the sentiment algorithms used:
+ * * VADER https://github.com/cjhutto/vaderSentiment
+ * * AFINN http://corpustext.com/reference/sentiment_afinn.html
+ * * Senticon https://ieeexplore.ieee.org/document/8721408
+ * * Pattern https://github.com/clips/pattern
+ * * wink https://github.com/winkjs/wink-sentiment
+ *
+ * @pattern ((>|>=|<|<=)\s*(-?\d?\.?\d+))|((not)?\s*(very|extremely)?\s*(positive|neutral|negative))
+ * @examples ["is negative", "> 0.2"]
+ * */
+export type VaderSentimentComparison = string;
+
+export type ModUserNoteLabel =
+    'BOT_BAN'
+    | 'PERMA_BAN'
+    | 'BAN'
+    | 'ABUSE_WARNING'
+    | 'SPAM_WARNING'
+    | 'SPAM_WATCH'
+    | 'SOLID_CONTRIBUTOR'
+    | 'HELPFUL_USER';
+
+export const modUserNoteLabels = ['BOT_BAN', 'PERMA_BAN', 'BAN', 'ABUSE_WARNING', 'SPAM_WARNING', 'SPAM_WATCH', 'SOLID_CONTRIBUTOR', 'HELPFUL_USER'];
+
+export type ModActionType =
+    'INVITE' |
+    'NOTE' |
+    'REMOVAL' |
+    'SPAM' |
+    'APPROVAL';
+
+export type UserNoteType =
+    'gooduser' |
+    'spamwatch' |
+    'spamwarn' |
+    'abusewarn' |
+    'ban' |
+    'permban' |
+    'botban' |
+    string;
+
+export const userNoteTypes = ['gooduser', 'spamwatch', 'spamwarn', 'abusewarn', 'ban', 'permban', 'botban'];
+
+export type ConfigFragmentValidationFunc = (data: object, fetched: boolean) => boolean;
+
+export interface WikiContext {
+    wiki: string
+    subreddit?: string
+}
+
+export interface ExternalUrlContext {
+    url: string
+}
+
+export interface UrlContext {
+    value: string
+    context: WikiContext | ExternalUrlContext
+}

src/Common/Infrastructure/Comparisons.ts

@@ -1,15 +1,212 @@
 import {StringOperator} from "./Atomic";
 import {Duration} from "dayjs/plugin/duration";
+import InvalidRegexError from "../../Utils/InvalidRegexError";
+import dayjs, {Dayjs, OpUnitType} from "dayjs";
+import {CMError, SimpleError} from "../../Utils/Errors";
+import {escapeRegex, parseDuration, parseDurationFromString, parseStringToRegex} from "../../util";
+import {ReportType} from "./Reddit";
 
 export interface DurationComparison {
     operator: StringOperator,
     duration: Duration
 }
 
-export interface GenericComparison {
+export interface GenericComparison extends HasDisplayText {
     operator: StringOperator,
     value: number,
     isPercent: boolean,
     extra?: string,
+    groups?: Record<string, string>
     displayText: string,
+    duration?: Duration
+    durationText?: string
+}
+
+export interface HasDisplayText {
+    displayText: string
+}
+
+export interface RangedComparison extends HasDisplayText {
+    range: [number, number]
+    not: boolean
+}
+
+export const asGenericComparison = (val: any): val is GenericComparison => {
+    return typeof val === 'object' && 'value' in val;
+}
+
+export const GENERIC_VALUE_COMPARISON = /^\s*(?<opStr>>|>=|<|<=)\s*(?<value>-?\d?\.?\d+)(?<extra>\s+.*)*$/
+export const GENERIC_VALUE_COMPARISON_URL = 'https://regexr.com/60dq4';
+export const parseGenericValueComparison = (val: string, options?: {
+    requireDuration?: boolean,
+    reg?: RegExp
+}): GenericComparison => {
+
+    const {
+        requireDuration = false,
+        reg = GENERIC_VALUE_COMPARISON,
+    } = options || {};
+
+    const matches = val.match(reg);
+
+    if (matches === null) {
+        throw new InvalidRegexError(reg, val)
+    }
+
+    const groups = matches.groups as any;
+
+    let duration: Duration | undefined;
+    let durationText: string | undefined;
+
+    try {
+        const durationResult = parseDurationFromString(val, false);
+        if(durationResult.length > 1) {
+            throw new SimpleError(`Must only have one Duration value, found ${durationResult.length} in: ${val}`);
+        }
+        duration = durationResult[0].duration;
+        durationText = durationResult[0].original;
+    } catch (e) {
+        // if it returns an invalid regex just means they didn't
+        if (requireDuration || !(e instanceof InvalidRegexError)) {
+            throw e;
+        }
+    }
+
+    const displayParts = [`${groups.opStr} ${groups.value}`];
+    const hasPercent = typeof groups.percent === 'string' && groups.percent.trim() !== '';
+    if(hasPercent) {
+        displayParts.push('%');
+    }
+
+    const {
+        opStr,
+        value,
+        percent,
+        extra,
+        ...rest
+    } = matches.groups || {};
+
+    const extraGroups: Record<string,string> = {};
+    let hasExtraGroups = false;
+
+    for(const [k,v] of Object.entries(rest)) {
+        if(typeof v === 'string' && v.trim() !== '') {
+            extraGroups[k] = v;
+            hasExtraGroups = true;
+        }
+    }
+
+    return {
+        operator: groups.opStr as StringOperator,
+        value: Number.parseFloat(groups.value),
+        isPercent: hasPercent,
+        extra: groups.extra,
+        groups: hasExtraGroups ? extraGroups : undefined,
+        displayText: displayParts.join(''),
+        duration,
+        durationText,
+    }
+}
+const GENERIC_VALUE_PERCENT_COMPARISON = /^\s*(?<opStr>>|>=|<|<=)\s*(?<value>\d+)\s*(?<percent>%)?(?<extra>.*)$/
+const GENERIC_VALUE_PERCENT_COMPARISON_URL = 'https://regexr.com/60a16';
+export const parseGenericValueOrPercentComparison = (val: string, options?: {requireDuration: boolean}): GenericComparison => {
+    return parseGenericValueComparison(val, {...(options ?? {}), reg: GENERIC_VALUE_PERCENT_COMPARISON});
+}
+/**
+ * Named groups: operator, time, unit
+ * */
+const DURATION_COMPARISON_REGEX: RegExp = /^\s*(?<opStr>>|>=|<|<=)\s*(?<time>\d+)\s*(?<unit>days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)\s*$/;
+const DURATION_COMPARISON_REGEX_URL = 'https://regexr.com/609n8';
+export const parseDurationComparison = (val: string): DurationComparison => {
+    const result = parseGenericValueComparison(val, {requireDuration: true});
+    if(result.isPercent) {
+        throw new InvalidRegexError(DURATION_COMPARISON_REGEX, val, DURATION_COMPARISON_REGEX_URL, 'Duration comparison value cannot be a percentage');
+    }
+    if(result.value < 0) {
+        throw new InvalidRegexError(DURATION_COMPARISON_REGEX, val, DURATION_COMPARISON_REGEX_URL,'Duration value cannot be negative');
+    }
+    return {
+        operator: result.operator as StringOperator,
+        duration: result.duration as Duration
+    }
+}
+export const dateComparisonTextOp = (val1: Dayjs, strOp: StringOperator, val2: Dayjs, granularity?: OpUnitType): boolean => {
+    switch (strOp) {
+        case '>':
+            return val1.isBefore(val2, granularity);
+        case '>=':
+            return val1.isSameOrBefore(val2, granularity);
+        case '<':
+            return val1.isAfter(val2, granularity);
+        case '<=':
+            return val1.isSameOrAfter(val2, granularity);
+        default:
+            throw new Error(`${strOp} was not a recognized operator`);
+    }
+}
+export const compareDurationValue = (comp: DurationComparison, date: Dayjs) => {
+    const dateToCompare = dayjs().subtract(comp.duration.asSeconds(), 'seconds');
+    return dateComparisonTextOp(date, comp.operator, dateToCompare);
+}
+export const comparisonTextOp = (val1: number, strOp: string, val2: number): boolean => {
+    switch (strOp) {
+        case '>':
+            return val1 > val2;
+        case '>=':
+            return val1 >= val2;
+        case '<':
+            return val1 < val2;
+        case '<=':
+            return val1 <= val2;
+        default:
+            throw new Error(`${strOp} was not a recognized operator`);
+    }
+}
+
+export interface ReportComparison extends Omit<GenericComparison, 'groups'> {
+    reportType?: ReportType
+    reasonRegex?: RegExp
+    reasonMatch?: string
+}
+
+const REPORT_COMPARISON = /^\s*(?<opStr>>|>=|<|<=)\s*(?<value>\d+)(?<percent>\s*%)?(?:\s+(?<reportType>mods?|users?))?(?:\s+(?<reasonMatch>["'].*["']|\/.*\/))?.*(?<time>\d+)?\s*(?<unit>days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)?\s*$/i
+const REPORT_REASON_LITERAL = /["'](.*)["']/i
+export const parseReportComparison = (str: string): ReportComparison => {
+    const generic = parseGenericValueComparison(str, {reg: REPORT_COMPARISON});
+
+
+    const {
+        groups: {
+            reportType,
+            reasonMatch
+        } = {},
+        ...rest
+    } = generic;
+
+    const result: ReportComparison = {...rest, reasonMatch};
+
+    if(reportType !== undefined) {
+        if(reportType.toLocaleLowerCase().includes('mod')) {
+            result.reportType = 'mod' as ReportType;
+        } else if (reportType.toLocaleLowerCase().includes('user')) {
+            result.reportType = 'user' as ReportType;
+        }
+    }
+    if(reasonMatch !== undefined) {
+        const literalMatch = reasonMatch.match(REPORT_REASON_LITERAL);
+        if(literalMatch !== null) {
+            const cleanLiteralMatch = `/.*${escapeRegex(literalMatch[1].trim())}.*/`;
+            result.reasonRegex = parseStringToRegex(cleanLiteralMatch, 'i');
+            if(result.reasonRegex === undefined) {
+                throw new CMError(`Could not convert reason match value to Regex: ${cleanLiteralMatch}`, {isSerious: false})
+            }
+        } else {
+            result.reasonRegex = parseStringToRegex(reasonMatch, 'i');
+            if(result.reasonRegex === undefined) {
+                throw new CMError(`Could not convert reason match value to Regex: ${reasonMatch}`, {isSerious: false})
+            }
+        }
+    }
+
+    return result;
 }

src/Common/Infrastructure/Filters/FilterCriteria.ts

@@ -1,4 +1,14 @@
-import {CompareValue, CompareValueOrPercent, DurationComparor, ModeratorNameCriteria, ModeratorNames} from "../Atomic";
+import {
+    CompareValue,
+    CompareValueOrPercent,
+    DurationComparor,
+    ModeratorNameCriteria,
+    ModeratorNames, ModActionType,
+    ModUserNoteLabel
+} from "../Atomic";
+import {ActivityType} from "../Reddit";
+import {GenericComparison, parseGenericValueComparison} from "../Comparisons";
+import {parseStringToRegexOrLiteralSearch} from "../../../util";
 
 /**
  * Different attributes a `Subreddit` can be in. Only include a property if you want to check it.
@@ -55,42 +65,40 @@ export const defaultStrongSubredditCriteriaOptions = {
 
 export type FilterCriteriaDefaultBehavior = 'replace' | 'merge';
 
-export interface UserNoteCriteria {
-    /**
-     * User Note type key to search for
-     * @examples ["spamwarn"]
-     * */
-    type: string;
+export interface UserSubredditHistoryCriteria {
     /**
      * Number of occurrences of this type. Ignored if `search` is `current`
      *
      * A string containing a comparison operator and/or a value to compare number of occurrences against
      *
-     * The syntax is `(< OR > OR <= OR >=) <number>[percent sign] [ascending|descending]`
+     * The syntax is `(< OR > OR <= OR >=) <number>[percent sign] [in timeRange] [ascending|descending]`
+     *
+     * If `timeRange` is given then only notes/mod actions that occur between timeRange and NOW will be returned. `timeRange` is ignored if search is `current`
      *
      * @examples [">= 1"]
      * @default ">= 1"
-     * @pattern ^\s*(?<opStr>>|>=|<|<=)\s*(?<value>\d+)\s*(?<percent>%?)\s*(?<extra>asc.*|desc.*)*$
+     * @pattern ^\s*(?<opStr>>|>=|<|<=)\s*(?<value>\d+)\s*(?<percent>%?)\s*(?<duration>in\s+\d+\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?))?\s*(?<extra>asc.*|desc.*)*$
      * */
     count?: string;
 
     /**
-     * How to test the notes for this Author:
+     * How to test the Toolbox Notes or Mod Actions for this Author:
      *
      * ### current
      *
-     * Only the most recent note is checked for `type`
+     * Only the most recent note is checked for criteria
      *
      * ### total
      *
-     * The `count` comparison of `type` must be found within all notes
+     * `count` comparison of mod actions/notes must be found within all history
      *
      * * EX `count: > 3`   => Must have more than 3 notes of `type`, total
      * * EX `count: <= 25%` => Must have 25% or less of notes of `type`, total
+     * * EX: `count: > 3 in 1 week` => Must have more than 3 notes within the last week
      *
      * ### consecutive
      *
-     * The `count` **number** of `type` notes must be found in a row.
+     * The `count` **number** of mod actions/notes must be found in a row.
      *
      * You may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`
      *
@@ -104,7 +112,126 @@ export interface UserNoteCriteria {
     search?: 'current' | 'consecutive' | 'total'
 }
 
-export const authorCriteriaProperties = ['name', 'flairCssClass', 'flairText', 'flairTemplate', 'isMod', 'userNotes', 'age', 'linkKarma', 'commentKarma', 'totalKarma', 'verified', 'shadowBanned', 'description', 'isContributor'];
+export interface UserNoteCriteria extends UserSubredditHistoryCriteria {
+    /**
+     * User Note type key to search for
+     * @examples ["spamwarn"]
+     * */
+    type: string;
+}
+
+export interface ModActionCriteria extends UserSubredditHistoryCriteria {
+    type?: ModActionType | ModActionType[]
+    activityType?: ActivityType | ActivityType[]
+}
+
+export interface FullModActionCriteria extends Omit<ModActionCriteria, 'count'> {
+    type?: ModActionType[]
+    count?: GenericComparison
+    activityType?: ActivityType[]
+}
+
+export interface ModNoteCriteria extends ModActionCriteria {
+    noteType?: ModUserNoteLabel | ModUserNoteLabel[]
+    note?: string | string[]
+}
+
+export interface FullModNoteCriteria extends FullModActionCriteria, Omit<ModNoteCriteria, 'note' | 'count' | 'type' | 'activityType'> {
+    noteType?: ModUserNoteLabel[]
+    note?: RegExp[]
+}
+
+const arrayableModNoteProps = ['activityType','noteType','note'];
+
+export const asModNoteCriteria = (val: any): val is ModNoteCriteria => {
+    return val !== null && typeof val === 'object' && ('noteType' in val || 'note' in val);
+}
+
+export const toFullModNoteCriteria = (val: ModNoteCriteria): FullModNoteCriteria => {
+
+    const result = Object.entries(val).reduce((acc: FullModNoteCriteria, curr) => {
+        const [k,v] = curr;
+
+        if(v === undefined) {
+            return acc;
+        }
+
+        const rawVal = arrayableModNoteProps.includes(k) && !Array.isArray(v) ? [v] : v;
+
+        switch(k) {
+            case 'search':
+                acc.search = rawVal;
+                break;
+            case 'count':
+                acc.count = parseGenericValueComparison(rawVal);
+                break;
+            case 'activityType':
+            case 'noteType':
+                acc[k] = rawVal;
+                break;
+            case 'note':
+                acc[k] = rawVal.map((x: string) => parseStringToRegexOrLiteralSearch(x))
+        }
+
+        return acc;
+    }, {});
+
+    result.type = ['NOTE'];
+    return result;
+}
+
+
+export interface ModLogCriteria extends ModActionCriteria {
+    action?: string | string[]
+    details?: string | string[]
+    description?: string | string[]
+}
+
+export interface FullModLogCriteria extends FullModActionCriteria, Omit<ModLogCriteria, 'action' | 'details' | 'description' | 'count' | 'type' | 'activityType'> {
+    action?: RegExp[]
+    details?: RegExp[]
+    description?: RegExp[]
+}
+
+const arrayableModLogProps = ['type','activityType','action','description','details', 'type'];
+
+export const asModLogCriteria = (val: any): val is ModLogCriteria => {
+    return val !== null && typeof val === 'object' && !asModNoteCriteria(val) && ('action' in val || 'details' in val || 'description' in val || 'activityType' in val || 'search' in val || 'count' in val || 'type' in val);
+}
+
+export const toFullModLogCriteria = (val: ModLogCriteria): FullModLogCriteria => {
+
+    return Object.entries(val).reduce((acc: FullModLogCriteria, curr) => {
+        const [k,v] = curr;
+
+        if(v === undefined) {
+            return acc;
+        }
+
+        const rawVal = arrayableModLogProps.includes(k) && !Array.isArray(v) ? [v] : v;
+
+        switch(k) {
+            case 'search':
+                acc.search = rawVal;
+                break;
+            case 'count':
+                acc.count = parseGenericValueComparison(rawVal);
+                break;
+            case 'activityType':
+            case 'type':
+                acc[k as keyof FullModLogCriteria] = rawVal;
+                break;
+            case 'action':
+            case 'description':
+            case 'details':
+                acc[k as keyof FullModLogCriteria] = rawVal.map((x: string) => parseStringToRegexOrLiteralSearch(x))
+        }
+
+        return acc;
+    }, {});
+}
+
+export const authorCriteriaProperties = ['name', 'flairCssClass', 'flairText', 'flairTemplate', 'isMod', 'userNotes', 'modActions', 'age', 'linkKarma', 'commentKarma', 'totalKarma', 'verified', 'shadowBanned', 'description', 'isContributor'];
 
 /**
  * Criteria with which to test against the author of an Activity. The outcome of the test is based on:
@@ -159,6 +286,8 @@ export interface AuthorCriteria {
      * */
     userNotes?: UserNoteCriteria[]
 
+    modActions?: (ModNoteCriteria | ModLogCriteria)[]
+
     /**
      * Test the age of the Author's account (when it was created) against this comparison
      *
@@ -228,7 +357,35 @@ export interface AuthorCriteria {
      * Is the author an approved user (contributor)?
      * */
     isContributor?: boolean
-} // properties calculated/derived by CM -- not provided as plain values by reddit
+}
+
+/**
+ * When testing AuthorCriteria test properties in order of likelihood to require an API call to complete
+ * */
+export const orderedAuthorCriteriaProps: (keyof AuthorCriteria)[] = [
+    'name', // never needs an api call, returned/cached with activity info
+    // none of these normally need api calls unless activity is a skeleton generated by CM (not normal)
+    // all are part of cached activity data
+    'flairCssClass',
+    'flairText',
+    'flairTemplate',
+    // usernotes are cached longer than author by default (5 min vs 60 seconds)
+    'userNotes',
+    // requires fetching/getting cached author.
+    // If fetching and user is shadowbanned none of the individual author data below will be retrievable either so always do this first
+    'shadowBanned',
+    // individual props require fetching/getting cached
+    'age',
+    'linkKarma',
+    'commentKarma',
+    'totalKarma',
+    'verified',
+    'description',
+    'isMod', // requires fetching mods for subreddit
+    'isContributor', // requires fetching contributors for subreddit
+    'modActions', // requires fetching mod notes/actions for author (shortest cache TTL)
+];
+
 export interface ActivityState {
     /**
      * * true/false => test whether Activity is removed or not
@@ -248,20 +405,40 @@ export interface ActivityState {
     approved?: boolean | ModeratorNames | ModeratorNames[] | ModeratorNameCriteria
     score?: CompareValue
     /**
-     * A string containing a comparison operator and a value to compare against
+     * A string containing a comparison operator, a value to compare against, an (optional) report type filter, an (optional) qualifier for report reason, and an (optional) time constraint
      *
-     * The syntax is `(< OR > OR <= OR >=) <number>`
+     * The syntax is `(< OR > OR <= OR >=) number[%] [type] [reasonQualifier] [timeValue] [timeUnit]`
+     *
+     * If only comparison and number is given then defaults to TOTAL reports on an Activity.
      *
      * * EX `> 2`  => greater than 2 total reports
      *
-     * Defaults to TOTAL reports on an Activity. Suffix the value with the report type to check that type:
+     * Type (optional) determines which type of reports to look at:
      *
-     * * EX `> 3 mod` => greater than 3 mod reports
-     * * EX `>= 1 user` => greater than 1 user report
+     * * `mod` -- mod reports
+     *   * EX `> 3 mod` => greater than 3 mod reports
+     * * `user` -- user reports
+     *   * EX `> 3 user` => greater than 3 user reports
      *
-     * @pattern ^\s*(>|>=|<|<=)\s*(\d+)\s*(%?)(.*)$
+     * Report reason qualifiers can be:
+     *
+     * * enclosed double or single quotes -- report reason contains
+     *   * EX `> 1 "misinformation" => greater than 1 report with reason containing "misinformation"
+     * * enclosed in backslashes -- match regex
+     *   * EX `> 1 \harassment towards .*\` => greater than 1 report with reason matching regex \harassment towards .*\
+     *
+     * Type and reason qualifiers can be used together:
+     *
+     * EX `> 2 user "misinformation" => greater than 2 user reports with reasons containing "misinformation"
+     *
+     * The time constraint filter reports created between NOW and [timeConstraint] in the past:
+     *
+     * * `> 3 in 30 minutes` => more than 3 reports created between NOW and 30 minutes ago
+     * * `> 2 user "misinformation" in 2 hours` => more than 2 user reports containing "misinformation" created between NOW and 2 hours ago
+     *
+     * @pattern ^\s*(>|>=|<|<=)\s*(\d+)(\s*%)?(\s+(?:mods?|users?))?(\s+(?:["'].*["']|\/.*\/))?.*(\d+)?\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)?\s*$
      * */
-    reports?: CompareValue
+    reports?: string
     age?: DurationComparor
     /**
      * Test whether the activity is present in dispatched/delayed activities
@@ -330,6 +507,18 @@ export interface SubmissionState extends ActivityState {
      * Is the submission a reddit-hosted image or video?
      * */
     isRedditMediaDomain?: boolean
+
+    /**
+     * Compare the upvote ratio for this Submission, expressed as a whole number
+     *
+     * Can be either a comparison string or a number. If a number then CM assumes upvote ratio must be greater than or equal to this.
+     *
+     * Example:
+     *
+     * * `< 90` => less than 90% upvoted
+     * * 45 => greater than or equal to 45% upvoted
+     * */
+    upvoteRatio?: number | CompareValue
 }
 
 export const cmActivityProperties = ['submissionState', 'score', 'reports', 'removed', 'deleted', 'filtered', 'age', 'title'];

src/Common/Infrastructure/Includes.ts

@@ -0,0 +1,45 @@
+import {ConfigFormat} from "./Atomic";
+
+export interface IncludesData {
+    /**
+     * The special-form path to the config fragment to retrieve.
+     *
+     * If value starts with `wiki:` then the proceeding value will be used to get a wiki page from the current subreddit
+     *
+     *  * EX `wiki:botconfig/mybot` tries to get `https://reddit.com/r/currentSubreddit/wiki/botconfig/mybot`
+     *
+     * If the value starts with `wiki:` and ends with `|someValue` then `someValue` will be used as the base subreddit for the wiki page
+     *
+     * * EX `wiki:replytemplates/test|ContextModBot` tries to get `https://reddit.com/r/ContextModBot/wiki/replytemplates/test`
+     *
+     * If the value starts with `url:` then the value is fetched as an external url and expects raw text returned
+     *
+     * * EX `url:https://pastebin.com/raw/38qfL7mL` tries to get the text response of `https://pastebin.com/raw/38qfL7mL`
+     * */
+    path: string
+    /**
+     * An unused hint about the content type. Not implemented yet
+     * */
+    type?: ConfigFormat
+    /**
+     * Control caching for the config fragment.
+     *
+     * If not specified the value for `wikiTTL` will be used
+     *
+     * * If true then value is cached forever
+     * * If false then value is never cached
+     * * If a number then the number of seconds to cache value
+     * * If 'response' then CM will attempt to use Cache-Control or Expires headers from the response to determine how long to cache the value
+     * */
+    ttl?: number | boolean | 'response'
+}
+
+export type IncludesUrl = `url:${string}`;
+export type IncludesWiki = `wiki:${string}`;
+export type IncludesString = IncludesUrl | IncludesWiki;
+
+export type IncludesType = string | IncludesData;
+
+export const asIncludesData = (val: any): val is IncludesData => {
+    return val !== null && typeof val === 'object' && 'path' in val;
+}

src/Common/Infrastructure/Reddit.ts

@@ -30,3 +30,48 @@ export interface CachedFetchedActivitiesResult {
 export interface FetchedActivitiesResult extends CachedFetchedActivitiesResult {
     post: SnoowrapActivity[]
 }
+
+export type ReportType = 'mod' | 'user';
+
+export interface Report {
+    reason: string
+    type: ReportType
+    author?: string
+    snoozed: boolean
+    canSnooze: boolean
+}
+
+export type RawRedditUserReport = [
+    string, // reason
+    number, // number of reports with this reason
+    boolean, // is report snoozed
+    boolean // can the reports be snoozed
+];
+
+export type RawRedditModReport = [string, string];
+
+export const activityReports = (activity: SnoowrapActivity): Report[] => {
+    const reports: Report[] = [];
+    for(const r of (activity.user_reports as unknown as RawRedditUserReport[])) {
+        const report = {
+            reason: r[0],
+            type: 'user' as ReportType,
+            snoozed: r[2],
+            canSnooze: r[3]
+        };
+        for(let i = 0; i < r[1]; i++) {
+            reports.push(report);
+        }
+    }
+
+    for(const r of (activity.mod_reports as unknown as RawRedditModReport[])) {
+        reports.push({
+            reason: r[0],
+            type: 'mod' as ReportType,
+            author: r[1],
+            snoozed: false,
+            canSnooze: false
+        })
+    }
+    return reports;
+}

src/Common/Infrastructure/RuleShapes.ts

@@ -1,10 +1,12 @@
 import {StructuredRunnableBase} from "./Runnable";
-import {RuleSetObjectJson} from "../../Rule/RuleSet";
+import {RuleSetConfigObject} from "../../Rule/RuleSet";
 import {RuleObjectJsonTypes} from "../types";
+import {IncludesData} from "./Includes";
 
-export type RuleJson = RuleObjectJsonTypes | string;
-export type RuleObjectJson = Exclude<RuleJson, string>
-export type StructuredRuleObjectJson = Omit<RuleObjectJson, 'authorIs' | 'itemIs'> & StructuredRunnableBase
-export type StructuredRuleSetObjectJson = Omit<RuleSetObjectJson, 'rules'> & {
-    rules: StructuredRuleObjectJson[]
+export type RuleConfigData = RuleObjectJsonTypes | string | IncludesData;
+export type RuleConfigHydratedData = Exclude<RuleConfigData, IncludesData>
+export type RuleConfigObject = Exclude<RuleConfigHydratedData, string>
+export type StructuredRuleConfigObject = Omit<RuleConfigObject, 'authorIs' | 'itemIs'> & StructuredRunnableBase
+export type StructuredRuleSetConfigObject = Omit<RuleSetConfigObject, 'rules'> & {
+    rules: StructuredRuleConfigObject[]
 }

src/Common/Infrastructure/Runnable.ts

@@ -10,11 +10,16 @@ export interface RunnableBaseOptions extends Omit<RunnableBaseJson, 'itemIs' | '
     authorIs?: MinimalOrFullFilter<AuthorCriteria>
 }
 
-export interface StructuredRunnableBase {
+export interface StructuredRunnableBase extends RunnableBaseJson {
     itemIs?: MinimalOrFullFilter<TypedActivityState>
     authorIs?: MinimalOrFullFilter<AuthorCriteria>
 }
 
+export interface TypedStructuredRunnableBase<T> extends TypedRunnableBaseData<T> {
+    itemIs?: MinimalOrFullFilter<T>
+    authorIs?: MinimalOrFullFilter<AuthorCriteria>
+}
+
 export interface RunnableBaseJson {
     /**
      * A list of criteria to test the state of the `Activity` against before running the check.
@@ -31,3 +36,10 @@ export interface RunnableBaseJson {
      * */
     authorIs?: MinimalOrFullFilterJson<AuthorCriteria>
 }
+
+export interface TypedRunnableBaseData<T extends TypedActivityState> extends RunnableBaseJson {
+    /**
+     * If present then these Author criteria are checked before running the Check. If criteria fails then the Check will fail.
+     * */
+    authorIs?: MinimalOrFullFilterJson<AuthorCriteria>
+}

src/Common/LangaugeProcessing.ts

@@ -0,0 +1,493 @@
+import {containerBootstrap} from '@nlpjs/core';
+import {Language, LanguageGuess, LanguageType} from '@nlpjs/language';
+import {Nlp} from '@nlpjs/nlp';
+import {SentimentIntensityAnalyzer} from 'vader-sentiment';
+import wink from 'wink-sentiment';
+import {SnoowrapActivity} from "./Infrastructure/Reddit";
+import {
+    asGenericComparison, comparisonTextOp,
+    GenericComparison,
+    parseGenericValueComparison,
+    RangedComparison
+} from "./Infrastructure/Comparisons";
+import {asSubmission, between, formatNumber} from "../util";
+import {CMError, MaybeSeriousErrorWithCause} from "../Utils/Errors";
+import InvalidRegexError from "../Utils/InvalidRegexError";
+import {StringOperator} from "./Infrastructure/Atomic";
+import {LangEs} from "@nlpjs/lang-es";
+import {LangDe} from "@nlpjs/lang-de";
+import {LangEn} from "@nlpjs/lang-en";
+import {LangFr} from "@nlpjs/lang-fr";
+
+export type SentimentAnalysisType = 'vader' | 'afinn' | 'senticon' | 'pattern' | 'wink';
+
+export const sentimentQuantifier = {
+    'extremely negative': -0.6,
+    'very negative': -0.3,
+    'negative': -0.1,
+    'neutral': 0,
+    'positive': 0.1,
+    'very positive': 0.3,
+    'extremely positive': 0.6,
+}
+
+export const sentimentQuantifierRanges = [
+    {
+        range: [Number.MIN_SAFE_INTEGER, -0.6],
+        quant: 'extremely negative'
+    },
+    {
+        range: [-0.6, -0.3],
+        quant: 'very negative'
+    },
+    {
+        range: [-0.3, -0.1],
+        quant: 'negative'
+    },
+    {
+        range: [-0.1, 0.1],
+        quant: 'neutral'
+    },
+    {
+        range: [0.1, 0.3],
+        quant: 'positive'
+    },
+    {
+        range: [0.3, 0.6],
+        quant: 'very positive'
+    },
+    {
+        range: [0.6, Number.MAX_SAFE_INTEGER],
+        quant: 'extremely positive'
+    }
+]
+
+const scoreToSentimentText = (val: number) => {
+    for (const segment of sentimentQuantifierRanges) {
+        if (between(val, segment.range[0], segment.range[1], false, true)) {
+            return segment.quant;
+        }
+    }
+    throw new Error('should not hit this!');
+}
+
+export interface SentimentResult {
+    comparative: number
+    type: SentimentAnalysisType
+    sentiment: string
+    weight: number
+    tokens: number
+    matchedTokens?: number,
+    usableResult: true | string
+}
+
+export interface StringSentiment {
+    results: SentimentResult[]
+    score: number
+    scoreWeighted: number
+    sentiment: string
+    sentimentWeighted: string
+    guessedLanguage: LanguageGuessResult
+    usedLanguage: LanguageType
+    usableScore: boolean
+    reason?: string
+}
+
+export interface ActivitySentiment extends StringSentiment {
+    activity: SnoowrapActivity
+}
+
+export interface StringSentimentTestResult extends StringSentiment {
+    passes: boolean
+    test: GenericComparison | RangedComparison
+}
+
+export interface ActivitySentimentTestResult extends StringSentimentTestResult {
+    activity: SnoowrapActivity
+}
+
+export interface ActivitySentimentOptions {
+    testOn?: ('title' | 'body')[]
+    /**
+     * Make the analyzer assume a language if it cannot determine one itself.
+     *
+     * This is very useful for the analyzer when it is parsing short pieces of content. For example, if you know your subreddit is majority english speakers this will make the analyzer return "neutral" sentiment instead of "not detected language".
+     *
+     * Defaults to 'en'
+     *
+     * @example ["en"]
+     * @default en
+     * */
+    defaultLanguage?: string | null | false
+
+    /**
+     * Helps the analyzer coerce a low confidence language guess into a known-used languages in two ways:
+     *
+     * If the analyzer's
+     *   * *best* guess is NOT one of these
+     *     * but it did guess one of these
+     *     * and its guess is above requiredLanguageConfidence score then use the hinted language instead of best guess
+     *   * OR text content is very short (4 words or less)
+     *     * and the best guess was below the requiredLanguageConfidence score
+     *     * and none of guesses was a hinted language then use the defaultLanguage
+     *
+     * Defaults to popular romance languages: ['en', 'es', 'de', 'fr']
+     *
+     * @example [["en", "es", "de", "fr"]]
+     * @default ["en", "es", "de", "fr"]
+     * */
+    languageHints?: string[]
+
+    /**
+     * Required confidence to use a guessed language as the best guess. Score from 0 to 1.
+     *
+     * Defaults to 0.9
+     *
+     * @example [0.9]
+     * @default 0.9
+     * */
+    requiredLanguageConfidence?: number
+}
+
+export type SentimentCriteriaTest = GenericComparison | RangedComparison;
+
+export const availableSentimentLanguages = ['en', 'es', 'de', 'fr'];
+
+export const textComparison = /(?<not>not)?\s*(?<modifier>very|extremely)?\s*(?<sentiment>positive|neutral|negative)/i;
+
+export const parseTextToNumberComparison = (val: string): RangedComparison | GenericComparison => {
+
+    let genericError: Error | undefined;
+    try {
+        return parseGenericValueComparison(val);
+    } catch (e) {
+        genericError = e as Error;
+        // now try text match
+    }
+
+    const matches = val.match(textComparison);
+    if (matches === null) {
+        const textError = new InvalidRegexError(textComparison, val);
+        throw new CMError(`Sentiment value did not match a valid numeric comparison or valid text: \n ${genericError.message} \n ${textError.message}`);
+    }
+    const groups = matches.groups as any;
+
+    const negate = groups.not !== undefined && groups.not !== '';
+
+    if (groups.sentiment === 'neutral') {
+        if (negate) {
+            return {
+                displayText: 'not neutral (not -0.1 to 0.1)',
+                range: [-0.1, 0.1],
+                not: true,
+            }
+        }
+        return {
+            displayText: 'is neutral (-0.1 to 0.1)',
+            range: [-0.1, 0.1],
+            not: false
+        }
+    }
+
+    const compoundSentimentText = `${groups.modifier !== undefined && groups.modifier !== '' ? `${groups.modifier} ` : ''}${groups.sentiment}`.toLocaleLowerCase();
+    // @ts-ignore
+    const numericVal = sentimentQuantifier[compoundSentimentText] as number;
+    if (numericVal === undefined) {
+        throw new CMError(`Sentiment given did not match any known phrases: '${compoundSentimentText}'`);
+    }
+
+    let operator: StringOperator;
+    if (negate) {
+        operator = numericVal > 0 ? '<' : '>';
+    } else {
+        operator = numericVal > 0 ? '>=' : '<=';
+    }
+
+    return {
+        operator,
+        value: numericVal,
+        isPercent: false,
+        displayText: `is${negate ? ' not ' : ' '}${compoundSentimentText} (${operator} ${numericVal})`
+    }
+}
+
+let nlp: Nlp;
+let container: any;
+
+const bootstrapNlp = async () => {
+
+    container = await containerBootstrap();
+    container.use(Language);
+    container.use(Nlp);
+    container.use(LangEs);
+    container.use(LangDe);
+    container.use(LangEn);
+    container.use(LangFr);
+    nlp = container.get('nlp');
+    nlp.settings.autoSave = false;
+    nlp.addLanguage('en');
+    nlp.addLanguage('es');
+    nlp.addLanguage('de');
+    nlp.addLanguage('fr');
+    nlp.nluManager.guesser.processExtraSentences();
+    await nlp.train();
+}
+
+export const getNlp = async () => {
+    if (nlp === undefined) {
+        await bootstrapNlp();
+    }
+
+    return nlp;
+}
+
+export const getActivityContent = (item: SnoowrapActivity, options?: ActivitySentimentOptions): string => {
+    const {
+        testOn = ['body', 'title'],
+    } = options || {};
+
+    // determine what content we are testing
+    let contents: string[] = [];
+    if (asSubmission(item)) {
+        for (const l of testOn) {
+            switch (l) {
+                case 'title':
+                    contents.push(item.title);
+                    break;
+                case 'body':
+                    if (item.is_self) {
+                        contents.push(item.selftext);
+                    }
+                    break;
+            }
+        }
+    } else {
+        contents.push(item.body)
+    }
+
+    return contents.join(' ');
+}
+
+export const getLanguageTypeFromValue = async (val: string): Promise<LanguageType> => {
+
+    if (nlp === undefined) {
+        await bootstrapNlp();
+    }
+
+    const langObj = container.get('Language') as Language;
+
+    const cleanVal = val.trim().toLocaleLowerCase();
+
+    const foundLang = Object.values(langObj.languagesAlpha2).find(x => x.alpha2 === cleanVal || x.alpha3 === cleanVal || x.name.toLocaleLowerCase() === cleanVal);
+    if (foundLang === undefined) {
+        throw new MaybeSeriousErrorWithCause(`Could not find Language with identifier '${val}'`, {isSerious: false});
+    }
+    const {alpha2, alpha3, name: language} = foundLang;
+    return {
+        alpha2,
+        alpha3,
+        language
+    };
+}
+
+export interface LanguageGuessResult {
+    bestGuess: LanguageGuess
+    guesses: LanguageGuess[]
+    requiredConfidence: number
+    sparse: boolean
+    language: LanguageType
+    usedDefault: boolean
+}
+
+export const getContentLanguage = async (content: string, options?: ActivitySentimentOptions): Promise<LanguageGuessResult> => {
+
+    const {
+        defaultLanguage = 'en',
+        requiredLanguageConfidence = 0.9,
+        languageHints = availableSentimentLanguages
+    } = options || {};
+
+    if (nlp === undefined) {
+        await bootstrapNlp();
+    }
+
+    const spaceNormalizedTokens = content.trim().split(' ').filter(x => x !== ''.trim());
+
+    const lang = container.get('Language') as Language;
+    // would like to improve this https://github.com/axa-group/nlp.js/issues/761
+    const guesses = lang.guess(content, null, 4);
+    let bestLang = guesses[0];
+    const shortContent = spaceNormalizedTokens.length <= 4;
+
+    const altBest = languageHints.includes(bestLang.alpha2) ? undefined : guesses.find(x => x.score >= requiredLanguageConfidence && languageHints.includes(x.alpha2));
+
+    // coerce best guess into a supported language that has a good enough confidence
+    if(!shortContent && altBest !== undefined) {
+        bestLang = altBest;
+    }
+
+    let usedLang: LanguageType = bestLang;
+    let usedDefault = false;
+
+    if (typeof defaultLanguage === 'string' && (bestLang.score < requiredLanguageConfidence || (shortContent && !languageHints.includes(bestLang.alpha2)))) {
+        usedLang = await getLanguageTypeFromValue(defaultLanguage);
+        usedDefault = true;
+    }
+
+    return {
+        guesses,
+        bestGuess: bestLang,
+        requiredConfidence: requiredLanguageConfidence,
+        sparse: shortContent,
+        language: usedLang,
+        usedDefault
+    }
+}
+
+export const getActivitySentiment = async (item: SnoowrapActivity, options?: ActivitySentimentOptions): Promise<ActivitySentiment> => {
+
+    const result = await getStringSentiment(getActivityContent(item, options), options);
+
+    return {
+        ...result,
+        activity: item
+    }
+}
+
+export const getStringSentiment = async (contentStr: string, options?: ActivitySentimentOptions): Promise<StringSentiment> => {
+
+    const langResult = await getContentLanguage(contentStr, options);
+
+    let usedLanguage: LanguageType = langResult.language;
+
+    const spaceNormalizedTokens = contentStr.trim().split(' ').filter(x => x !== ''.trim());
+
+    const results: SentimentResult[] = [];
+
+    const nlpResult = await nlp.process(langResult.language.alpha2, contentStr);
+
+    results.push({
+        comparative: nlpResult.sentiment.average,
+        type: nlpResult.sentiment.type as SentimentAnalysisType,
+        sentiment: scoreToSentimentText(nlpResult.sentiment.average),
+        weight: 1,
+        matchedTokens: nlpResult.sentiment.numHits,
+        tokens: nlpResult.sentiment.numWords,
+        usableResult: availableSentimentLanguages.includes(langResult.language.alpha2) ? true : (nlpResult.sentiment.numHits / nlpResult.sentiment.numWords) >= 0.5 ? true : `${langResult.sparse ? 'Content was too short to guess language' : 'Unsupported language'} and less than 50% of tokens matched`,
+    });
+
+    // only run vader/wink if either
+    //
+    // * content was short which means we aren't confident on language guess
+    // * OR language is english (guessed or explicitly set as language fallback by user due to low confidence)
+    //
+    if (langResult.sparse || langResult.language.alpha2 === 'en') {
+
+        // neg post neu are ratios of *recognized* tokens in the content
+        // when neu is close to 1 its either extremely neutral or no tokens were recognized
+        const vaderScore = SentimentIntensityAnalyzer.polarity_scores(contentStr);
+        const vaderRes: SentimentResult = {
+            comparative: vaderScore.compound,
+            type: 'vader',
+            sentiment: scoreToSentimentText(vaderScore.compound),
+            // may want to weight higher in the future...
+            weight: 1,
+            tokens: spaceNormalizedTokens.length,
+            usableResult: langResult.language.alpha2 === 'en' ? true : (vaderScore.neu < 0.5 ? true : `Unable to guess language and unable to determine if more than 50% of tokens are negative or not matched`)
+        };
+        results.push(vaderRes);
+
+        const winkScore = wink(contentStr);
+        const matchedTokens = winkScore.tokenizedPhrase.filter(x => x.score !== undefined);
+        const matchedMeaningfulTokens = winkScore.tokenizedPhrase.filter(x => x.tag === 'word' || x.tag === 'emoji');
+        // normalizedScore is range of -5 to +5 -- convert to -1 to +1
+        const winkAdjusted = (winkScore.normalizedScore * 2) / 10;
+        const winkRes: SentimentResult = {
+            comparative: winkAdjusted,
+            type: 'wink',
+            sentiment: scoreToSentimentText(winkAdjusted),
+            weight: 1,
+            matchedTokens: matchedTokens.length,
+            tokens: winkScore.tokenizedPhrase.length,
+            usableResult: langResult.language.alpha2 === 'en' ? true : ((matchedTokens.length / matchedMeaningfulTokens.length) > 0.5 ? true : 'Unable to guess language and less than 50% of tokens matched')
+        };
+        results.push(winkRes);
+
+        if ((vaderRes.usableResult == true || winkRes.usableResult === true) && usedLanguage.alpha2 !== 'en') {
+            // since we are confident enough to use one of these then we are assuming language is mostly english
+            usedLanguage = await getLanguageTypeFromValue('en');
+        }
+    }
+
+    const score = results.reduce((acc, curr) => acc + curr.comparative, 0) / results.length;
+    const sentiment = scoreToSentimentText(score);
+
+    const weightSum = results.reduce((acc, curr) => acc + curr.weight, 0);
+    const weightedScores = results.reduce((acc, curr) => acc + (curr.weight * curr.comparative), 0);
+    const weightedScore = weightedScores / weightSum;
+    const weightedSentiment = scoreToSentimentText(weightedScore);
+
+    const actSentResult: StringSentiment = {
+        results,
+        score,
+        sentiment,
+        scoreWeighted: weightedScore,
+        sentimentWeighted: weightedSentiment,
+        guessedLanguage: langResult,
+        usedLanguage,
+        usableScore: results.filter(x => x.usableResult === true).length > 0,
+    }
+
+    if (!actSentResult.usableScore) {
+        if (actSentResult.guessedLanguage.sparse) {
+            actSentResult.reason = 'Content may be supported language but was too short to guess accurately and no algorithm matched enough tokens to be considered confident.';
+        } else {
+            actSentResult.reason = 'Unsupported language'
+        }
+    }
+
+    return actSentResult;
+}
+
+export const testActivitySentiment = async (item: SnoowrapActivity, criteria: SentimentCriteriaTest, options?: ActivitySentimentOptions): Promise<ActivitySentimentTestResult> => {
+    const sentimentResult = await getActivitySentiment(item, options);
+
+    const testResult = testSentiment(sentimentResult, criteria);
+
+    return {
+        ...testResult,
+        activity: item
+    }
+}
+
+export const testSentiment = (sentimentResult: StringSentiment, criteria: SentimentCriteriaTest): StringSentimentTestResult => {
+
+    if (!sentimentResult.usableScore) {
+        return {
+            passes: false,
+            test: criteria,
+            ...sentimentResult,
+        }
+    }
+
+    if (asGenericComparison(criteria)) {
+        return {
+            passes: comparisonTextOp(sentimentResult.scoreWeighted, criteria.operator, criteria.value),
+            test: criteria,
+            ...sentimentResult,
+        }
+    } else {
+        if (criteria.not) {
+            return {
+                passes: sentimentResult.scoreWeighted < criteria.range[0] || sentimentResult.scoreWeighted > criteria.range[1],
+                test: criteria,
+                ...sentimentResult,
+            }
+        }
+        return {
+            passes: sentimentResult.scoreWeighted >= criteria.range[0] || sentimentResult.scoreWeighted <= criteria.range[1],
+            test: criteria,
+            ...sentimentResult,
+        }
+    }
+}

src/Common/Migrations/Database/MigrationUtil.ts

@@ -0,0 +1,80 @@
+import {TableIndex} from "typeorm";
+
+/**
+ * Boilerplate for creating generic index
+ * */
+export const index = (prefix: string, columns: string[], unique = true) => new TableIndex({
+    name: `IDX_${unique ? 'UN_' : ''}${prefix}_${columns.join('-')}`,
+    columnNames: columns,
+    isUnique: unique,
+});
+/**
+ * Create index on id column
+ * */
+export const idIndex = (prefix: string, unique: boolean) => index(prefix, ['id'], unique);
+
+/**
+ * Boilerplate primary key column for random ID
+ * */
+export const randomIdColumn = () => ({
+    name: 'id',
+    type: 'varchar',
+    length: '20',
+    isPrimary: true,
+    isUnique: true,
+});
+
+/**
+ * Create a time data column based on database type
+ * */
+export const timeAtColumn = (columnName: string, dbType: string, nullable?: boolean) => {
+    const dbSpecifics = dbType === 'postgres' ? {
+        type: 'timestamptz'
+    } : {
+        type: 'datetime',
+        // required to get millisecond precision on mysql/mariadb
+        // https://mariadb.com/kb/en/datetime/
+        // https://dev.mysql.com/doc/refman/8.0/en/fractional-seconds.html
+        length: '3',
+    }
+    return {
+        name: columnName,
+        isNullable: nullable ?? false,
+        ...dbSpecifics
+    }
+}
+export const createdAtColumn = (type: string) => timeAtColumn('createdAt', type);
+const updatedAtColumn = (type: string) => timeAtColumn('updatedAt', type);
+const createdUpdatedAtColumns = (type: string) => [
+    timeAtColumn('createdAt', type),
+    timeAtColumn('updatedAt', type)
+];
+export const createdAtIndex = (prefix: string) => index(prefix, ['createdAt'], false);
+
+const updatedAtIndex = (prefix: string) => index(prefix, ['updatedAt'], false);
+const createdUpdatedAtIndices = (prefix: string) => {
+    return [
+        createdAtIndex(prefix),
+        updatedAtIndex(prefix)
+    ]
+}
+/**
+ * Boilerplate for filter (itemIs, authorIs) FK column -- uses FK is filter ID
+ * */
+const filterColumn = (name: string) => ({
+    name,
+    type: 'varchar',
+    length: '20',
+    isNullable: true
+});
+const authorIsColumn = () => filterColumn('authorIs');
+const itemIsColumn = () => filterColumn('itemIs');
+export const filterColumns = () => ([authorIsColumn(), itemIsColumn()]);
+const authorIsIndex = (prefix: string) => index(prefix, ['authorIs']);
+const itemIsIndex = (prefix: string) => index(prefix, ['itemIs']);
+export const filterIndices = (prefix: string) => {
+    return [
+        authorIsIndex(prefix),
+        itemIsIndex(prefix)
+    ]
+}

src/Common/Migrations/Database/Server/1642180264563-initApi.ts

@@ -1,86 +1,12 @@
-import {MigrationInterface, QueryRunner, Table, TableIndex, TableColumn, TableForeignKey} from "typeorm";
-
-const randomIdColumn = () => ({
-    name: 'id',
-    type: 'varchar',
-    length: '20',
-    isPrimary: true,
-    isUnique: true,
-});
-
-const timeAtColumn = (columnName: string, dbType: string, nullable?: boolean) => {
-    const dbSpecifics = dbType === 'postgres' ? {
-        type: 'timestamptz'
-    } : {
-        type: 'datetime',
-        // required to get millisecond precision on mysql/mariadb
-        // https://mariadb.com/kb/en/datetime/
-        // https://dev.mysql.com/doc/refman/8.0/en/fractional-seconds.html
-        length: '3',
-    }
-    return {
-        name: columnName,
-        isNullable: nullable ?? false,
-        ...dbSpecifics
-    }
-}
-
-const createdAtColumn = (type: string) => timeAtColumn('createdAt', type);
-const updatedAtColumn = (type: string) => timeAtColumn('updatedAt', type);
-
-const createdUpdatedAtColumns = (type: string) => [
-    timeAtColumn('createdAt', type),
-    timeAtColumn('updatedAt', type)
-];
-
-
-const createdAtIndex = (prefix: string) => new TableIndex({
-    name: `IDX_${prefix}_createdAt`,
-    columnNames: ['createdAt']
-});
-
-const updatedAtIndex = (prefix: string) => new TableIndex({
-    name: `IDX_${prefix}_updatedAt`,
-    columnNames: ['updatedAt']
-})
-
-const createdUpdatedAtIndices = (prefix: string) => {
-    return [
-        createdAtIndex(prefix),
-        updatedAtIndex(prefix)
-    ]
-}
-
-const filterColumn = (name: string) => ({
-    name,
-    type: 'varchar',
-    length: '20',
-    isNullable: true
-});
-
-const authorIsColumn = () => filterColumn('authorIs');
-const itemIsColumn = () => filterColumn('itemIs');
-
-const filterColumns = () => ([authorIsColumn(), itemIsColumn()]);
-
-const authorIsIndex = (prefix: string) => new TableIndex({
-    name: `IDX_${prefix}_authorIs`,
-    columnNames: ['authorIs'],
-    isUnique: true,
-});
-
-const itemIsIndex = (prefix: string) => new TableIndex({
-    name: `IDX_${prefix}_itemIs`,
-    columnNames: ['itemIs'],
-    isUnique: true
-});
-
-const filterIndices = (prefix: string) => {
-    return [
-        authorIsIndex(prefix),
-        itemIsIndex(prefix)
-    ]
-}
+import {MigrationInterface, QueryRunner, Table, TableIndex} from "typeorm";
+import {
+    createdAtColumn,
+    createdAtIndex,
+    filterColumns,
+    filterIndices,
+    randomIdColumn,
+    timeAtColumn
+} from "../MigrationUtil";
 
 export class initApi1642180264563 implements MigrationInterface {
     name = 'initApi1642180264563'

src/Common/Migrations/Database/Server/1653586738904-indexes.ts

@@ -0,0 +1,90 @@
+import {MigrationInterface, QueryRunner, Table} from "typeorm"
+import {idIndex, index} from "../MigrationUtil";
+
+export class indexes1653586738904 implements MigrationInterface {
+
+    public async up(queryRunner: QueryRunner): Promise<void> {
+
+        queryRunner.connection.logger.logSchemaBuild('Starting Index Add/Update Migration');
+        queryRunner.connection.logger.logSchemaBuild('IF YOU HAVE A LARGE DATABASE THIS MAY TAKE SEVERAL MINUTES! DO NOT STOP CONTEXTMOD WHILE MIGRATION IS IN PROGRESS!');
+
+        // unique ids due to random id
+        const uniqueIdTableNames = [
+            'Manager',
+            'CMEvent',
+            'FilterResult',
+            'FilterCriteriaResult',
+            'RunnableResult',
+            'RulePremise',
+            'RuleResult',
+            'RuleSetResult',
+            'ActionPremise',
+            'ActionResult',
+            'CheckResult',
+            'RunResult'
+        ];
+
+        for (const tableName of uniqueIdTableNames) {
+            const cmTable = await queryRunner.getTable(tableName);
+            await queryRunner.createIndex(cmTable as Table, idIndex(tableName, true));
+        }
+
+        // additional indexes
+
+        const actSource = await queryRunner.getTable('ActivitySource');
+        await queryRunner.createIndex(actSource as Table, idIndex('ActivitySource', false));
+
+        const event = await queryRunner.getTable('CMEvent');
+        await queryRunner.createIndices(event as Table, [index('CMEvent', ['activity_id'], false)]);
+
+        // FilterCriteriaResult criteriaId filterResultId
+
+        const fcrTable = await queryRunner.getTable('FilterCriteriaResult');
+        await queryRunner.createIndices(fcrTable as Table, [
+            index('FilterCriteriaResult', ['criteriaId'], false),
+            index('FilterCriteriaResult', ['filterResultId'], false)
+        ]);
+
+
+        // FilterCriteria id
+
+        const fcTable = await queryRunner.getTable('FilterCriteria');
+        await queryRunner.createIndices(fcTable as Table, [
+            idIndex('FilterCriteriaResult', false),
+        ]);
+
+        // RunnableResult resultId runnableId
+
+        const rrTable = await queryRunner.getTable('RunnableResult');
+        await queryRunner.createIndices(rrTable as Table, [
+            index('RunnableResult', ['resultId'], false),
+            index('RunnableResult', ['runnableId'], false)
+        ]);
+
+        // ActionResult checkResultId premiseId
+
+        const arTable = await queryRunner.getTable('ActionResult');
+        await queryRunner.createIndices(arTable as Table, [
+            index('ActionResult', ['checkResultId'], false),
+            index('ActionResult', ['premiseId'], false)
+        ]);
+
+        // CheckResult runId
+
+        const crTable = await queryRunner.getTable('CheckResult');
+        await queryRunner.createIndices(crTable as Table, [
+            index('CheckResult', ['runId'], false),
+        ]);
+
+        // RunResult eventId
+
+        const runResTable = await queryRunner.getTable('RunResult');
+        await queryRunner.createIndices(runResTable as Table, [
+            index('RunResult', ['eventId'], false),
+        ]);
+    }
+
+    public async down(queryRunner: QueryRunner): Promise<void> {
+    }
+
+}

src/Common/Migrations/Database/Server/1657632517934-reportTracking.ts

@@ -0,0 +1,62 @@
+import {MigrationInterface, QueryRunner, Table, TableIndex} from "typeorm"
+import {createdAtColumn, createdAtIndex, idIndex, index, randomIdColumn} from "../MigrationUtil";
+
+export class reportTracking1657632517934 implements MigrationInterface {
+
+    public async up(queryRunner: QueryRunner): Promise<void> {
+
+        const dbType = queryRunner.connection.driver.options.type;
+
+        await queryRunner.createTable(
+            new Table({
+                name: 'ActivityReport',
+                columns: [
+                    randomIdColumn(),
+                    {
+                        name: 'activityId',
+                        type: 'varchar',
+                        length: '20',
+                        isNullable: false
+                    },
+                    {
+                        name: 'reason',
+                        type: 'varchar',
+                        length: '500',
+                        isNullable: false
+                    },
+                    {
+                        name: 'type',
+                        type: 'varchar',
+                        length: '200',
+                        isNullable: false
+                    },
+                    {
+                        name: 'author',
+                        type: 'varchar',
+                        length: '100',
+                        isNullable: true
+                    },
+                    {
+                        name: 'granularity',
+                        type: 'int',
+                        isNullable: false
+                    },
+                    createdAtColumn(dbType),
+                ],
+                indices: [
+                    idIndex('ActivityReport', true),
+                    index('ActivityReport', ['activityId'], false),
+                    index('ActivityReportReason', ['reason'], false),
+                    createdAtIndex('report'),
+                ]
+            }),
+            true,
+            true,
+            true
+        );
+    }
+
+    public async down(queryRunner: QueryRunner): Promise<void> {
+    }
+
+}

src/Common/Typings/support.d.ts

@@ -3,9 +3,11 @@ declare module 'snoowrap/dist/errors' {
     export interface InvalidUserError extends Error {
 
     }
+
     export interface NoCredentialsError extends Error {
 
     }
+
     export interface InvalidMethodCallError extends Error {
 
     }
@@ -26,9 +28,138 @@ declare module 'snoowrap/dist/errors' {
 }
 
 declare module 'winston-null' {
-  import TransportStream from "winston-transport";
+    import TransportStream from "winston-transport";
 
     export class NullTransport extends TransportStream {
 
     }
 }
+
+declare module '@nlpjs/*' {
+
+    declare interface SentimentResult {
+        score: number,
+        average: number,
+        numWords: number,
+        numHits: number,
+        type: string,
+        language: string
+    }
+
+    declare interface NLPSentimentResult extends Omit<SentimentResult, 'language'> {
+        vote: string
+        locale: string
+    }
+
+
+    declare module '@nlpjs/language' {
+
+        export interface LanguageType {
+            alpha3: string,
+            alpha2: string,
+            language: string,
+        }
+
+        export interface LanguageObj {
+            alpha3: string,
+            alpha2: string,
+            name: string,
+        }
+
+        export interface LanguageGuess extends LanguageType {
+            score: number
+        }
+
+        export class Language {
+            guess(val: string, allowedList?: string[] | null, limit?: number): LanguageGuess[];
+
+            guessBest(val: string, allowedList?: string[] | null): LanguageGuess;
+
+            /**
+             * Key is alpha2 lang IE en es de fr
+             * */
+            languagesAlpha2: Record<string, LanguageObj>;
+            /**
+             * Key is alpha3 lang IE eng spa deu fra
+             * */
+            languagesAlpha3: Record<string, LanguageObj>;
+        }
+    }
+
+    declare module '@nlpjs/sentiment' {
+
+        declare interface SentimentPipelineResult {
+            utterance: string
+            locale: string
+            settings: { tag: string }
+            tokens: string[]
+            sentiment: SentimentResult
+        }
+
+        declare interface SentimentPipelineInput {
+            utterance: string
+            locale: string
+
+            [key: string]: any
+        }
+
+        export class SentimentAnalyzer {
+            constructor(settings?: { language?: string }, container?: any)
+
+            container: any
+
+            process(srcInput: SentimentPipelineInput, settings?: object): Promise<SentimentPipelineResult>
+        }
+    }
+
+    declare module '@nlpjs/nlp' {
+
+        declare interface NlpResult {
+            locale: string
+            language: string
+            languageGuessed: boolean
+            sentiment: NLPSentimentResult
+        }
+
+        export class Nlp {
+            settings: any;
+            nluManager: any;
+
+            constructor(settings?: { language?: string }, container?: any)
+
+            // locale language languageGuessed sentiment
+            process(locale: string, utterance?: string, srcContext?: object, settings?: object): Promise<NlpResult>
+            addLanguage(locale: string)
+            train(): Promise<any>;
+
+        }
+    }
+
+    declare module '@nlpjs/lang-es' {
+        export const LangEs: any
+    }
+    declare module '@nlpjs/lang-en' {
+        export const LangEn: any
+    }
+    declare module '@nlpjs/lang-de' {
+        export const LangDe: any
+    }
+    declare module '@nlpjs/lang-fr' {
+        export const LangFr: any
+    }
+    declare module '@nlpjs/nlu' {
+        export const Nlu: any
+    }
+
+    declare module '@nlpjs/core' {
+        export const Container: any
+        export const containerBootstrap: any
+    }
+}
+
+
+declare module 'wink-sentiment' {
+    function sentiment(phrase: string): { score: number, normalizedScore: number, tokenizedPhrase: any[] };
+
+    export default sentiment;
+}

src/Common/Typings/vader-sentiment.d.ts

@@ -0,0 +1,50 @@
+declare module 'vader-sentiment' {
+    export const REGEX_REMOVE_PUNCTUATION: RegExp;
+    export const B_INCR: number;
+    export const B_DECR: number;
+    export const C_INCR: number;
+    export const N_SCALER: number;
+    export const PUNC_LIST: string[];
+    export const NEGATE: string[];
+    export const BOOSTER_DICT: Record<string, number>;
+    export const SPECIAL_CASE_IDIOMS: Record<string, number>;
+
+    export interface Scores {
+        neg: number
+        neu: number
+        pos: number
+        compound: number
+    }
+
+    export function negated(input_words: string[], include_nt: boolean = true): boolean;
+    export function normalize(score: number, alpha: number): number;
+    export function allcap_differential(words: string[]): boolean;
+    export function scalar_inc_dec(word: string, valence: number, is_cap_diff: boolean): number
+    export function is_upper_function(word: string): boolean
+
+    export class SentiText {
+        public text: string;
+        public words_and_emoticons: string[];
+        public is_cap_diff: boolean;
+
+        constructor(text: string);
+
+        get_words_plus_punc(): Record<string, string>;
+        get_words_and_emoticons(): string[];
+    }
+
+    export class SentimentIntensityAnalyzer {
+
+        static polarity_scores(text: string): Scores;
+        static sentiment_valence(valence: number, sentiText: SentiText, item: string, index: number, sentiments: number[]);
+        static least_check(valence: number, words_and_emoticons: string[], index: number): number;
+        static but_check(words_and_emoticons: string[], sentiments: number[]): number[]
+        static idioms_check(valence: number, words_and_emoticons: string[], index: number): number;
+        static never_check(valence: number, words_and_emoticons: string[], start_i: number, index: number): number
+        static punctuation_emphasis(sum_s: any, text: string);
+        static amplify_ep(text: string): number;
+        static amplify_qm(text: string): number;
+        static sift_sentiment_scores(sentiments: number[]): number[];
+        static score_valence(sentiments: number[], text: string): Scores;
+    }
+}

src/Common/defaults.ts

@@ -3,7 +3,17 @@ import path from "path";
 import {FilterCriteriaDefaults} from "./Infrastructure/Filters/FilterShapes";
 
 export const cacheOptDefaults = {ttl: 60, max: 500, checkPeriod: 600};
-export const cacheTTLDefaults = {authorTTL: 60, userNotesTTL: 300, wikiTTL: 300, submissionTTL: 60, commentTTL: 60, filterCriteriaTTL: 60, subredditTTL: 600, selfTTL: 60};
+export const cacheTTLDefaults = {
+    authorTTL: 60,
+    userNotesTTL: 300,
+    modNotesTTL: 60,
+    wikiTTL: 300,
+    submissionTTL: 60,
+    commentTTL: 60,
+    filterCriteriaTTL: 60,
+    subredditTTL: 600,
+    selfTTL: 60
+};
 
 export const createHistoricalDisplayDefaults = (): HistoricalStatsDisplay => ({
     checksRunTotal: 0,
@@ -32,4 +42,4 @@ export const filterCriteriaDefault: FilterCriteriaDefaults = {
 export const defaultDataDir = path.resolve(__dirname, '../..');
 export const defaultConfigFilenames = ['config.json', 'config.yaml'];
 
-export const VERSION = '0.10.12';
+export const VERSION = '0.11.4';

src/Common/interfaces.ts

@@ -6,7 +6,6 @@ import Comment from "snoowrap/dist/objects/Comment";
 import RedditUser from "snoowrap/dist/objects/RedditUser";
 import {DataSource} from "typeorm";
 import {JsonOperatorConfigDocument, YamlOperatorConfigDocument} from "./Config/Operator";
-import {CommentCheckJson, SubmissionCheckJson} from "../Check";
 import {SafeDictionary} from "ts-essentials";
 import {RuleResultEntity} from "./Entities/RuleResultEntity";
 import {Dayjs} from "dayjs";
@@ -43,8 +42,16 @@ import {
     ItemOptions
 } from "./Infrastructure/Filters/FilterShapes";
 import {LoggingOptions, LogLevel, StrongLoggingOptions} from "./Infrastructure/Logging";
-import {DatabaseConfig, DatabaseDriver, DatabaseDriverConfig, DatabaseDriverType} from "./Infrastructure/Database";
+import {
+    DatabaseConfig,
+    DatabaseDriver,
+    DatabaseDriverConfig,
+    DatabaseDriverType
+} from "./Infrastructure/Database";
 import {ActivityType} from "./Infrastructure/Reddit";
+import {InfluxDB, WriteApi} from "@influxdata/influxdb-client";
+import {InfluxConfig} from "./Influx/interfaces";
+import {InfluxClient} from "./Influx/InfluxClient";
 
 
 export interface ReferenceSubmission {
@@ -461,6 +468,17 @@ export interface TTLConfig {
      * @default 50
      * */
     selfTTL?: number | boolean
+
+    /**
+     * Amount of time, in seconds, Mod Notes should be cached
+     *
+     * * If `0` or `true` will cache indefinitely (not recommended)
+     * * If `false` will not cache
+     *
+     * @examples [60]
+     * @default 60
+     * */
+    modNotesTTL?: number | boolean;
 }
 
 export interface CacheConfig extends TTLConfig {
@@ -714,19 +732,14 @@ export interface SearchAndReplaceRegExp {
 }
 
 export interface NamedGroup {
-    [name: string]: string
-}
-
-export interface GlobalRegExResult {
-    match: string,
-    groups: string[],
-    named: NamedGroup | undefined
+    [name: string]: any
 }
 
 export interface RegExResult {
-    matched: boolean,
-    matches: string[],
-    global: GlobalRegExResult[]
+    match: string,
+    groups: string[],
+    index: number
+    named: NamedGroup
 }
 
 export type StrongCache = {
@@ -737,6 +750,7 @@ export type StrongCache = {
     commentTTL: number | boolean,
     subredditTTL: number | boolean,
     selfTTL: number | boolean,
+    modNotesTTL: number | boolean,
     filterCriteriaTTL: number | boolean,
     provider: CacheOptions
     actionedEventsMax?: number,
@@ -1107,6 +1121,8 @@ export interface BotInstanceJsonConfig {
         retention?: EventRetentionPolicyRange
     }
 
+    influxConfig?: InfluxConfig
+
     /**
      * Settings related to bot behavior for subreddits it is managing
      * */
@@ -1362,6 +1378,8 @@ export interface OperatorJsonConfig {
         retention?: EventRetentionPolicyRange
     }
 
+    influxConfig?: InfluxConfig
+
     /**
      * Set global snoowrap options as well as default snoowrap config for all bots that don't specify their own
      * */
@@ -1568,6 +1586,7 @@ export interface BotInstanceConfig extends BotInstanceJsonConfig {
     database: DataSource
     snoowrap: SnoowrapOptions
     databaseStatisticsDefaults: DatabaseStatisticsOperatorConfig
+    opInflux?: InfluxClient,
     subreddits: {
         names?: string[],
         exclude?: string[],
@@ -1608,6 +1627,7 @@ export interface OperatorConfig extends OperatorJsonConfig {
         retention?: EventRetentionPolicyRange
     }
     database: DataSource
+    influx?: InfluxClient,
     web: {
         database: DataSource,
         databaseConfig: {
@@ -1676,6 +1696,7 @@ export interface LogInfo {
     labels?: string[]
     bot?: string
     user?: string
+    transport?: string[]
 }
 
 export interface ActionResult extends ActionProcessResult {
@@ -1888,8 +1909,6 @@ export interface TextMatchOptions {
     caseSensitive?: boolean
 }
 
-export type ActivityCheckJson = SubmissionCheckJson | CommentCheckJson;
-
 export interface PostBehaviorOptionConfig {
     recordTo?: RecordOutputOption
     behavior?: PostBehaviorType
@@ -1942,7 +1961,6 @@ export interface ActivityDispatch extends Omit<ActivityDispatchConfig, 'delay'|
     author: string
     delay: Duration
     tardyTolerant?: boolean | Duration
-    processing: boolean
     action?: string
     type: ActivitySourceTypes
     dryRun?: boolean

src/Common/types.ts

@@ -18,7 +18,10 @@ import {RepostRuleJSONConfig} from "../Rule/RepostRule";
 import {DispatchActionJson} from "../Action/DispatchAction";
 import {CancelDispatchActionJson} from "../Action/CancelDispatchAction";
 import {ContributorActionJson} from "../Action/ContributorAction";
+import {SentimentRuleJSONConfig} from "../Rule/SentimentRule";
+import {ModNoteActionJson} from "../Action/ModNoteAction";
+import {IncludesData} from "./Infrastructure/Includes";
 
-export type RuleObjectJsonTypes = RecentActivityRuleJSONConfig | RepeatActivityJSONConfig | AuthorRuleJSONConfig | AttributionJSONConfig | HistoryJSONConfig | RegexRuleJSONConfig | RepostRuleJSONConfig
+export type RuleObjectJsonTypes = RecentActivityRuleJSONConfig | RepeatActivityJSONConfig | AuthorRuleJSONConfig | AttributionJSONConfig | HistoryJSONConfig | RegexRuleJSONConfig | RepostRuleJSONConfig | SentimentRuleJSONConfig
 
-export type ActionJson = CommentActionJson | FlairActionJson | ReportActionJson | LockActionJson | RemoveActionJson | ApproveActionJson | BanActionJson | UserNoteActionJson | MessageActionJson | UserFlairActionJson | DispatchActionJson | CancelDispatchActionJson | ContributorActionJson | string;
+export type ActionJson = CommentActionJson | FlairActionJson | ReportActionJson | LockActionJson | RemoveActionJson | ApproveActionJson | BanActionJson | UserNoteActionJson | MessageActionJson | UserFlairActionJson | DispatchActionJson | CancelDispatchActionJson | ContributorActionJson | ModNoteActionJson | string | IncludesData;

src/ConfigBuilder.ts

@@ -1,24 +1,30 @@
 import winston, {Logger} from "winston";
 import {
-    asNamedCriteria,
-    buildCacheOptionsFromProvider, buildCachePrefix, buildFilter, castToBool,
+    asNamedCriteria, asWikiContext,
+    buildCachePrefix, buildFilter, castToBool,
     createAjvFactory, fileOrDirectoryIsWriteable,
     mergeArr, mergeFilters,
     normalizeName,
     overwriteMerge,
-    parseBool, randomId,
-    readConfigFile, removeFromSourceIfKeysExistsInDestination,
+    parseBool, parseExternalUrl, parseUrlContext, parseWikiContext, randomId,
+    readConfigFile,
     removeUndefinedKeys, resolvePathFromEnvWithRelative
 } from "./util";
-import {CommentCheck} from "./Check/CommentCheck";
-import {SubmissionCheck} from "./Check/SubmissionCheck";
 
 import Ajv, {Schema} from 'ajv';
 import * as appSchema from './Schema/App.json';
+import * as runSchema from './Schema/Run.json';
+import * as checkSchema from './Schema/Check.json';
 import * as operatorSchema from './Schema/OperatorConfig.json';
-import {JSONConfig} from "./JsonConfig";
+//import * as rulesetSchema from './Schema/RuleSet.json';
+import {SubredditConfigHydratedData, SubredditConfigData} from "./SubredditConfigData";
 import LoggedError from "./Utils/LoggedError";
-import {CheckStructuredJson} from "./Check";
+import {
+    ActivityCheckConfigData,
+    ActivityCheckConfigHydratedData,
+    CheckConfigHydratedData,
+    CheckConfigObject
+} from "./Check";
 import {
     DEFAULT_POLLING_INTERVAL,
     DEFAULT_POLLING_LIMIT,
@@ -37,15 +43,11 @@ import {
     OperatorFileConfig,
     PostBehavior
 } from "./Common/interfaces";
-import {isRuleSetJSON, RuleSetJson, RuleSetObjectJson} from "./Rule/RuleSet";
+import {isRuleSetJSON, RuleSetConfigData, RuleSetConfigHydratedData, RuleSetConfigObject} from "./Rule/RuleSet";
 import deepEqual from "fast-deep-equal";
-import {
-    ActionJson
-} from "./Common/types";
 import {isActionJson} from "./Action";
-import {getLogger, resolveLogDir} from "./Utils/loggerFactory";
+import {getLogger} from "./Utils/loggerFactory";
 import {GetEnvVars} from 'env-cmd';
-import {operatorConfig} from "./Utils/CommandConfig";
 import merge from 'deepmerge';
 import * as process from "process";
 import {
@@ -59,57 +61,53 @@ import objectHash from "object-hash";
 import {
     createAppDatabaseConnection,
     createDatabaseConfig,
-    createDatabaseConnection,
     createWebDatabaseConnection
 } from "./Utils/databaseUtils";
 import path from 'path';
 import {
     JsonOperatorConfigDocument,
-    OperatorConfigDocumentInterface,
     YamlOperatorConfigDocument
 } from "./Common/Config/Operator";
-import {
-    ConfigDocumentInterface
-} from "./Common/Config/AbstractConfigDocument";
 import {Document as YamlDocument} from "yaml";
-import {SimpleError} from "./Utils/Errors";
+import {CMError, SimpleError} from "./Utils/Errors";
 import {ErrorWithCause} from "pony-cause";
-import {RunJson, RunStructuredJson} from "./Run";
+import {RunConfigHydratedData, RunConfigData, RunConfigObject} from "./Run";
 import {AuthorRuleConfig} from "./Rule/AuthorRule";
 import {
-    CacheProvider, ConfigFormat,
+    CacheProvider, ConfigFormat, ConfigFragmentValidationFunc,
     PollOn
 } from "./Common/Infrastructure/Atomic";
 import {
-    AuthorOptions,
     FilterCriteriaDefaults,
     FilterCriteriaDefaultsJson,
-    FilterOptionsJson,
-    MaybeAnonymousCriteria,
     MaybeAnonymousOrStringCriteria, MinimalOrFullFilter, MinimalOrFullFilterJson, NamedCriteria
 } from "./Common/Infrastructure/Filters/FilterShapes";
-import {AuthorCriteria, TypedActivityState, TypedActivityStates} from "./Common/Infrastructure/Filters/FilterCriteria";
+import {AuthorCriteria, TypedActivityState} from "./Common/Infrastructure/Filters/FilterCriteria";
 import {StrongLoggingOptions} from "./Common/Infrastructure/Logging";
-import {DatabaseDriver, DatabaseDriverConfig, DatabaseDriverType} from "./Common/Infrastructure/Database";
+import {DatabaseDriver, DatabaseDriverType} from "./Common/Infrastructure/Database";
 import {parseFromJsonOrYamlToObject} from "./Common/Config/ConfigUtil";
-import {RunnableBaseJson, RunnableBaseOptions, StructuredRunnableBase} from "./Common/Infrastructure/Runnable";
+import {RunnableBaseJson, StructuredRunnableBase} from "./Common/Infrastructure/Runnable";
 import {
-    RuleJson,
-    RuleObjectJson,
-    StructuredRuleObjectJson,
-    StructuredRuleSetObjectJson
+    RuleConfigData, RuleConfigHydratedData,
+    RuleConfigObject,
 } from "./Common/Infrastructure/RuleShapes";
-import {ActionObjectJson, StructuredActionObjectJson} from "./Common/Infrastructure/ActionShapes";
+import {
+    ActionConfigHydratedData, ActionConfigObject,
+} from "./Common/Infrastructure/ActionShapes";
+import {SubredditResources} from "./Subreddit/SubredditResources";
+import {asIncludesData, IncludesData, IncludesString} from "./Common/Infrastructure/Includes";
+import ConfigParseError from "./Utils/ConfigParseError";
+import {InfluxClient} from "./Common/Influx/InfluxClient";
 
 export interface ConfigBuilderOptions {
     logger: Logger,
 }
 
-export const validateJson = (config: object, schema: Schema, logger: Logger): any => {
+export const validateJson = <T>(config: object, schema: Schema, logger: Logger): T => {
     const ajv = createAjvFactory(logger);
     const valid = ajv.validate(schema, config);
     if (valid) {
-        return config;
+        return config as unknown as T;
     } else {
         logger.error('Json config was not valid. Please use schema to check validity.', {leaf: 'Config'});
         if (Array.isArray(ajv.errors)) {
@@ -162,30 +160,217 @@ export class ConfigBuilder {
 
     constructor(options: ConfigBuilderOptions) {
 
-        this.configLogger = options.logger.child({leaf: 'Config'}, mergeArr);
+        this.configLogger = options.logger.child({labels: ['Config']}, mergeArr);
         this.logger = options.logger;
     }
 
-    validateJson(config: object): JSONConfig {
-        const validConfig = validateJson(config, appSchema, this.logger);
-        return validConfig as JSONConfig;
+    validateJson(config: object): SubredditConfigData {
+        return validateJson<SubredditConfigData>(config, appSchema, this.logger);
     }
 
-    parseToStructured(config: JSONConfig, filterCriteriaDefaultsFromBot?: FilterCriteriaDefaults, postCheckBehaviorDefaultsFromBot: PostBehavior = {}): RunStructuredJson[] {
-        let namedRules: Map<string, RuleObjectJson> = new Map();
-        let namedActions: Map<string, ActionObjectJson> = new Map();
-        const {checks = [], runs = [], filterCriteriaDefaults, postCheckBehaviorDefaults} = config;
+    async hydrateConfigFragment<T>(val: IncludesData | string | object, resource: SubredditResources, validateFunc?: ConfigFragmentValidationFunc): Promise<T[]> {
+        let includes: IncludesData | undefined = undefined;
+        if(typeof val === 'string') {
+            const strContextResult = parseUrlContext(val);
+            if(strContextResult !== undefined) {
+                this.configLogger.debug(`Detected ${asWikiContext(strContextResult) !== undefined ? 'REDDIT WIKI' : 'URL'} type Config Fragment from string: ${val}`);
+                includes = {
+                    path: val as IncludesString
+                };
+            } else {
+                this.configLogger.debug(`Did not detect Config Fragment as a URL resource: ${val}`);
+            }
+        } else if (asIncludesData(val)) {
+            includes = val;
+            const strContextResult = parseUrlContext(val.path);
+            if(strContextResult === undefined) {
+                throw new ConfigParseError(`Could not detect Config Fragment path as a valid URL Resource. Resource must be prefixed with either 'url:' or 'wiki:' -- ${val.path}`);
+            }
+        }
+
+        if(includes === undefined) {
+            if(Array.isArray(val)) {
+                return val as unknown as T[];
+            } else {
+                return [val as unknown as T];
+            }
+        }
+
+       const resolvedFragment = await resource.getConfigFragment(includes, validateFunc);
+        if(Array.isArray(resolvedFragment)) {
+            return resolvedFragment
+        }
+        return [resolvedFragment as T];
+    }
+
+    async hydrateConfig(config: SubredditConfigData, resource: SubredditResources): Promise<SubredditConfigHydratedData> {
+        const {
+            runs = [],
+            checks = [],
+            ...restConfig
+        } = config;
 
         if(checks.length > 0 && runs.length > 0) {
             // cannot have both checks and runs at top-level
             throw new Error(`Subreddit configuration cannot contain both 'checks' and 'runs' at top-level.`);
         }
 
+        // TODO consolidate this with parseToStructured
         const realRuns  = runs;
         if(checks.length > 0) {
             realRuns.push({name: 'Run1', checks: checks});
         }
 
+        const hydratedRuns: RunConfigHydratedData[] = [];
+
+        let runIndex = 1;
+        for(const r of realRuns) {
+
+            let hydratedRunArr: RunConfigData | RunConfigData[];
+
+            try {
+                hydratedRunArr = await this.hydrateConfigFragment<RunConfigData>(r, resource, <RunConfigData>(data: object, fetched: boolean) => {
+                    if (fetched) {
+                        if (Array.isArray(data)) {
+                            for (const runData of data) {
+                                validateJson<RunConfigData>(runData, runSchema, this.logger);
+                            }
+                        } else {
+                            validateJson<RunConfigData>(data, runSchema, this.logger);
+                        }
+                        return true;
+                    }
+                    return true;
+                });
+            } catch (e: any) {
+                throw new CMError(`Could not fetch or validate Run #${runIndex}`, {cause: e});
+            }
+
+            for(const hydratedRunVal of hydratedRunArr) {
+                if (typeof hydratedRunVal === 'string') {
+                    throw new ConfigParseError(`Run Config Fragment #${runIndex} was not in a recognized Config Fragment format. Given: ${hydratedRunVal}`);
+                }
+
+                // validate run with unhydrated checks
+                const preValidatedRun = hydratedRunVal as RunConfigData;
+
+                const {checks, ...rest} = preValidatedRun;
+
+                const hydratedChecks: CheckConfigHydratedData[] = [];
+                let checkIndex = 1;
+                for (const c of preValidatedRun.checks) {
+                    let hydratedCheckDataArr: ActivityCheckConfigHydratedData[];
+
+                    try {
+                        hydratedCheckDataArr = await this.hydrateConfigFragment<ActivityCheckConfigHydratedData>(c, resource, (data: object, fetched: boolean) => {
+                            if (fetched) {
+                                if (Array.isArray(data)) {
+                                    for (const checkObj of data) {
+                                        validateJson<ActivityCheckConfigHydratedData>(checkObj, checkSchema, this.logger);
+                                    }
+                                } else {
+                                    validateJson<ActivityCheckConfigHydratedData>(data, checkSchema, this.logger);
+                                }
+                                return true;
+                            }
+                            return true;
+                        });
+                    } catch (e: any) {
+                        throw new CMError(`Could not fetch or validate Check Config Fragment #${checkIndex} in Run #${runIndex}`, {cause: e});
+                    }
+
+                    for (const hydratedCheckData of hydratedCheckDataArr) {
+                        if (typeof hydratedCheckData === 'string') {
+                            throw new ConfigParseError(`Check #${checkIndex} in Run #${runIndex} was not in a recognized include format. Given: ${hydratedCheckData}`);
+                        }
+
+                        const preValidatedCheck = hydratedCheckData as ActivityCheckConfigHydratedData;
+
+                        const {rules, actions, ...rest} = preValidatedCheck;
+                        const hydratedCheckConfigData: CheckConfigHydratedData = rest;
+
+                        if (rules !== undefined) {
+                            const hydratedRulesOrSets: (RuleSetConfigHydratedData | RuleConfigHydratedData)[] = [];
+
+                            let ruleIndex = 1;
+                            for (const r of rules) {
+                                let hydratedRuleOrSetArr: (RuleConfigHydratedData | RuleSetConfigHydratedData)[];
+                                try {
+                                    hydratedRuleOrSetArr = await this.hydrateConfigFragment<(RuleSetConfigHydratedData | RuleConfigHydratedData)>(r, resource);
+                                } catch (e: any) {
+                                    throw new CMError(`Rule Config Fragment #${ruleIndex} in Check #${checkIndex} could not be fetched`, {cause: e});
+                                }
+                                for (const hydratedRuleOrSet of hydratedRuleOrSetArr) {
+                                    if (typeof hydratedRuleOrSet === 'string') {
+                                        hydratedRulesOrSets.push(hydratedRuleOrSet);
+                                    } else if (isRuleSetJSON(hydratedRuleOrSet)) {
+                                        const hydratedRulesetRules: RuleConfigHydratedData[] = [];
+                                        for (const rsr of hydratedRuleOrSet.rules) {
+                                            const hydratedRuleSetRuleArr = await this.hydrateConfigFragment<RuleConfigHydratedData>(rsr, resource);
+                                            for(const rsrData of hydratedRuleSetRuleArr) {
+                                                // either a string or rule data at this point
+                                                // we will validate the whole check again so this rule will be validated eventually
+                                                hydratedRulesetRules.push(rsrData)
+                                            }
+                                        }
+                                        hydratedRuleOrSet.rules = hydratedRulesetRules;
+                                        hydratedRulesOrSets.push(hydratedRuleOrSet);
+                                    } else {
+                                        hydratedRulesOrSets.push(hydratedRuleOrSet);
+                                    }
+                                    ruleIndex++;
+                                }
+                            }
+                            hydratedCheckConfigData.rules = hydratedRulesOrSets;
+                        }
+
+                        if (actions !== undefined) {
+                            const hydratedActions: ActionConfigHydratedData[] = [];
+
+                            let actionIndex = 1;
+                            for (const a of actions) {
+                                let hydratedActionArr: ActionConfigHydratedData[];
+                                try {
+                                    hydratedActionArr = await this.hydrateConfigFragment<ActionConfigHydratedData>(a, resource);
+                                } catch (e: any) {
+                                    throw new CMError(`Action Config Fragment #${actionIndex} in Check #${checkIndex} could not be fetched`, {cause: e});
+                                }
+                                for (const hydratedAction of hydratedActionArr) {
+                                    hydratedActions.push(hydratedAction);
+                                    actionIndex++;
+                                }
+                            }
+                            hydratedCheckConfigData.actions = hydratedActions;
+                        }
+
+                        hydratedChecks.push(hydratedCheckConfigData);
+                        checkIndex++;
+                    }
+                }
+
+                const hydratedRun: RunConfigHydratedData = {...rest, checks: hydratedChecks};
+
+                hydratedRuns.push(hydratedRun);
+                runIndex++;
+            }
+        }
+
+        const hydratedConfig: SubredditConfigHydratedData = {...restConfig, runs: hydratedRuns};
+
+        const validatedHydratedConfig = validateJson<SubredditConfigHydratedData>(hydratedConfig, appSchema, this.logger);
+
+        return validatedHydratedConfig;
+    }
+
+    async parseToStructured(config: SubredditConfigData, resource: SubredditResources, filterCriteriaDefaultsFromBot?: FilterCriteriaDefaults, postCheckBehaviorDefaultsFromBot: PostBehavior = {}): Promise<RunConfigObject[]> {
+        let namedRules: Map<string, RuleConfigObject> = new Map();
+        let namedActions: Map<string, ActionConfigObject> = new Map();
+        const {filterCriteriaDefaults, postCheckBehaviorDefaults} = config;
+
+        const hydratedConfig = await this.hydrateConfig(config, resource);
+
+        const {runs: realRuns = []} = hydratedConfig;
+
         for(const r of realRuns) {
             for (const c of r.checks) {
                 const {rules = [], actions = []} = c;
@@ -194,11 +379,11 @@ export class ConfigBuilder {
             }
         }
 
-        const [namedAuthorFilters, namedItemFilters] = extractNamedFilters({...config, runs: realRuns});
+        const [namedAuthorFilters, namedItemFilters] = extractNamedFilters({...hydratedConfig, runs: realRuns});
 
         const configFilterDefaults = filterCriteriaDefaults === undefined ? undefined : buildDefaultFilterCriteriaFromJson(filterCriteriaDefaults, namedAuthorFilters, namedItemFilters);
 
-        const structuredRuns: RunStructuredJson[] = [];
+        const structuredRuns: RunConfigObject[] = [];
 
         const namedFilters = insertNameFilters(namedAuthorFilters, namedItemFilters);
 
@@ -210,7 +395,7 @@ export class ConfigBuilder {
 
             const configFilterDefaultsFromRun = filterCriteriaDefaultsFromRun === undefined ? undefined : buildDefaultFilterCriteriaFromJson(filterCriteriaDefaultsFromRun, namedAuthorFilters, namedItemFilters);
 
-            const structuredChecks: CheckStructuredJson[] = [];
+            const structuredChecks: CheckConfigObject[] = [];
             for (const c of r.checks) {
                 const {rules = [], actions = [], authorIs = {}, itemIs = []} = c;
                 const strongRules = insertNamedRules(rules, namedRules, namedAuthorFilters, namedItemFilters);
@@ -227,7 +412,7 @@ export class ConfigBuilder {
                     rules: strongRules,
                     actions: strongActions,
                     ...postCheckBehaviors
-                } as CheckStructuredJson;
+                } as CheckConfigObject;
                 structuredChecks.push(strongCheck);
             }
             structuredRuns.push({
@@ -283,19 +468,19 @@ export const buildDefaultFilterCriteriaFromJson = (val: FilterCriteriaDefaultsJs
     return def;
 }
 
-export const extractNamedRules = (rules: Array<RuleSetJson | RuleJson>, namedRules: Map<string, RuleObjectJson> = new Map()): Map<string, RuleObjectJson> => {
+export const extractNamedRules = (rules: Array<RuleSetConfigData | RuleConfigData>, namedRules: Map<string, RuleConfigObject> = new Map()): Map<string, RuleConfigObject> => {
     //const namedRules = new Map();
     for (const r of rules) {
-        let rulesToAdd: RuleObjectJson[] = [];
+        let rulesToAdd: RuleConfigObject[] = [];
         if ((typeof r === 'object')) {
-            if ((r as RuleObjectJson).kind !== undefined) {
+            if ((r as RuleConfigObject).kind !== undefined) {
                 // itsa rule
-                const rule = r as RuleObjectJson;
+                const rule = r as RuleConfigObject;
                 if (rule.name !== undefined) {
                     rulesToAdd.push(rule);
                 }
             } else {
-                const ruleSet = r as RuleSetJson;
+                const ruleSet = r as RuleSetConfigData;
                 const nestedNamed = extractNamedRules(ruleSet.rules);
                 rulesToAdd = [...nestedNamed.values()];
             }
@@ -306,7 +491,7 @@ export const extractNamedRules = (rules: Array<RuleSetJson | RuleJson>, namedRul
                 const ruleNoName = {...rest};
 
                 if (namedRules.has(normalName)) {
-                    const {name: nn, ...ruleRest} = namedRules.get(normalName) as RuleObjectJson;
+                    const {name: nn, ...ruleRest} = namedRules.get(normalName) as RuleConfigObject;
                     if (!deepEqual(ruleRest, ruleNoName)) {
                         throw new Error(`Rule names must be unique (case-insensitive). Conflicting name: ${name}`);
                     }
@@ -352,7 +537,7 @@ const parseFilterJson = <T>(addToFilter: FilterJsonFuncArg<T>) => (val: MinimalO
     }
 }
 
-export const extractNamedFilters = (config: JSONConfig, namedAuthorFilters: Map<string, NamedCriteria<AuthorCriteria>> = new Map(), namedItemFilters: Map<string, NamedCriteria<TypedActivityState>> = new Map()): [Map<string, NamedCriteria<AuthorCriteria>>, Map<string, NamedCriteria<TypedActivityState>>] => {
+export const extractNamedFilters = (config: SubredditConfigHydratedData, namedAuthorFilters: Map<string, NamedCriteria<AuthorCriteria>> = new Map(), namedItemFilters: Map<string, NamedCriteria<TypedActivityState>> = new Map()): [Map<string, NamedCriteria<AuthorCriteria>>, Map<string, NamedCriteria<TypedActivityState>>] => {
     const addToAuthors = addToNamedFilter(namedAuthorFilters, 'authorIs');
     const addToItems = addToNamedFilter(namedItemFilters, 'itemIs');
 
@@ -407,7 +592,7 @@ export const extractNamedFilters = (config: JSONConfig, namedAuthorFilters: Map<
     parseAuthorIs(filterCriteriaDefaults?.authorIs);
     parseItemIs(filterCriteriaDefaults?.itemIs);
 
-    for (const r of runs as RunJson[]) {
+    for (const r of runs) {
 
         const {
             filterCriteriaDefaults: filterCriteriaDefaultsFromRun
@@ -508,13 +693,13 @@ export const insertNameFilters = (namedAuthorFilters: Map<string, NamedCriteria<
     return runnableOpts;
 }
 
-export const insertNamedRules = (rules: Array<RuleSetJson | RuleJson>, namedRules: Map<string, RuleObjectJson> = new Map(), namedAuthorFilters: Map<string, NamedCriteria<AuthorCriteria>> = new Map(), namedItemFilters: Map<string, NamedCriteria<TypedActivityState>> = new Map()): (StructuredRuleSetObjectJson | StructuredRuleObjectJson)[] => {
+export const insertNamedRules = (rules: Array<RuleSetConfigHydratedData | RuleConfigHydratedData>, namedRules: Map<string, RuleConfigObject> = new Map(), namedAuthorFilters: Map<string, NamedCriteria<AuthorCriteria>> = new Map(), namedItemFilters: Map<string, NamedCriteria<TypedActivityState>> = new Map()): (RuleSetConfigObject | RuleConfigObject)[] => {
 
     const namedFilters = insertNameFilters(namedAuthorFilters, namedItemFilters);
 
-    const strongRules: (StructuredRuleSetObjectJson | StructuredRuleObjectJson)[] = [];
+    const strongRules: (RuleSetConfigObject | RuleConfigObject)[] = [];
     for (const r of rules) {
-        let rule: StructuredRuleObjectJson | undefined;
+        let rule: RuleConfigObject | undefined;
         if (typeof r === 'string') {
             const foundRule = namedRules.get(r.toLowerCase());
             if (foundRule === undefined) {
@@ -523,20 +708,20 @@ export const insertNamedRules = (rules: Array<RuleSetJson | RuleJson>, namedRule
             rule = {
                 ...foundRule,
                 ...namedFilters(foundRule)
-            } as StructuredRuleObjectJson
+            } as RuleConfigObject
             //strongRules.push(foundRule);
         } else if (isRuleSetJSON(r)) {
             const {rules: sr, ...rest} = r;
             const setRules = insertNamedRules(sr, namedRules, namedAuthorFilters, namedItemFilters);
-            const strongSet = {rules: setRules, ...rest} as StructuredRuleSetObjectJson;
+            const strongSet = {rules: setRules, ...rest} as RuleSetConfigObject;
             strongRules.push(strongSet);
         } else {
-            rule = {...r, ...namedFilters(r)} as StructuredRuleObjectJson;
+            rule = {...r, ...namedFilters(r)} as RuleConfigObject;
         }
 
         if(rule !== undefined) {
             if(rule.kind === 'author') {
-                const authorRuleConfig = rule as (StructuredRuleObjectJson & AuthorRuleConfig);
+                const authorRuleConfig = rule as (RuleConfigObject & AuthorRuleConfig);
                 const filters = namedFilters({authorIs: {include: authorRuleConfig.include, exclude: authorRuleConfig.exclude}});
                 const builtFilter = buildFilter(filters.authorIs as MinimalOrFullFilter<AuthorCriteria>);
                 rule = {
@@ -546,14 +731,14 @@ export const insertNamedRules = (rules: Array<RuleSetJson | RuleJson>, namedRule
                     exclude: builtFilter.exclude
                 }
             }
-            strongRules.push(rule as StructuredRuleObjectJson);
+            strongRules.push(rule as RuleConfigObject);
         }
     }
 
     return strongRules;
 }
 
-export const extractNamedActions = (actions: Array<ActionJson>, namedActions: Map<string, ActionObjectJson> = new Map()): Map<string, ActionObjectJson> => {
+export const extractNamedActions = (actions: Array<ActionConfigHydratedData>, namedActions: Map<string, ActionConfigObject> = new Map()): Map<string, ActionConfigObject> => {
     for (const a of actions) {
         if (!(typeof a === 'string')) {
             if (isActionJson(a) && a.name !== undefined) {
@@ -562,7 +747,7 @@ export const extractNamedActions = (actions: Array<ActionJson>, namedActions: Ma
                 const actionNoName = {...rest};
                 if (namedActions.has(normalName)) {
                     // @ts-ignore
-                    const {name: nn, ...aRest} = namedActions.get(normalName) as ActionObjectJson;
+                    const {name: nn, ...aRest} = namedActions.get(normalName) as ActionConfigObject;
                     if (!deepEqual(aRest, actionNoName)) {
                         throw new Error(`Actions names must be unique (case-insensitive). Conflicting name: ${a.name}`);
                     }
@@ -575,20 +760,20 @@ export const extractNamedActions = (actions: Array<ActionJson>, namedActions: Ma
     return namedActions;
 }
 
-export const insertNamedActions = (actions: Array<ActionJson>, namedActions: Map<string, ActionObjectJson> = new Map(), namedAuthorFilters: Map<string, NamedCriteria<AuthorCriteria>> = new Map(), namedItemFilters: Map<string, NamedCriteria<TypedActivityState>> = new Map()): Array<StructuredActionObjectJson> => {
+export const insertNamedActions = (actions: Array<ActionConfigHydratedData>, namedActions: Map<string, ActionConfigObject> = new Map(), namedAuthorFilters: Map<string, NamedCriteria<AuthorCriteria>> = new Map(), namedItemFilters: Map<string, NamedCriteria<TypedActivityState>> = new Map()): Array<ActionConfigObject> => {
 
     const namedFilters = insertNameFilters(namedAuthorFilters, namedItemFilters);
 
-    const strongActions: Array<StructuredActionObjectJson> = [];
+    const strongActions: Array<ActionConfigObject> = [];
     for (const a of actions) {
         if (typeof a === 'string') {
             const foundAction = namedActions.get(a.toLowerCase());
             if (foundAction === undefined) {
                 throw new Error(`No named Action with the name ${a} was found`);
             }
-            strongActions.push({...foundAction, ...namedFilters(foundAction)} as StructuredActionObjectJson);
+            strongActions.push({...foundAction, ...namedFilters(foundAction)} as ActionConfigObject);
         } else {
-            strongActions.push({...a, ...namedFilters(a)} as StructuredActionObjectJson);
+            strongActions.push({...a, ...namedFilters(a)} as ActionConfigObject);
         }
     }
 
@@ -1024,6 +1209,7 @@ export const buildOperatorConfigWithDefaults = async (data: OperatorJsonConfig):
             migrations = {},
             retention,
         } = {},
+        influxConfig,
         web: {
             port = 8085,
             maxLogs = 200,
@@ -1141,6 +1327,13 @@ export const buildOperatorConfigWithDefaults = async (data: OperatorJsonConfig):
     }
     const webDbConfig = createDatabaseConfig(realdbConnectionWeb);
 
+    let influx: InfluxClient | undefined = undefined;
+    if(influxConfig !== undefined) {
+        const tags = friendly !== undefined ? {server: friendly} : undefined;
+        influx = new InfluxClient(influxConfig, appLogger, tags);
+        await influx.isReady();
+    }
+
     const config: OperatorConfig = {
         mode,
         operator: {
@@ -1160,6 +1353,7 @@ export const buildOperatorConfigWithDefaults = async (data: OperatorJsonConfig):
             migrations,
             retention,
         },
+        influx,
         userAgent,
         web: {
             database: await createWebDatabaseConnection(webDbConfig, appLogger),
@@ -1214,6 +1408,7 @@ export const buildBotConfig = (data: BotInstanceJsonConfig, opConfig: OperatorCo
         databaseConfig: {
             retention: retentionFromOp,
         } = {},
+        influx: opInflux
     } = opConfig;
     const {
         name: botName,
@@ -1239,6 +1434,7 @@ export const buildBotConfig = (data: BotInstanceJsonConfig, opConfig: OperatorCo
         databaseConfig: {
             retention,
         } = {},
+        influxConfig,
         flowControlDefaults,
         credentials = {},
         subreddits: {
@@ -1370,6 +1566,7 @@ export const buildBotConfig = (data: BotInstanceJsonConfig, opConfig: OperatorCo
         databaseConfig: {
           retention: retention ?? retentionFromOp
         },
+        opInflux,
         subreddits: {
             names,
             exclude,

src/JsonConfig.ts

@@ -1,23 +0,0 @@
-import {CheckJson, CommentCheckJson, SubmissionCheckJson} from "./Check";
-import {ActivityCheckJson, ManagerOptions} from "./Common/interfaces";
-import {RunJson} from "./Run";
-
-export interface JSONConfig extends ManagerOptions {
-    /**
-     * A list of all the checks that should be run for a subreddit.
-     *
-     * Checks are split into two lists -- submission or comment -- based on kind and run independently.
-     *
-     * Checks in each list are run in the order found in the configuration.
-     *
-     * When a check "passes", and actions are performed, then all subsequent checks are skipped.
-     * @minItems 1
-     * */
-    checks?: ActivityCheckJson[]
-
-    /**
-     * A list of sets of Checks to run
-     * @minItems 1
-     * */
-    runs?: RunJson[]
-}

src/Rule/AttributionRule.ts

@@ -10,10 +10,9 @@ import {getAttributionIdentifier} from "../Utils/SnoowrapUtils";
 import dayjs from "dayjs";
 import {
     asSubmission, buildFilter, buildSubredditFilter,
-    comparisonTextOp, convertSubredditsRawToStrong,
+    convertSubredditsRawToStrong,
     FAIL,
     formatNumber, getActivitySubredditName, isActivityWindowConfig, isSubmission,
-    parseGenericValueOrPercentComparison,
     parseSubredditName,
     PASS, windowConfigToWindowCriteria
 } from "../util";
@@ -27,6 +26,7 @@ import {
     HistoryFiltersOptions
 } from "../Common/Infrastructure/ActivityWindow";
 import {FilterOptions} from "../Common/Infrastructure/Filters/FilterShapes";
+import {comparisonTextOp, parseGenericValueOrPercentComparison} from "../Common/Infrastructure/Comparisons";
 
 
 export interface AttributionCriteria {

src/Rule/HistoryRule.ts

@@ -8,10 +8,9 @@ import Submission from "snoowrap/dist/objects/Submission";
 import dayjs from "dayjs";
 import {
     asSubmission,
-    comparisonTextOp,
     FAIL,
     formatNumber, getActivitySubredditName, historyFilterConfigToOptions, isSubmission,
-    parseGenericValueOrPercentComparison, parseSubredditName,
+    parseSubredditName,
     PASS,
     percentFromString, removeUndefinedKeys, toStrongSubredditState, windowConfigToWindowCriteria
 } from "../util";
@@ -20,6 +19,7 @@ import {SubredditCriteria} from "../Common/Infrastructure/Filters/FilterCriteria
 import {CompareValueOrPercent} from "../Common/Infrastructure/Atomic";
 import {ActivityWindowConfig, ActivityWindowCriteria} from "../Common/Infrastructure/ActivityWindow";
 import {ErrorWithCause} from "pony-cause";
+import {comparisonTextOp, parseGenericValueOrPercentComparison} from "../Common/Infrastructure/Comparisons";
 
 export interface CommentThresholdCriteria extends ThresholdCriteria {
     /**

src/Rule/RecentActivityRule.ts

@@ -11,7 +11,7 @@ import {
     asSubmission, bitsToHexLength,
     // blockHashImage,
     compareImages,
-    comparisonTextOp, convertSubredditsRawToStrong,
+    convertSubredditsRawToStrong,
     FAIL,
     formatNumber,
     getActivitySubredditName, imageCompareMaxConcurrencyGuess,
@@ -19,7 +19,7 @@ import {
     isSubmission,
     isValidImageURL,
     objectToStringSummary,
-    parseGenericValueOrPercentComparison, parseRedditEntity,
+    parseRedditEntity,
     parseStringToRegex,
     parseSubredditName,
     parseUsableLinkIdentifier,
@@ -41,6 +41,7 @@ import {
     SubredditCriteria
 } from "../Common/Infrastructure/Filters/FilterCriteria";
 import {ActivityWindow, ActivityWindowConfig} from "../Common/Infrastructure/ActivityWindow";
+import {comparisonTextOp, parseGenericValueOrPercentComparison} from "../Common/Infrastructure/Comparisons";
 
 const parseLink = parseUsableLinkIdentifier();
 

src/Rule/RegexRule.ts

@@ -3,8 +3,7 @@ import {Comment} from "snoowrap";
 import Submission from "snoowrap/dist/objects/Submission";
 import {
     asSubmission,
-    comparisonTextOp, FAIL, isExternalUrlSubmission, isSubmission, parseGenericValueComparison,
-    parseGenericValueOrPercentComparison, parseRegex, parseStringToRegex,
+    FAIL, isExternalUrlSubmission, isSubmission, parseRegex, parseStringToRegex,
     PASS, triggeredIndicator, windowConfigToWindowCriteria
 } from "../util";
 import {
@@ -14,6 +13,11 @@ import dayjs from 'dayjs';
 import {SimpleError} from "../Utils/Errors";
 import {JoinOperands} from "../Common/Infrastructure/Atomic";
 import {ActivityWindowConfig} from "../Common/Infrastructure/ActivityWindow";
+import {
+    comparisonTextOp,
+    parseGenericValueComparison,
+    parseGenericValueOrPercentComparison
+} from "../Common/Infrastructure/Comparisons";
 
 export interface RegexCriteria {
     /**
@@ -376,8 +380,10 @@ export class RegexRule extends Rule {
 
         for (const c of contents) {
             const results = parseRegex(reg, c);
-            if (results.matched) {
-                m = m.concat(results.matches);
+            if(results !== undefined) {
+                for(const r of results) {
+                    m.push(r.match);
+                }
             }
         }
         return m;

src/Rule/RepeatActivityRule.ts

@@ -3,12 +3,10 @@ import {Comment, RedditUser} from "snoowrap";
 import {
     activityWindowText,
     asSubmission,
-    comparisonTextOp,
     FAIL,
     getActivitySubredditName, isActivityWindowConfig,
     isExternalUrlSubmission,
     isRedditMedia,
-    parseGenericValueComparison,
     parseSubredditName,
     parseUsableLinkIdentifier as linkParser,
     PASS,
@@ -23,7 +21,6 @@ import {
 } from "../Common/interfaces";
 import Submission from "snoowrap/dist/objects/Submission";
 import dayjs from "dayjs";
-import Fuse from 'fuse.js'
 import {StrongSubredditCriteria, SubredditCriteria} from "../Common/Infrastructure/Filters/FilterCriteria";
 import {
     ActivityWindow,
@@ -31,6 +28,7 @@ import {
     ActivityWindowCriteria,
     HistoryFiltersOptions
 } from "../Common/Infrastructure/ActivityWindow";
+import {comparisonTextOp, parseGenericValueComparison} from "../Common/Infrastructure/Comparisons";
 
 const parseUsableLinkIdentifier = linkParser();
 

src/Rule/RepostRule.ts

@@ -3,11 +3,8 @@ import {Listing, SearchOptions} from "snoowrap";
 import Submission from "snoowrap/dist/objects/Submission";
 import Comment from "snoowrap/dist/objects/Comment";
 import {
-    compareDurationValue,
-    comparisonTextOp,
     FAIL, formatNumber,
-    isRepostItemResult, parseDurationComparison, parseGenericValueComparison,
-    parseUsableLinkIdentifier,
+    isRepostItemResult, parseUsableLinkIdentifier,
     PASS, searchAndReplace, stringSameness, triggeredIndicator, windowConfigToWindowCriteria, wordCount
 } from "../util";
 import {
@@ -18,13 +15,16 @@ import {
 } from "../Common/interfaces";
 import objectHash from "object-hash";
 import {getAttributionIdentifier} from "../Utils/SnoowrapUtils";
-import Fuse from "fuse.js";
-import leven from "leven";
 import {YoutubeClient, commentsAsRepostItems} from "../Utils/ThirdParty/YoutubeClient";
 import dayjs from "dayjs";
 import {rest} from "lodash";
 import {CompareValue, DurationComparor, JoinOperands, SearchFacetType} from "../Common/Infrastructure/Atomic";
 import {ActivityWindow, ActivityWindowConfig} from "../Common/Infrastructure/ActivityWindow";
+import {
+    compareDurationValue, comparisonTextOp,
+    parseDurationComparison,
+    parseGenericValueComparison
+} from "../Common/Infrastructure/Comparisons";
 
 const parseYtIdentifier = parseUsableLinkIdentifier();
 

src/Rule/RuleFactory.ts

@@ -1,6 +1,6 @@
 import {RecentActivityRule, RecentActivityRuleJSONConfig} from "./RecentActivityRule";
 import RepeatActivityRule, {RepeatActivityJSONConfig} from "./RepeatActivityRule";
-import {Rule, RuleJSONConfig, StructuredRuleJson} from "./index";
+import {Rule} from "./index";
 import AuthorRule, {AuthorRuleJSONConfig} from "./AuthorRule";
 import {AttributionJSONConfig, AttributionRule} from "./AttributionRule";
 import {Logger} from "winston";
@@ -10,9 +10,11 @@ import {SubredditResources} from "../Subreddit/SubredditResources";
 import Snoowrap from "snoowrap";
 import {RepostRule, RepostRuleJSONConfig} from "./RepostRule";
 import {StructuredFilter} from "../Common/Infrastructure/Filters/FilterShapes";
+import {SentimentRule, SentimentRuleJSONConfig} from "./SentimentRule";
+import {StructuredRuleConfigObject} from "../Common/Infrastructure/RuleShapes";
 
 export function ruleFactory
-(config: StructuredRuleJson, logger: Logger, subredditName: string, resources: SubredditResources, client: Snoowrap): Rule {
+(config: StructuredRuleConfigObject, logger: Logger, subredditName: string, resources: SubredditResources, client: Snoowrap): Rule {
     let cfg;
     switch (config.kind) {
         case 'recentActivity':
@@ -37,7 +39,10 @@ export function ruleFactory
         case 'repost':
             cfg = config as StructuredFilter<RepostRuleJSONConfig>;
             return new RepostRule({...cfg, logger, subredditName, resources, client});
+        case 'sentiment':
+            cfg = config as StructuredFilter<SentimentRuleJSONConfig>;
+            return new SentimentRule({...cfg, logger, subredditName, resources, client});
         default:
-            throw new Error('rule "kind" was not recognized.');
+            throw new Error(`Rule with kind '${config.kind}' was not recognized.`);
     }
 }

src/Rule/RuleSet.ts

@@ -1,17 +1,20 @@
-import {IRule, Triggerable, Rule, RuleJSONConfig, StructuredRuleJson} from "./index";
+import {IRule, Rule} from "./index";
 import Snoowrap, {Comment, Submission} from "snoowrap";
 import {ruleFactory} from "./RuleFactory";
 import {createAjvFactory, mergeArr} from "../util";
 import {Logger} from "winston";
-import {JoinCondition, RuleResult, RuleSetResult} from "../Common/interfaces";
+import {JoinCondition, RuleSetResult} from "../Common/interfaces";
 import * as RuleSchema from '../Schema/Rule.json';
-import Ajv from 'ajv';
 import {SubredditResources} from "../Subreddit/SubredditResources";
 import {runCheckOptions} from "../Subreddit/Manager";
 import {RuleResultEntity} from "../Common/Entities/RuleResultEntity";
-import {RuleSetResultEntity} from "../Common/Entities/RuleSetResultEntity";
 import {JoinOperands} from "../Common/Infrastructure/Atomic";
-import {RuleJson, RuleObjectJson, StructuredRuleObjectJson} from "../Common/Infrastructure/RuleShapes";
+import {
+    RuleConfigData,
+    RuleConfigHydratedData,
+    RuleConfigObject,
+    StructuredRuleConfigObject
+} from "../Common/Infrastructure/RuleShapes";
 
 export class RuleSet implements IRuleSet {
     rules: Rule[] = [];
@@ -99,7 +102,7 @@ export interface IRuleSet extends JoinCondition {
 }
 
 export interface RuleSetOptions extends IRuleSet {
-    rules: Array<StructuredRuleObjectJson>,
+    rules: Array<StructuredRuleConfigObject>,
     logger: Logger
     subredditName: string
     resources: SubredditResources
@@ -109,20 +112,24 @@ export interface RuleSetOptions extends IRuleSet {
 /**
  * A RuleSet is a "nested" set of `Rule` objects that can be used to create more complex AND/OR behavior. Think of the outcome of a `RuleSet` as the result of all of its run `Rule` objects (based on `condition`)
  * */
-export interface RuleSetJson extends JoinCondition {
+export interface RuleSetConfigData extends JoinCondition {
     /**
      * Can be `Rule` or the `name` of any **named** `Rule` in your subreddit's configuration
      * @minItems 1
      * */
-    rules: Array<RuleJson>
+    rules: RuleConfigData[]
 }
 
-export interface RuleSetObjectJson extends RuleSetJson {
-    rules: Array<RuleObjectJson>
+export interface RuleSetConfigHydratedData extends RuleSetConfigData {
+    rules: RuleConfigHydratedData[]
 }
 
-export const isRuleSetJSON = (obj: object): obj is RuleSetJson => {
-    return (obj as RuleSetJson).rules !== undefined;
+export interface RuleSetConfigObject extends RuleSetConfigHydratedData {
+    rules: RuleConfigObject[]
+}
+
+export const isRuleSetJSON = (obj: object): obj is RuleSetConfigData => {
+    return (obj as RuleSetConfigData).rules !== undefined;
 }
 
 export const isRuleSet = (obj: object): obj is RuleSet => {

src/Rule/SentimentRule.ts

@@ -0,0 +1,248 @@
+import {Rule, RuleJSONConfig, RuleOptions} from "./index";
+import {Comment} from "snoowrap";
+import Submission from "snoowrap/dist/objects/Submission";
+import {
+    formatNumber,
+    triggeredIndicator, windowConfigToWindowCriteria
+} from "../util";
+
+import dayjs from 'dayjs';
+import {map as mapAsync} from 'async';
+import {
+    comparisonTextOp,
+    GenericComparison,
+    parseGenericValueOrPercentComparison,
+    RangedComparison
+} from "../Common/Infrastructure/Comparisons";
+import {ActivityWindowConfig, ActivityWindowCriteria} from "../Common/Infrastructure/ActivityWindow";
+import {VaderSentimentComparison} from "../Common/Infrastructure/Atomic";
+import {RuleResult} from "../Common/interfaces";
+import {SnoowrapActivity} from "../Common/Infrastructure/Reddit";
+import {
+    ActivitySentimentOptions,
+    ActivitySentimentTestResult,
+    parseTextToNumberComparison,
+    testActivitySentiment
+} from "../Common/LangaugeProcessing";
+
+export class SentimentRule extends Rule {
+
+    sentimentVal: string;
+    sentiment: GenericComparison | RangedComparison;
+
+    historical?: HistoricalSentiment;
+
+    testOn: ('title' | 'body')[]
+
+    constructor(options: SentimentRuleOptions) {
+        super(options);
+
+        this.sentimentVal = options.sentiment;
+        this.sentiment = parseTextToNumberComparison(options.sentiment);
+        this.testOn = options.testOn ?? ['title', 'body'];
+
+        if(options.historical !== undefined) {
+            const {
+                window,
+                sentiment: historicalSentiment = this.sentimentVal,
+                mustMatchCurrent = false,
+                totalMatching = '> 0',
+            } = options.historical
+
+            this.historical = {
+                sentiment: parseTextToNumberComparison(historicalSentiment),
+                sentimentVal: historicalSentiment,
+                window: windowConfigToWindowCriteria(window),
+                mustMatchCurrent,
+                totalMatching: parseGenericValueOrPercentComparison(totalMatching),
+            };
+        }
+    }
+
+    getKind(): string {
+        return 'sentiment';
+    }
+
+    getSpecificPremise(): object {
+        return {
+            sentiment: this.sentiment,
+        }
+    }
+
+    protected async process(item: Submission | Comment): Promise<[boolean, RuleResult]> {
+
+        let ogResult = await this.testActivity(item, this.sentiment);
+        let historicResults: ActivitySentimentTestResult[] | undefined;
+
+        if(this.historical !== undefined && (!this.historical.mustMatchCurrent || ogResult.passes)) {
+            const {
+                sentiment = this.sentiment,
+                window,
+            } = this.historical;
+            const history = await this.resources.getAuthorActivities(item.author, window);
+
+            historicResults = await mapAsync(history, async (x: SnoowrapActivity) => await this.testActivity(x, sentiment)); // history.map(x => this.testActivity(x, sentiment));
+        }
+
+
+
+        const logSummary: string[] = [];
+
+        const sentimentTest = this.sentiment.displayText;
+        const historicalSentimentTest = this.historical !== undefined ? this.historical.sentiment.displayText : undefined;
+
+        let triggered = false;
+        let averageScore: number;
+        let averageWindowScore: number | undefined;
+        let humanWindow: string | undefined;
+        let historicalPassed: string | undefined;
+        let totalMatchingText: string | undefined;
+
+        if(historicResults === undefined) {
+            triggered = ogResult.passes;
+            averageScore = ogResult.scoreWeighted;
+            logSummary.push(`${triggeredIndicator(triggered)} Current Activity Sentiment '${ogResult.sentiment} (${ogResult.scoreWeighted})' ${triggered ? 'PASSED' : 'DID NOT PASS'} sentiment test '${sentimentTest}'`);
+            if(!triggered && this.historical !== undefined && this.historical.mustMatchCurrent) {
+                logSummary.push(`Did not check Historical because 'mustMatchCurrent' is true`);
+            }
+        } else {
+
+            const {
+                totalMatching,
+                sentiment,
+            } = this.historical as HistoricalSentiment;
+
+            totalMatchingText = totalMatching.displayText;
+            const allResults = historicResults
+            const passed = allResults.filter(x => x.passes);
+            averageScore = passed.reduce((acc, curr) => acc + curr.scoreWeighted,0) / passed.length;
+            averageWindowScore = allResults.reduce((acc, curr) => acc + curr.scoreWeighted,0) / allResults.length;
+
+            const firstActivity = allResults[0].activity;
+            const lastActivity = allResults[allResults.length - 1].activity;
+
+            const humanRange = dayjs.duration(dayjs(firstActivity.created_utc * 1000).diff(dayjs(lastActivity.created_utc * 1000))).humanize();
+
+            humanWindow = `${allResults.length} Activities (${humanRange})`;
+
+            const {operator, value, isPercent} = totalMatching;
+            if(isPercent) {
+                const passPercentVal = passed.length/allResults.length
+                triggered = comparisonTextOp(passPercentVal, operator, (value/100));
+                historicalPassed = `${passed.length} (${formatNumber(passPercentVal)}%)`;
+            } else {
+                triggered = comparisonTextOp(passed.length, operator, value);
+                historicalPassed = `${passed.length}`;
+            }
+            logSummary.push(`${triggeredIndicator(triggered)} ${historicalPassed} historical activities of ${humanWindow} passed sentiment test '${sentiment.displayText}' which ${triggered ? 'MET' : 'DID NOT MEET'} threshold '${totalMatching.displayText}'`);
+        }
+
+        const result = logSummary.join(' || ');
+        this.logger.verbose(result);
+
+        return Promise.resolve([triggered, this.getResult(triggered, {
+            result,
+            data: {
+                results: {
+                    triggered,
+                    sentimentTest,
+                    historicalSentimentTest,
+                    averageScore,
+                    averageWindowScore,
+                    window: humanWindow,
+                    totalMatching: totalMatchingText
+                }
+            }
+        })]);
+    }
+
+    protected async testActivity(a: (Submission | Comment), criteria: GenericComparison | RangedComparison): Promise<ActivitySentimentTestResult> {
+        return await testActivitySentiment(a, criteria, {testOn: this.testOn});
+    }
+}
+
+/**
+ * Test the Sentiment of Activities from the Author history
+ *
+ * If this is defined then the `totalMatching` threshold must pass for the Rule to trigger
+ *
+ * If `sentiment` is defined here it overrides the top-level `sentiment` value
+ *
+ * */
+interface HistoricalSentimentConfig {
+    window: ActivityWindowConfig
+
+    sentiment?: VaderSentimentComparison
+
+    /**
+     * When `true` the original Activity being checked MUST match desired sentiment before the Rule considers any history
+     *
+     * @default false
+     * */
+    mustMatchCurrent?: boolean
+
+    /**
+     * A string containing a comparison operator and a value to compare Activities from history that pass the given `sentiment` comparison
+     *
+     * The syntax is `(< OR > OR <= OR >=) <number>[percent sign]`
+     *
+     * * EX `> 12`  => greater than 12 activities passed given `sentiment` comparison
+     * * EX `<= 10%` => less than 10% of all Activities from history passed given `sentiment` comparison
+     *
+     * @pattern ^\s*(>|>=|<|<=)\s*(\d+)\s*(%?)(.*)$
+     * @default "> 0"
+     * @examples ["> 0","> 10%"]
+     * */
+    totalMatching: string
+}
+
+interface HistoricalSentiment extends Omit<HistoricalSentimentConfig, 'sentiment' | 'window' | 'totalMatching'> {
+    sentiment: GenericComparison | RangedComparison,
+    sentimentVal: string
+    window: ActivityWindowCriteria
+    totalMatching: GenericComparison
+}
+
+interface SentimentConfig extends ActivitySentimentOptions {
+
+    sentiment: VaderSentimentComparison
+
+    /**
+     * Test the Sentiment of Activities from the Author history
+     *
+     * If this is defined then the `totalMatching` threshold must pass for the Rule to trigger
+     *
+     * If `sentiment` is defined here it overrides the top-level `sentiment` value
+     *
+     * */
+    historical?: HistoricalSentimentConfig
+
+    /**
+     * Which content from an Activity to test for `sentiment` against
+     *
+     * Only used if the Activity being tested is a Submission -- Comments are only tested against their body
+     *
+     * If more than one type of content is specified then all text is tested together as one string
+     *
+     * @default ["title", "body"]
+     * */
+    testOn?: ('title' | 'body')[]
+}
+
+export interface SentimentRuleOptions extends SentimentConfig, RuleOptions {
+}
+
+/**
+ * Test the calculated VADER sentiment for an Activity to determine if the text context is negative, neutral, or positive in tone.
+ *
+ * More about VADER Sentiment: https://github.com/cjhutto/vaderSentiment
+ *
+ * */
+export interface SentimentRuleJSONConfig extends SentimentConfig, RuleJSONConfig {
+    /**
+     * @examples ["sentiment"]
+     * */
+    kind: 'sentiment'
+}
+
+export default SentimentRule;

src/Rule/index.ts

@@ -185,10 +185,5 @@ export interface RuleJSONConfig extends IRule {
      * The kind of rule to run
      * @examples ["recentActivity", "repeatActivity", "author", "attribution", "history"]
      */
-    kind: 'recentActivity' | 'repeatActivity' | 'author' | 'attribution' | 'history' | 'regex' | 'repost'
-}
-
-
-export interface StructuredRuleJson extends Omit<RuleJSONConfig, 'authorIs' | 'itemIs'>, StructuredRunnableBase {
-
+    kind: 'recentActivity' | 'repeatActivity' | 'author' | 'attribution' | 'history' | 'regex' | 'repost' | 'sentiment'
 }

src/Run/index.ts

@@ -1,6 +1,11 @@
-import {asStructuredCommentCheckJson, asStructuredSubmissionCheckJson, Check, CheckStructuredJson} from "../Check";
 import {
-    ActivityCheckJson,
+    ActivityCheckConfigHydratedData,
+    ActivityCheckConfigValue, ActivityCheckObject,
+    asStructuredCommentCheckJson,
+    asStructuredSubmissionCheckJson,
+    Check,
+} from "../Check";
+import {
     PostBehavior, PostBehaviorOption,
     RunResult
 } from "../Common/interfaces";
@@ -20,6 +25,7 @@ import {RuleResultEntity} from "../Common/Entities/RuleResultEntity";
 import {RunnableBase} from "../Common/RunnableBase";
 import {RunnableBaseJson, RunnableBaseOptions, StructuredRunnableBase} from "../Common/Infrastructure/Runnable";
 import {FilterCriteriaDefaults} from "../Common/Infrastructure/Filters/FilterShapes";
+import {IncludesData} from "../Common/Infrastructure/Includes";
 
 export class Run extends RunnableBase {
     name: string;
@@ -296,7 +302,7 @@ export interface IRun extends PostBehavior, RunnableBaseJson {
     enable?: boolean,
 }
 
-export interface RunOptions extends RunStructuredJson, RunnableBaseOptions {
+export interface RunOptions extends RunConfigObject, RunnableBaseOptions {
     // submissionChecks?: SubmissionCheck[]
     // commentChecks?: CommentCheck[]
     //checks: CheckStructuredJson[]
@@ -308,10 +314,16 @@ export interface RunOptions extends RunStructuredJson, RunnableBaseOptions {
     emitter: EventEmitter;
 }
 
-export interface RunJson extends IRun {
-    checks: ActivityCheckJson[]
+export interface RunConfigData extends IRun {
+    checks: ActivityCheckConfigValue[]
 }
 
-export interface RunStructuredJson extends Omit<RunJson, 'authorIs' | 'itemIs' | 'checks'>, StructuredRunnableBase {
-    checks: CheckStructuredJson[]
+export type RunConfigValue = string | IncludesData | RunConfigData;
+
+export interface RunConfigHydratedData extends IRun {
+    checks: ActivityCheckConfigHydratedData[]
+}
+
+export interface RunConfigObject extends Omit<RunConfigHydratedData, 'authorIs' | 'itemIs'>, StructuredRunnableBase {
+    checks: ActivityCheckObject[]
 }

src/Schema/OperatorConfig.json

@@ -119,6 +119,19 @@
                     "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
                     "type": "string"
                 },
+                "modActions": {
+                    "items": {
+                        "anyOf": [
+                            {
+                                "$ref": "#/definitions/ModNoteCriteria"
+                            },
+                            {
+                                "$ref": "#/definitions/ModLogCriteria"
+                            }
+                        ]
+                    },
+                    "type": "array"
+                },
                 "name": {
                     "description": "A list of reddit usernames (case-insensitive) to match against. Do not include the \"u/\" prefix\n\n EX to match against /u/FoxxMD and /u/AnotherUser use [\"FoxxMD\",\"AnotherUser\"]",
                     "examples": [
@@ -245,6 +258,9 @@
                     },
                     "type": "object"
                 },
+                "influxConfig": {
+                    "$ref": "#/definitions/InfluxConfig"
+                },
                 "name": {
                     "type": "string"
                 },
@@ -565,8 +581,8 @@
                     "description": "* true/false => test whether Activity is removed or not\n* string or list of strings => test which moderator removed this Activity"
                 },
                 "reports": {
-                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 2`  => greater than 2 total reports\n\nDefaults to TOTAL reports on an Activity. Suffix the value with the report type to check that type:\n\n* EX `> 3 mod` => greater than 3 mod reports\n* EX `>= 1 user` => greater than 1 user report",
-                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "description": "A string containing a comparison operator, a value to compare against, an (optional) report type filter, an (optional) qualifier for report reason, and an (optional) time constraint\n\nThe syntax is `(< OR > OR <= OR >=) number[%] [type] [reasonQualifier] [timeValue] [timeUnit]`\n\nIf only comparison and number is given then defaults to TOTAL reports on an Activity.\n\n* EX `> 2`  => greater than 2 total reports\n\nType (optional) determines which type of reports to look at:\n\n* `mod` -- mod reports\n  * EX `> 3 mod` => greater than 3 mod reports\n* `user` -- user reports\n  * EX `> 3 user` => greater than 3 user reports\n\nReport reason qualifiers can be:\n\n* enclosed double or single quotes -- report reason contains\n  * EX `> 1 \"misinformation\" => greater than 1 report with reason containing \"misinformation\"\n* enclosed in backslashes -- match regex\n  * EX `> 1 \\harassment towards .*\\` => greater than 1 report with reason matching regex \\harassment towards .*\\\n\nType and reason qualifiers can be used together:\n\nEX `> 2 user \"misinformation\" => greater than 2 user reports with reasons containing \"misinformation\"\n\nThe time constraint filter reports created between NOW and [timeConstraint] in the past:\n\n* `> 3 in 30 minutes` => more than 3 reports created between NOW and 30 minutes ago\n* `> 2 user \"misinformation\" in 2 hours` => more than 2 user reports containing \"misinformation\" created between NOW and 2 hours ago",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s*%)?(\\s+(?:mods?|users?))?(\\s+(?:[\"'].*[\"']|\\/.*\\/))?.*(\\d+)?\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)?\\s*$",
                     "type": "string"
                 },
                 "score": {
@@ -853,6 +869,43 @@
             },
             "type": "object"
         },
+        "InfluxConfig": {
+            "properties": {
+                "credentials": {
+                    "$ref": "#/definitions/InfluxCredentials"
+                },
+                "defaultTags": {
+                    "$ref": "#/definitions/Record<string,string>"
+                }
+            },
+            "required": [
+                "credentials"
+            ],
+            "type": "object"
+        },
+        "InfluxCredentials": {
+            "properties": {
+                "bucket": {
+                    "type": "string"
+                },
+                "org": {
+                    "type": "string"
+                },
+                "token": {
+                    "type": "string"
+                },
+                "url": {
+                    "type": "string"
+                }
+            },
+            "required": [
+                "bucket",
+                "org",
+                "token",
+                "url"
+            ],
+            "type": "object"
+        },
         "LoggingOptions": {
             "properties": {
                 "console": {
@@ -965,6 +1018,241 @@
             },
             "type": "object"
         },
+        "ModLogCriteria": {
+            "properties": {
+                "action": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "activityType": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "comment",
+                                    "submission"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "comment",
+                                "submission"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                },
+                "count": {
+                    "default": ">= 1",
+                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [in timeRange] [ascending|descending]`\n\nIf `timeRange` is given then only notes/mod actions that occur between timeRange and NOW will be returned. `timeRange` is ignored if search is `current`",
+                    "examples": [
+                        ">= 1"
+                    ],
+                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<duration>in\\s+\\d+\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?))?\\s*(?<extra>asc.*|desc.*)*$",
+                    "type": "string"
+                },
+                "description": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "details": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "search": {
+                    "default": "current",
+                    "description": "How to test the Toolbox Notes or Mod Actions for this Author:\n\n### current\n\nOnly the most recent note is checked for criteria\n\n### total\n\n`count` comparison of mod actions/notes must be found within all history\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n* EX: `count: > 3 in 1 week` => Must have more than 3 notes within the last week\n\n### consecutive\n\nThe `count` **number** of mod actions/notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
+                    "enum": [
+                        "consecutive",
+                        "current",
+                        "total"
+                    ],
+                    "examples": [
+                        "current"
+                    ],
+                    "type": "string"
+                },
+                "type": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "APPROVAL",
+                                    "INVITE",
+                                    "NOTE",
+                                    "REMOVAL",
+                                    "SPAM"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "APPROVAL",
+                                "INVITE",
+                                "NOTE",
+                                "REMOVAL",
+                                "SPAM"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                }
+            },
+            "type": "object"
+        },
+        "ModNoteCriteria": {
+            "properties": {
+                "activityType": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "comment",
+                                    "submission"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "comment",
+                                "submission"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                },
+                "count": {
+                    "default": ">= 1",
+                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [in timeRange] [ascending|descending]`\n\nIf `timeRange` is given then only notes/mod actions that occur between timeRange and NOW will be returned. `timeRange` is ignored if search is `current`",
+                    "examples": [
+                        ">= 1"
+                    ],
+                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<duration>in\\s+\\d+\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?))?\\s*(?<extra>asc.*|desc.*)*$",
+                    "type": "string"
+                },
+                "note": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "noteType": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "ABUSE_WARNING",
+                                    "BAN",
+                                    "BOT_BAN",
+                                    "HELPFUL_USER",
+                                    "PERMA_BAN",
+                                    "SOLID_CONTRIBUTOR",
+                                    "SPAM_WARNING",
+                                    "SPAM_WATCH"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "ABUSE_WARNING",
+                                "BAN",
+                                "BOT_BAN",
+                                "HELPFUL_USER",
+                                "PERMA_BAN",
+                                "SOLID_CONTRIBUTOR",
+                                "SPAM_WARNING",
+                                "SPAM_WATCH"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                },
+                "search": {
+                    "default": "current",
+                    "description": "How to test the Toolbox Notes or Mod Actions for this Author:\n\n### current\n\nOnly the most recent note is checked for criteria\n\n### total\n\n`count` comparison of mod actions/notes must be found within all history\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n* EX: `count: > 3 in 1 week` => Must have more than 3 notes within the last week\n\n### consecutive\n\nThe `count` **number** of mod actions/notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
+                    "enum": [
+                        "consecutive",
+                        "current",
+                        "total"
+                    ],
+                    "examples": [
+                        "current"
+                    ],
+                    "type": "string"
+                },
+                "type": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "APPROVAL",
+                                    "INVITE",
+                                    "NOTE",
+                                    "REMOVAL",
+                                    "SPAM"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "APPROVAL",
+                                "INVITE",
+                                "NOTE",
+                                "REMOVAL",
+                                "SPAM"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                }
+            },
+            "type": "object"
+        },
         "ModeratorNameCriteria": {
             "properties": {
                 "behavior": {
@@ -1226,6 +1514,17 @@
                         "boolean"
                     ]
                 },
+                "modNotesTTL": {
+                    "default": 60,
+                    "description": "Amount of time, in seconds, Mod Notes should be cached\n\n* If `0` or `true` will cache indefinitely (not recommended)\n* If `false` will not cache",
+                    "examples": [
+                        60
+                    ],
+                    "type": [
+                        "number",
+                        "boolean"
+                    ]
+                },
                 "provider": {
                     "anyOf": [
                         {
@@ -1410,6 +1709,9 @@
             },
             "type": "object"
         },
+        "Record<string,string>": {
+            "type": "object"
+        },
         "RedditCredentials": {
             "description": "Credentials required for the bot to interact with Reddit's API\n\nThese credentials will provided to both the API and Web interface unless otherwise specified with the `web.credentials` property\n\nRefer to the [required credentials table](https://github.com/FoxxMD/context-mod/blob/master/docs/operatorConfiguration.md#minimum-required-configuration) to see what is necessary to run the bot.",
             "examples": [
@@ -1647,8 +1949,8 @@
                     "description": "* true/false => test whether Activity is removed or not\n* string or list of strings => test which moderator removed this Activity"
                 },
                 "reports": {
-                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 2`  => greater than 2 total reports\n\nDefaults to TOTAL reports on an Activity. Suffix the value with the report type to check that type:\n\n* EX `> 3 mod` => greater than 3 mod reports\n* EX `>= 1 user` => greater than 1 user report",
-                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "description": "A string containing a comparison operator, a value to compare against, an (optional) report type filter, an (optional) qualifier for report reason, and an (optional) time constraint\n\nThe syntax is `(< OR > OR <= OR >=) number[%] [type] [reasonQualifier] [timeValue] [timeUnit]`\n\nIf only comparison and number is given then defaults to TOTAL reports on an Activity.\n\n* EX `> 2`  => greater than 2 total reports\n\nType (optional) determines which type of reports to look at:\n\n* `mod` -- mod reports\n  * EX `> 3 mod` => greater than 3 mod reports\n* `user` -- user reports\n  * EX `> 3 user` => greater than 3 user reports\n\nReport reason qualifiers can be:\n\n* enclosed double or single quotes -- report reason contains\n  * EX `> 1 \"misinformation\" => greater than 1 report with reason containing \"misinformation\"\n* enclosed in backslashes -- match regex\n  * EX `> 1 \\harassment towards .*\\` => greater than 1 report with reason matching regex \\harassment towards .*\\\n\nType and reason qualifiers can be used together:\n\nEX `> 2 user \"misinformation\" => greater than 2 user reports with reasons containing \"misinformation\"\n\nThe time constraint filter reports created between NOW and [timeConstraint] in the past:\n\n* `> 3 in 30 minutes` => more than 3 reports created between NOW and 30 minutes ago\n* `> 2 user \"misinformation\" in 2 hours` => more than 2 user reports containing \"misinformation\" created between NOW and 2 hours ago",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s*%)?(\\s+(?:mods?|users?))?(\\s+(?:[\"'].*[\"']|\\/.*\\/))?.*(\\d+)?\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)?\\s*$",
                     "type": "string"
                 },
                 "score": {
@@ -1682,6 +1984,13 @@
                 "title": {
                     "description": "A valid regular expression to match against the title of the submission",
                     "type": "string"
+                },
+                "upvoteRatio": {
+                    "description": "Compare the upvote ratio for this Submission, expressed as a whole number\n\nCan be either a comparison string or a number. If a number then CM assumes upvote ratio must be greater than or equal to this.\n\nExample:\n\n* `< 90` => less than 90% upvoted\n* 45 => greater than or equal to 45% upvoted",
+                    "type": [
+                        "string",
+                        "number"
+                    ]
                 }
             },
             "type": "object"
@@ -1749,16 +2058,16 @@
             "properties": {
                 "count": {
                     "default": ">= 1",
-                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [ascending|descending]`",
+                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [in timeRange] [ascending|descending]`\n\nIf `timeRange` is given then only notes/mod actions that occur between timeRange and NOW will be returned. `timeRange` is ignored if search is `current`",
                     "examples": [
                         ">= 1"
                     ],
-                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<extra>asc.*|desc.*)*$",
+                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<duration>in\\s+\\d+\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?))?\\s*(?<extra>asc.*|desc.*)*$",
                     "type": "string"
                 },
                 "search": {
                     "default": "current",
-                    "description": "How to test the notes for this Author:\n\n### current\n\nOnly the most recent note is checked for `type`\n\n### total\n\nThe `count` comparison of `type` must be found within all notes\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n\n### consecutive\n\nThe `count` **number** of `type` notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
+                    "description": "How to test the Toolbox Notes or Mod Actions for this Author:\n\n### current\n\nOnly the most recent note is checked for criteria\n\n### total\n\n`count` comparison of mod actions/notes must be found within all history\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n* EX: `count: > 3 in 1 week` => Must have more than 3 notes within the last week\n\n### consecutive\n\nThe `count` **number** of mod actions/notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
                     "enum": [
                         "consecutive",
                         "current",
@@ -1934,6 +2243,9 @@
             "$ref": "#/definitions/DatabaseStatisticsOperatorJsonConfig",
             "description": "Set defaults for the frequency time series stats are collected"
         },
+        "influxConfig": {
+            "$ref": "#/definitions/InfluxConfig"
+        },
         "logging": {
             "$ref": "#/definitions/LoggingOptions",
             "description": "Settings to configure global logging defaults"

src/Schema/Rule.json

@@ -1,6 +1,9 @@
 {
     "$schema": "http://json-schema.org/draft-07/schema#",
     "anyOf": [
+        {
+            "$ref": "#/definitions/IncludesData"
+        },
         {
             "$ref": "#/definitions/RecentActivityRuleJSONConfig"
         },
@@ -22,6 +25,9 @@
         {
             "$ref": "#/definitions/RepostRuleJSONConfig"
         },
+        {
+            "$ref": "#/definitions/SentimentRuleJSONConfig"
+        },
         {
             "type": "string"
         }
@@ -104,8 +110,8 @@
                     "description": "* true/false => test whether Activity is removed or not\n* string or list of strings => test which moderator removed this Activity"
                 },
                 "reports": {
-                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 2`  => greater than 2 total reports\n\nDefaults to TOTAL reports on an Activity. Suffix the value with the report type to check that type:\n\n* EX `> 3 mod` => greater than 3 mod reports\n* EX `>= 1 user` => greater than 1 user report",
-                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "description": "A string containing a comparison operator, a value to compare against, an (optional) report type filter, an (optional) qualifier for report reason, and an (optional) time constraint\n\nThe syntax is `(< OR > OR <= OR >=) number[%] [type] [reasonQualifier] [timeValue] [timeUnit]`\n\nIf only comparison and number is given then defaults to TOTAL reports on an Activity.\n\n* EX `> 2`  => greater than 2 total reports\n\nType (optional) determines which type of reports to look at:\n\n* `mod` -- mod reports\n  * EX `> 3 mod` => greater than 3 mod reports\n* `user` -- user reports\n  * EX `> 3 user` => greater than 3 user reports\n\nReport reason qualifiers can be:\n\n* enclosed double or single quotes -- report reason contains\n  * EX `> 1 \"misinformation\" => greater than 1 report with reason containing \"misinformation\"\n* enclosed in backslashes -- match regex\n  * EX `> 1 \\harassment towards .*\\` => greater than 1 report with reason matching regex \\harassment towards .*\\\n\nType and reason qualifiers can be used together:\n\nEX `> 2 user \"misinformation\" => greater than 2 user reports with reasons containing \"misinformation\"\n\nThe time constraint filter reports created between NOW and [timeConstraint] in the past:\n\n* `> 3 in 30 minutes` => more than 3 reports created between NOW and 30 minutes ago\n* `> 2 user \"misinformation\" in 2 hours` => more than 2 user reports containing \"misinformation\" created between NOW and 2 hours ago",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s*%)?(\\s+(?:mods?|users?))?(\\s+(?:[\"'].*[\"']|\\/.*\\/))?.*(\\d+)?\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)?\\s*$",
                     "type": "string"
                 },
                 "score": {
@@ -574,6 +580,19 @@
                     "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
                     "type": "string"
                 },
+                "modActions": {
+                    "items": {
+                        "anyOf": [
+                            {
+                                "$ref": "#/definitions/ModNoteCriteria"
+                            },
+                            {
+                                "$ref": "#/definitions/ModLogCriteria"
+                            }
+                        ]
+                    },
+                    "type": "array"
+                },
                 "name": {
                     "description": "A list of reddit usernames (case-insensitive) to match against. Do not include the \"u/\" prefix\n\n EX to match against /u/FoxxMD and /u/AnotherUser use [\"FoxxMD\",\"AnotherUser\"]",
                     "examples": [
@@ -803,8 +822,8 @@
                     "description": "* true/false => test whether Activity is removed or not\n* string or list of strings => test which moderator removed this Activity"
                 },
                 "reports": {
-                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 2`  => greater than 2 total reports\n\nDefaults to TOTAL reports on an Activity. Suffix the value with the report type to check that type:\n\n* EX `> 3 mod` => greater than 3 mod reports\n* EX `>= 1 user` => greater than 1 user report",
-                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "description": "A string containing a comparison operator, a value to compare against, an (optional) report type filter, an (optional) qualifier for report reason, and an (optional) time constraint\n\nThe syntax is `(< OR > OR <= OR >=) number[%] [type] [reasonQualifier] [timeValue] [timeUnit]`\n\nIf only comparison and number is given then defaults to TOTAL reports on an Activity.\n\n* EX `> 2`  => greater than 2 total reports\n\nType (optional) determines which type of reports to look at:\n\n* `mod` -- mod reports\n  * EX `> 3 mod` => greater than 3 mod reports\n* `user` -- user reports\n  * EX `> 3 user` => greater than 3 user reports\n\nReport reason qualifiers can be:\n\n* enclosed double or single quotes -- report reason contains\n  * EX `> 1 \"misinformation\" => greater than 1 report with reason containing \"misinformation\"\n* enclosed in backslashes -- match regex\n  * EX `> 1 \\harassment towards .*\\` => greater than 1 report with reason matching regex \\harassment towards .*\\\n\nType and reason qualifiers can be used together:\n\nEX `> 2 user \"misinformation\" => greater than 2 user reports with reasons containing \"misinformation\"\n\nThe time constraint filter reports created between NOW and [timeConstraint] in the past:\n\n* `> 3 in 30 minutes` => more than 3 reports created between NOW and 30 minutes ago\n* `> 2 user \"misinformation\" in 2 hours` => more than 2 user reports containing \"misinformation\" created between NOW and 2 hours ago",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s*%)?(\\s+(?:mods?|users?))?(\\s+(?:[\"'].*[\"']|\\/.*\\/))?.*(\\d+)?\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)?\\s*$",
                     "type": "string"
                 },
                 "score": {
@@ -1297,6 +1316,60 @@
             },
             "type": "object"
         },
+        "HistoricalSentimentConfig": {
+            "description": "Test the Sentiment of Activities from the Author history\n\nIf this is defined then the `totalMatching` threshold must pass for the Rule to trigger\n\nIf `sentiment` is defined here it overrides the top-level `sentiment` value",
+            "properties": {
+                "mustMatchCurrent": {
+                    "default": false,
+                    "description": "When `true` the original Activity being checked MUST match desired sentiment before the Rule considers any history",
+                    "type": "boolean"
+                },
+                "sentiment": {
+                    "description": "Test the calculated VADER sentiment (compound) score for an Activity using this comparison. Can be either a numerical or natural language\n\nSentiment values range from extremely negative to extremely positive in a numerical range of -1 to +1:\n\n* -0.6 => extremely negative\n* -0.3 => very negative\n* -0.1 => negative\n*    0 => neutral\n*  0.1 => positive\n*  0.3 => very positive\n*  0.6 => extremely positive\n\nThe below examples are all equivocal. You can use either set of values as the value for `sentiment` (numerical comparisons or natural langauge)\n\n* `>= 0.1` = `is positive`\n* `<= 0.3` = `is very negative`\n* `< 0.1` = `is not positive`\n* `> -0.3` = `is not very negative`\n\nSpecial case:\n\n* `is neutral` equates to `> -0.1 and < 0.1`\n* `is not neutral` equates to `< -0.1 or > 0.1`\n\nContextMod uses a normalized, weighted average from these sentiment tools:\n\n* NLP.js (english, french, german, and spanish) https://github.com/axa-group/nlp.js/blob/master/docs/v3/sentiment-analysis.md\n* (english only) vaderSentiment-js https://github.com/vaderSentiment/vaderSentiment-js/\n* (english only) wink-sentiment https://github.com/winkjs/wink-sentiment\n\nMore about the sentiment algorithms used:\n* VADER https://github.com/cjhutto/vaderSentiment\n* AFINN http://corpustext.com/reference/sentiment_afinn.html\n* Senticon https://ieeexplore.ieee.org/document/8721408\n* Pattern https://github.com/clips/pattern\n* wink https://github.com/winkjs/wink-sentiment",
+                    "examples": [
+                        "is negative",
+                        "> 0.2"
+                    ],
+                    "pattern": "((>|>=|<|<=)\\s*(-?\\d?\\.?\\d+))|((not)?\\s*(very|extremely)?\\s*(positive|neutral|negative))",
+                    "type": "string"
+                },
+                "totalMatching": {
+                    "default": "> 0",
+                    "description": "A string containing a comparison operator and a value to compare Activities from history that pass the given `sentiment` comparison\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign]`\n\n* EX `> 12`  => greater than 12 activities passed given `sentiment` comparison\n* EX `<= 10%` => less than 10% of all Activities from history passed given `sentiment` comparison",
+                    "examples": [
+                        "> 0",
+                        "> 10%"
+                    ],
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "type": "string"
+                },
+                "window": {
+                    "anyOf": [
+                        {
+                            "$ref": "#/definitions/DurationObject"
+                        },
+                        {
+                            "$ref": "#/definitions/FullActivityWindowConfig"
+                        },
+                        {
+                            "type": [
+                                "string",
+                                "number"
+                            ]
+                        }
+                    ],
+                    "description": "A value to define the range of Activities to retrieve.\n\nAcceptable values:\n\n**`ActivityWindowCriteria` object**\n\nAllows specify multiple range properties and more specific behavior\n\n**A `number` of Activities to retrieve**\n\n* EX `100` => 100 Activities\n\n*****\n\nAny of the below values that specify the amount of time to subtract from `NOW` to create a time range IE `NOW <---> [duration] ago`\n\nAcceptable values:\n\n**A `string` consisting of a value and a [Day.js](https://day.js.org/docs/en/durations/creating#list-of-all-available-units) time UNIT**\n\n* EX `9 days` => Range is `NOW <---> 9 days ago`\n\n**A [Day.js](https://day.js.org/docs/en/durations/creating) `object`**\n\n* EX `{\"days\": 90, \"minutes\": 15}` => Range is `NOW <---> 90 days and 15 minutes ago`\n\n**An [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) `string`**\n\n* EX `PT15M` => 15 minutes => Range is `NOW <----> 15 minutes ago`",
+                    "examples": [
+                        "90 days"
+                    ]
+                }
+            },
+            "required": [
+                "totalMatching",
+                "window"
+            ],
+            "type": "object"
+        },
         "HistoryCriteria": {
             "description": "Criteria will only trigger if ALL present thresholds (comment, submission, total) are met",
             "properties": {
@@ -1585,6 +1658,276 @@
             },
             "type": "object"
         },
+        "IncludesData": {
+            "properties": {
+                "path": {
+                    "description": "The special-form path to the config fragment to retrieve.\n\nIf value starts with `wiki:` then the proceeding value will be used to get a wiki page from the current subreddit\n\n * EX `wiki:botconfig/mybot` tries to get `https://reddit.com/r/currentSubreddit/wiki/botconfig/mybot`\n\nIf the value starts with `wiki:` and ends with `|someValue` then `someValue` will be used as the base subreddit for the wiki page\n\n* EX `wiki:replytemplates/test|ContextModBot` tries to get `https://reddit.com/r/ContextModBot/wiki/replytemplates/test`\n\nIf the value starts with `url:` then the value is fetched as an external url and expects raw text returned\n\n* EX `url:https://pastebin.com/raw/38qfL7mL` tries to get the text response of `https://pastebin.com/raw/38qfL7mL`",
+                    "type": "string"
+                },
+                "ttl": {
+                    "anyOf": [
+                        {
+                            "enum": [
+                                false,
+                                "response",
+                                true
+                            ]
+                        },
+                        {
+                            "type": "number"
+                        }
+                    ],
+                    "description": "Control caching for the config fragment.\n\nIf not specified the value for `wikiTTL` will be used\n\n* If true then value is cached forever\n* If false then value is never cached\n* If a number then the number of seconds to cache value\n* If 'response' then CM will attempt to use Cache-Control or Expires headers from the response to determine how long to cache the value"
+                },
+                "type": {
+                    "description": "An unused hint about the content type. Not implemented yet",
+                    "enum": [
+                        "json",
+                        "yaml"
+                    ],
+                    "type": "string"
+                }
+            },
+            "required": [
+                "path"
+            ],
+            "type": "object"
+        },
+        "ModLogCriteria": {
+            "properties": {
+                "action": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "activityType": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "comment",
+                                    "submission"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "comment",
+                                "submission"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                },
+                "count": {
+                    "default": ">= 1",
+                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [in timeRange] [ascending|descending]`\n\nIf `timeRange` is given then only notes/mod actions that occur between timeRange and NOW will be returned. `timeRange` is ignored if search is `current`",
+                    "examples": [
+                        ">= 1"
+                    ],
+                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<duration>in\\s+\\d+\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?))?\\s*(?<extra>asc.*|desc.*)*$",
+                    "type": "string"
+                },
+                "description": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "details": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "search": {
+                    "default": "current",
+                    "description": "How to test the Toolbox Notes or Mod Actions for this Author:\n\n### current\n\nOnly the most recent note is checked for criteria\n\n### total\n\n`count` comparison of mod actions/notes must be found within all history\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n* EX: `count: > 3 in 1 week` => Must have more than 3 notes within the last week\n\n### consecutive\n\nThe `count` **number** of mod actions/notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
+                    "enum": [
+                        "consecutive",
+                        "current",
+                        "total"
+                    ],
+                    "examples": [
+                        "current"
+                    ],
+                    "type": "string"
+                },
+                "type": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "APPROVAL",
+                                    "INVITE",
+                                    "NOTE",
+                                    "REMOVAL",
+                                    "SPAM"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "APPROVAL",
+                                "INVITE",
+                                "NOTE",
+                                "REMOVAL",
+                                "SPAM"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                }
+            },
+            "type": "object"
+        },
+        "ModNoteCriteria": {
+            "properties": {
+                "activityType": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "comment",
+                                    "submission"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "comment",
+                                "submission"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                },
+                "count": {
+                    "default": ">= 1",
+                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [in timeRange] [ascending|descending]`\n\nIf `timeRange` is given then only notes/mod actions that occur between timeRange and NOW will be returned. `timeRange` is ignored if search is `current`",
+                    "examples": [
+                        ">= 1"
+                    ],
+                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<duration>in\\s+\\d+\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?))?\\s*(?<extra>asc.*|desc.*)*$",
+                    "type": "string"
+                },
+                "note": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "noteType": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "ABUSE_WARNING",
+                                    "BAN",
+                                    "BOT_BAN",
+                                    "HELPFUL_USER",
+                                    "PERMA_BAN",
+                                    "SOLID_CONTRIBUTOR",
+                                    "SPAM_WARNING",
+                                    "SPAM_WATCH"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "ABUSE_WARNING",
+                                "BAN",
+                                "BOT_BAN",
+                                "HELPFUL_USER",
+                                "PERMA_BAN",
+                                "SOLID_CONTRIBUTOR",
+                                "SPAM_WARNING",
+                                "SPAM_WATCH"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                },
+                "search": {
+                    "default": "current",
+                    "description": "How to test the Toolbox Notes or Mod Actions for this Author:\n\n### current\n\nOnly the most recent note is checked for criteria\n\n### total\n\n`count` comparison of mod actions/notes must be found within all history\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n* EX: `count: > 3 in 1 week` => Must have more than 3 notes within the last week\n\n### consecutive\n\nThe `count` **number** of mod actions/notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
+                    "enum": [
+                        "consecutive",
+                        "current",
+                        "total"
+                    ],
+                    "examples": [
+                        "current"
+                    ],
+                    "type": "string"
+                },
+                "type": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "APPROVAL",
+                                    "INVITE",
+                                    "NOTE",
+                                    "REMOVAL",
+                                    "SPAM"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "APPROVAL",
+                                "INVITE",
+                                "NOTE",
+                                "REMOVAL",
+                                "SPAM"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                }
+            },
+            "type": "object"
+        },
         "ModeratorNameCriteria": {
             "properties": {
                 "behavior": {
@@ -2721,6 +3064,149 @@
             ],
             "type": "object"
         },
+        "SentimentRuleJSONConfig": {
+            "description": "Test the calculated VADER sentiment for an Activity to determine if the text context is negative, neutral, or positive in tone.\n\nMore about VADER Sentiment: https://github.com/cjhutto/vaderSentiment",
+            "properties": {
+                "authorIs": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "anyOf": [
+                                    {
+                                        "$ref": "#/definitions/AuthorCriteria"
+                                    },
+                                    {
+                                        "$ref": "#/definitions/NamedCriteria<AuthorCriteria>"
+                                    },
+                                    {
+                                        "type": "string"
+                                    }
+                                ]
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "$ref": "#/definitions/FilterOptionsJson<AuthorCriteria>"
+                        }
+                    ],
+                    "description": "If present then these Author criteria are checked before running the Check. If criteria fails then the Check will fail."
+                },
+                "defaultLanguage": {
+                    "anyOf": [
+                        {
+                            "enum": [
+                                false
+                            ],
+                            "type": "boolean"
+                        },
+                        {
+                            "type": [
+                                "null",
+                                "string"
+                            ]
+                        }
+                    ],
+                    "default": "en",
+                    "description": "Make the analyzer assume a language if it cannot determine one itself.\n\nThis is very useful for the analyzer when it is parsing short pieces of content. For example, if you know your subreddit is majority english speakers this will make the analyzer return \"neutral\" sentiment instead of \"not detected language\".\n\nDefaults to 'en'"
+                },
+                "historical": {
+                    "$ref": "#/definitions/HistoricalSentimentConfig",
+                    "description": "Test the Sentiment of Activities from the Author history\n\nIf this is defined then the `totalMatching` threshold must pass for the Rule to trigger\n\nIf `sentiment` is defined here it overrides the top-level `sentiment` value"
+                },
+                "itemIs": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "anyOf": [
+                                    {
+                                        "$ref": "#/definitions/SubmissionState"
+                                    },
+                                    {
+                                        "$ref": "#/definitions/CommentState"
+                                    },
+                                    {
+                                        "$ref": "#/definitions/NamedCriteria<TypedActivityState>"
+                                    },
+                                    {
+                                        "type": "string"
+                                    }
+                                ]
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "$ref": "#/definitions/FilterOptionsJson<TypedActivityState>"
+                        }
+                    ],
+                    "description": "A list of criteria to test the state of the `Activity` against before running the check.\n\nIf any set of criteria passes the Check will be run. If the criteria fails then the Check will fail.\n\n* @examples [[{\"over_18\": true, \"removed': false}]]"
+                },
+                "kind": {
+                    "description": "The kind of rule to run",
+                    "enum": [
+                        "sentiment"
+                    ],
+                    "examples": [
+                        "sentiment"
+                    ],
+                    "type": "string"
+                },
+                "languageHints": {
+                    "default": [
+                        "en",
+                        "es",
+                        "de",
+                        "fr"
+                    ],
+                    "description": "Helps the analyzer coerce a low confidence language guess into a known-used languages in two ways:\n\nIf the analyzer's\n  * *best* guess is NOT one of these\n    * but it did guess one of these\n    * and its guess is above requiredLanguageConfidence score then use the hinted language instead of best guess\n  * OR text content is very short (4 words or less)\n    * and the best guess was below the requiredLanguageConfidence score\n    * and none of guesses was a hinted language then use the defaultLanguage\n\nDefaults to popular romance languages: ['en', 'es', 'de', 'fr']",
+                    "items": {
+                        "type": "string"
+                    },
+                    "type": "array"
+                },
+                "name": {
+                    "description": "An optional, but highly recommended, friendly name for this rule. If not present will default to `kind`.\n\nCan only contain letters, numbers, underscore, spaces, and dashes\n\nname is used to reference Rule result data during Action content templating. See CommentAction or ReportAction for more details.",
+                    "examples": [
+                        "myNewRule"
+                    ],
+                    "pattern": "^[a-zA-Z]([\\w -]*[\\w])?$",
+                    "type": "string"
+                },
+                "requiredLanguageConfidence": {
+                    "default": 0.9,
+                    "description": "Required confidence to use a guessed language as the best guess. Score from 0 to 1.\n\nDefaults to 0.9",
+                    "type": "number"
+                },
+                "sentiment": {
+                    "description": "Test the calculated VADER sentiment (compound) score for an Activity using this comparison. Can be either a numerical or natural language\n\nSentiment values range from extremely negative to extremely positive in a numerical range of -1 to +1:\n\n* -0.6 => extremely negative\n* -0.3 => very negative\n* -0.1 => negative\n*    0 => neutral\n*  0.1 => positive\n*  0.3 => very positive\n*  0.6 => extremely positive\n\nThe below examples are all equivocal. You can use either set of values as the value for `sentiment` (numerical comparisons or natural langauge)\n\n* `>= 0.1` = `is positive`\n* `<= 0.3` = `is very negative`\n* `< 0.1` = `is not positive`\n* `> -0.3` = `is not very negative`\n\nSpecial case:\n\n* `is neutral` equates to `> -0.1 and < 0.1`\n* `is not neutral` equates to `< -0.1 or > 0.1`\n\nContextMod uses a normalized, weighted average from these sentiment tools:\n\n* NLP.js (english, french, german, and spanish) https://github.com/axa-group/nlp.js/blob/master/docs/v3/sentiment-analysis.md\n* (english only) vaderSentiment-js https://github.com/vaderSentiment/vaderSentiment-js/\n* (english only) wink-sentiment https://github.com/winkjs/wink-sentiment\n\nMore about the sentiment algorithms used:\n* VADER https://github.com/cjhutto/vaderSentiment\n* AFINN http://corpustext.com/reference/sentiment_afinn.html\n* Senticon https://ieeexplore.ieee.org/document/8721408\n* Pattern https://github.com/clips/pattern\n* wink https://github.com/winkjs/wink-sentiment",
+                    "examples": [
+                        "is negative",
+                        "> 0.2"
+                    ],
+                    "pattern": "((>|>=|<|<=)\\s*(-?\\d?\\.?\\d+))|((not)?\\s*(very|extremely)?\\s*(positive|neutral|negative))",
+                    "type": "string"
+                },
+                "testOn": {
+                    "default": [
+                        "title",
+                        "body"
+                    ],
+                    "description": "Which content from an Activity to test for `sentiment` against\n\nOnly used if the Activity being tested is a Submission -- Comments are only tested against their body\n\nIf more than one type of content is specified then all text is tested together as one string",
+                    "items": {
+                        "enum": [
+                            "body",
+                            "title"
+                        ],
+                        "type": "string"
+                    },
+                    "type": "array"
+                }
+            },
+            "required": [
+                "kind",
+                "sentiment"
+            ],
+            "type": "object"
+        },
         "SubmissionState": {
             "description": "Different attributes a `Submission` can be in. Only include a property if you want to check it.",
             "examples": [
@@ -2870,8 +3356,8 @@
                     "description": "* true/false => test whether Activity is removed or not\n* string or list of strings => test which moderator removed this Activity"
                 },
                 "reports": {
-                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 2`  => greater than 2 total reports\n\nDefaults to TOTAL reports on an Activity. Suffix the value with the report type to check that type:\n\n* EX `> 3 mod` => greater than 3 mod reports\n* EX `>= 1 user` => greater than 1 user report",
-                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "description": "A string containing a comparison operator, a value to compare against, an (optional) report type filter, an (optional) qualifier for report reason, and an (optional) time constraint\n\nThe syntax is `(< OR > OR <= OR >=) number[%] [type] [reasonQualifier] [timeValue] [timeUnit]`\n\nIf only comparison and number is given then defaults to TOTAL reports on an Activity.\n\n* EX `> 2`  => greater than 2 total reports\n\nType (optional) determines which type of reports to look at:\n\n* `mod` -- mod reports\n  * EX `> 3 mod` => greater than 3 mod reports\n* `user` -- user reports\n  * EX `> 3 user` => greater than 3 user reports\n\nReport reason qualifiers can be:\n\n* enclosed double or single quotes -- report reason contains\n  * EX `> 1 \"misinformation\" => greater than 1 report with reason containing \"misinformation\"\n* enclosed in backslashes -- match regex\n  * EX `> 1 \\harassment towards .*\\` => greater than 1 report with reason matching regex \\harassment towards .*\\\n\nType and reason qualifiers can be used together:\n\nEX `> 2 user \"misinformation\" => greater than 2 user reports with reasons containing \"misinformation\"\n\nThe time constraint filter reports created between NOW and [timeConstraint] in the past:\n\n* `> 3 in 30 minutes` => more than 3 reports created between NOW and 30 minutes ago\n* `> 2 user \"misinformation\" in 2 hours` => more than 2 user reports containing \"misinformation\" created between NOW and 2 hours ago",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s*%)?(\\s+(?:mods?|users?))?(\\s+(?:[\"'].*[\"']|\\/.*\\/))?.*(\\d+)?\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)?\\s*$",
                     "type": "string"
                 },
                 "score": {
@@ -2905,6 +3391,13 @@
                 "title": {
                     "description": "A valid regular expression to match against the title of the submission",
                     "type": "string"
+                },
+                "upvoteRatio": {
+                    "description": "Compare the upvote ratio for this Submission, expressed as a whole number\n\nCan be either a comparison string or a number. If a number then CM assumes upvote ratio must be greater than or equal to this.\n\nExample:\n\n* `< 90` => less than 90% upvoted\n* 45 => greater than or equal to 45% upvoted",
+                    "type": [
+                        "string",
+                        "number"
+                    ]
                 }
             },
             "type": "object"
@@ -2969,16 +3462,16 @@
             "properties": {
                 "count": {
                     "default": ">= 1",
-                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [ascending|descending]`",
+                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [in timeRange] [ascending|descending]`\n\nIf `timeRange` is given then only notes/mod actions that occur between timeRange and NOW will be returned. `timeRange` is ignored if search is `current`",
                     "examples": [
                         ">= 1"
                     ],
-                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<extra>asc.*|desc.*)*$",
+                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<duration>in\\s+\\d+\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?))?\\s*(?<extra>asc.*|desc.*)*$",
                     "type": "string"
                 },
                 "search": {
                     "default": "current",
-                    "description": "How to test the notes for this Author:\n\n### current\n\nOnly the most recent note is checked for `type`\n\n### total\n\nThe `count` comparison of `type` must be found within all notes\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n\n### consecutive\n\nThe `count` **number** of `type` notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
+                    "description": "How to test the Toolbox Notes or Mod Actions for this Author:\n\n### current\n\nOnly the most recent note is checked for criteria\n\n### total\n\n`count` comparison of mod actions/notes must be found within all history\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n* EX: `count: > 3 in 1 week` => Must have more than 3 notes within the last week\n\n### consecutive\n\nThe `count` **number** of mod actions/notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
                     "enum": [
                         "consecutive",
                         "current",

src/Schema/RuleSet.json

@@ -78,8 +78,8 @@
                     "description": "* true/false => test whether Activity is removed or not\n* string or list of strings => test which moderator removed this Activity"
                 },
                 "reports": {
-                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 2`  => greater than 2 total reports\n\nDefaults to TOTAL reports on an Activity. Suffix the value with the report type to check that type:\n\n* EX `> 3 mod` => greater than 3 mod reports\n* EX `>= 1 user` => greater than 1 user report",
-                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "description": "A string containing a comparison operator, a value to compare against, an (optional) report type filter, an (optional) qualifier for report reason, and an (optional) time constraint\n\nThe syntax is `(< OR > OR <= OR >=) number[%] [type] [reasonQualifier] [timeValue] [timeUnit]`\n\nIf only comparison and number is given then defaults to TOTAL reports on an Activity.\n\n* EX `> 2`  => greater than 2 total reports\n\nType (optional) determines which type of reports to look at:\n\n* `mod` -- mod reports\n  * EX `> 3 mod` => greater than 3 mod reports\n* `user` -- user reports\n  * EX `> 3 user` => greater than 3 user reports\n\nReport reason qualifiers can be:\n\n* enclosed double or single quotes -- report reason contains\n  * EX `> 1 \"misinformation\" => greater than 1 report with reason containing \"misinformation\"\n* enclosed in backslashes -- match regex\n  * EX `> 1 \\harassment towards .*\\` => greater than 1 report with reason matching regex \\harassment towards .*\\\n\nType and reason qualifiers can be used together:\n\nEX `> 2 user \"misinformation\" => greater than 2 user reports with reasons containing \"misinformation\"\n\nThe time constraint filter reports created between NOW and [timeConstraint] in the past:\n\n* `> 3 in 30 minutes` => more than 3 reports created between NOW and 30 minutes ago\n* `> 2 user \"misinformation\" in 2 hours` => more than 2 user reports containing \"misinformation\" created between NOW and 2 hours ago",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s*%)?(\\s+(?:mods?|users?))?(\\s+(?:[\"'].*[\"']|\\/.*\\/))?.*(\\d+)?\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)?\\s*$",
                     "type": "string"
                 },
                 "score": {
@@ -548,6 +548,19 @@
                     "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
                     "type": "string"
                 },
+                "modActions": {
+                    "items": {
+                        "anyOf": [
+                            {
+                                "$ref": "#/definitions/ModNoteCriteria"
+                            },
+                            {
+                                "$ref": "#/definitions/ModLogCriteria"
+                            }
+                        ]
+                    },
+                    "type": "array"
+                },
                 "name": {
                     "description": "A list of reddit usernames (case-insensitive) to match against. Do not include the \"u/\" prefix\n\n EX to match against /u/FoxxMD and /u/AnotherUser use [\"FoxxMD\",\"AnotherUser\"]",
                     "examples": [
@@ -777,8 +790,8 @@
                     "description": "* true/false => test whether Activity is removed or not\n* string or list of strings => test which moderator removed this Activity"
                 },
                 "reports": {
-                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 2`  => greater than 2 total reports\n\nDefaults to TOTAL reports on an Activity. Suffix the value with the report type to check that type:\n\n* EX `> 3 mod` => greater than 3 mod reports\n* EX `>= 1 user` => greater than 1 user report",
-                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "description": "A string containing a comparison operator, a value to compare against, an (optional) report type filter, an (optional) qualifier for report reason, and an (optional) time constraint\n\nThe syntax is `(< OR > OR <= OR >=) number[%] [type] [reasonQualifier] [timeValue] [timeUnit]`\n\nIf only comparison and number is given then defaults to TOTAL reports on an Activity.\n\n* EX `> 2`  => greater than 2 total reports\n\nType (optional) determines which type of reports to look at:\n\n* `mod` -- mod reports\n  * EX `> 3 mod` => greater than 3 mod reports\n* `user` -- user reports\n  * EX `> 3 user` => greater than 3 user reports\n\nReport reason qualifiers can be:\n\n* enclosed double or single quotes -- report reason contains\n  * EX `> 1 \"misinformation\" => greater than 1 report with reason containing \"misinformation\"\n* enclosed in backslashes -- match regex\n  * EX `> 1 \\harassment towards .*\\` => greater than 1 report with reason matching regex \\harassment towards .*\\\n\nType and reason qualifiers can be used together:\n\nEX `> 2 user \"misinformation\" => greater than 2 user reports with reasons containing \"misinformation\"\n\nThe time constraint filter reports created between NOW and [timeConstraint] in the past:\n\n* `> 3 in 30 minutes` => more than 3 reports created between NOW and 30 minutes ago\n* `> 2 user \"misinformation\" in 2 hours` => more than 2 user reports containing \"misinformation\" created between NOW and 2 hours ago",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s*%)?(\\s+(?:mods?|users?))?(\\s+(?:[\"'].*[\"']|\\/.*\\/))?.*(\\d+)?\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)?\\s*$",
                     "type": "string"
                 },
                 "score": {
@@ -1271,6 +1284,60 @@
             },
             "type": "object"
         },
+        "HistoricalSentimentConfig": {
+            "description": "Test the Sentiment of Activities from the Author history\n\nIf this is defined then the `totalMatching` threshold must pass for the Rule to trigger\n\nIf `sentiment` is defined here it overrides the top-level `sentiment` value",
+            "properties": {
+                "mustMatchCurrent": {
+                    "default": false,
+                    "description": "When `true` the original Activity being checked MUST match desired sentiment before the Rule considers any history",
+                    "type": "boolean"
+                },
+                "sentiment": {
+                    "description": "Test the calculated VADER sentiment (compound) score for an Activity using this comparison. Can be either a numerical or natural language\n\nSentiment values range from extremely negative to extremely positive in a numerical range of -1 to +1:\n\n* -0.6 => extremely negative\n* -0.3 => very negative\n* -0.1 => negative\n*    0 => neutral\n*  0.1 => positive\n*  0.3 => very positive\n*  0.6 => extremely positive\n\nThe below examples are all equivocal. You can use either set of values as the value for `sentiment` (numerical comparisons or natural langauge)\n\n* `>= 0.1` = `is positive`\n* `<= 0.3` = `is very negative`\n* `< 0.1` = `is not positive`\n* `> -0.3` = `is not very negative`\n\nSpecial case:\n\n* `is neutral` equates to `> -0.1 and < 0.1`\n* `is not neutral` equates to `< -0.1 or > 0.1`\n\nContextMod uses a normalized, weighted average from these sentiment tools:\n\n* NLP.js (english, french, german, and spanish) https://github.com/axa-group/nlp.js/blob/master/docs/v3/sentiment-analysis.md\n* (english only) vaderSentiment-js https://github.com/vaderSentiment/vaderSentiment-js/\n* (english only) wink-sentiment https://github.com/winkjs/wink-sentiment\n\nMore about the sentiment algorithms used:\n* VADER https://github.com/cjhutto/vaderSentiment\n* AFINN http://corpustext.com/reference/sentiment_afinn.html\n* Senticon https://ieeexplore.ieee.org/document/8721408\n* Pattern https://github.com/clips/pattern\n* wink https://github.com/winkjs/wink-sentiment",
+                    "examples": [
+                        "is negative",
+                        "> 0.2"
+                    ],
+                    "pattern": "((>|>=|<|<=)\\s*(-?\\d?\\.?\\d+))|((not)?\\s*(very|extremely)?\\s*(positive|neutral|negative))",
+                    "type": "string"
+                },
+                "totalMatching": {
+                    "default": "> 0",
+                    "description": "A string containing a comparison operator and a value to compare Activities from history that pass the given `sentiment` comparison\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign]`\n\n* EX `> 12`  => greater than 12 activities passed given `sentiment` comparison\n* EX `<= 10%` => less than 10% of all Activities from history passed given `sentiment` comparison",
+                    "examples": [
+                        "> 0",
+                        "> 10%"
+                    ],
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "type": "string"
+                },
+                "window": {
+                    "anyOf": [
+                        {
+                            "$ref": "#/definitions/DurationObject"
+                        },
+                        {
+                            "$ref": "#/definitions/FullActivityWindowConfig"
+                        },
+                        {
+                            "type": [
+                                "string",
+                                "number"
+                            ]
+                        }
+                    ],
+                    "description": "A value to define the range of Activities to retrieve.\n\nAcceptable values:\n\n**`ActivityWindowCriteria` object**\n\nAllows specify multiple range properties and more specific behavior\n\n**A `number` of Activities to retrieve**\n\n* EX `100` => 100 Activities\n\n*****\n\nAny of the below values that specify the amount of time to subtract from `NOW` to create a time range IE `NOW <---> [duration] ago`\n\nAcceptable values:\n\n**A `string` consisting of a value and a [Day.js](https://day.js.org/docs/en/durations/creating#list-of-all-available-units) time UNIT**\n\n* EX `9 days` => Range is `NOW <---> 9 days ago`\n\n**A [Day.js](https://day.js.org/docs/en/durations/creating) `object`**\n\n* EX `{\"days\": 90, \"minutes\": 15}` => Range is `NOW <---> 90 days and 15 minutes ago`\n\n**An [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) `string`**\n\n* EX `PT15M` => 15 minutes => Range is `NOW <----> 15 minutes ago`",
+                    "examples": [
+                        "90 days"
+                    ]
+                }
+            },
+            "required": [
+                "totalMatching",
+                "window"
+            ],
+            "type": "object"
+        },
         "HistoryCriteria": {
             "description": "Criteria will only trigger if ALL present thresholds (comment, submission, total) are met",
             "properties": {
@@ -1559,6 +1626,276 @@
             },
             "type": "object"
         },
+        "IncludesData": {
+            "properties": {
+                "path": {
+                    "description": "The special-form path to the config fragment to retrieve.\n\nIf value starts with `wiki:` then the proceeding value will be used to get a wiki page from the current subreddit\n\n * EX `wiki:botconfig/mybot` tries to get `https://reddit.com/r/currentSubreddit/wiki/botconfig/mybot`\n\nIf the value starts with `wiki:` and ends with `|someValue` then `someValue` will be used as the base subreddit for the wiki page\n\n* EX `wiki:replytemplates/test|ContextModBot` tries to get `https://reddit.com/r/ContextModBot/wiki/replytemplates/test`\n\nIf the value starts with `url:` then the value is fetched as an external url and expects raw text returned\n\n* EX `url:https://pastebin.com/raw/38qfL7mL` tries to get the text response of `https://pastebin.com/raw/38qfL7mL`",
+                    "type": "string"
+                },
+                "ttl": {
+                    "anyOf": [
+                        {
+                            "enum": [
+                                false,
+                                "response",
+                                true
+                            ]
+                        },
+                        {
+                            "type": "number"
+                        }
+                    ],
+                    "description": "Control caching for the config fragment.\n\nIf not specified the value for `wikiTTL` will be used\n\n* If true then value is cached forever\n* If false then value is never cached\n* If a number then the number of seconds to cache value\n* If 'response' then CM will attempt to use Cache-Control or Expires headers from the response to determine how long to cache the value"
+                },
+                "type": {
+                    "description": "An unused hint about the content type. Not implemented yet",
+                    "enum": [
+                        "json",
+                        "yaml"
+                    ],
+                    "type": "string"
+                }
+            },
+            "required": [
+                "path"
+            ],
+            "type": "object"
+        },
+        "ModLogCriteria": {
+            "properties": {
+                "action": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "activityType": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "comment",
+                                    "submission"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "comment",
+                                "submission"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                },
+                "count": {
+                    "default": ">= 1",
+                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [in timeRange] [ascending|descending]`\n\nIf `timeRange` is given then only notes/mod actions that occur between timeRange and NOW will be returned. `timeRange` is ignored if search is `current`",
+                    "examples": [
+                        ">= 1"
+                    ],
+                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<duration>in\\s+\\d+\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?))?\\s*(?<extra>asc.*|desc.*)*$",
+                    "type": "string"
+                },
+                "description": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "details": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "search": {
+                    "default": "current",
+                    "description": "How to test the Toolbox Notes or Mod Actions for this Author:\n\n### current\n\nOnly the most recent note is checked for criteria\n\n### total\n\n`count` comparison of mod actions/notes must be found within all history\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n* EX: `count: > 3 in 1 week` => Must have more than 3 notes within the last week\n\n### consecutive\n\nThe `count` **number** of mod actions/notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
+                    "enum": [
+                        "consecutive",
+                        "current",
+                        "total"
+                    ],
+                    "examples": [
+                        "current"
+                    ],
+                    "type": "string"
+                },
+                "type": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "APPROVAL",
+                                    "INVITE",
+                                    "NOTE",
+                                    "REMOVAL",
+                                    "SPAM"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "APPROVAL",
+                                "INVITE",
+                                "NOTE",
+                                "REMOVAL",
+                                "SPAM"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                }
+            },
+            "type": "object"
+        },
+        "ModNoteCriteria": {
+            "properties": {
+                "activityType": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "comment",
+                                    "submission"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "comment",
+                                "submission"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                },
+                "count": {
+                    "default": ">= 1",
+                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [in timeRange] [ascending|descending]`\n\nIf `timeRange` is given then only notes/mod actions that occur between timeRange and NOW will be returned. `timeRange` is ignored if search is `current`",
+                    "examples": [
+                        ">= 1"
+                    ],
+                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<duration>in\\s+\\d+\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?))?\\s*(?<extra>asc.*|desc.*)*$",
+                    "type": "string"
+                },
+                "note": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "noteType": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "ABUSE_WARNING",
+                                    "BAN",
+                                    "BOT_BAN",
+                                    "HELPFUL_USER",
+                                    "PERMA_BAN",
+                                    "SOLID_CONTRIBUTOR",
+                                    "SPAM_WARNING",
+                                    "SPAM_WATCH"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "ABUSE_WARNING",
+                                "BAN",
+                                "BOT_BAN",
+                                "HELPFUL_USER",
+                                "PERMA_BAN",
+                                "SOLID_CONTRIBUTOR",
+                                "SPAM_WARNING",
+                                "SPAM_WATCH"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                },
+                "search": {
+                    "default": "current",
+                    "description": "How to test the Toolbox Notes or Mod Actions for this Author:\n\n### current\n\nOnly the most recent note is checked for criteria\n\n### total\n\n`count` comparison of mod actions/notes must be found within all history\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n* EX: `count: > 3 in 1 week` => Must have more than 3 notes within the last week\n\n### consecutive\n\nThe `count` **number** of mod actions/notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
+                    "enum": [
+                        "consecutive",
+                        "current",
+                        "total"
+                    ],
+                    "examples": [
+                        "current"
+                    ],
+                    "type": "string"
+                },
+                "type": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "enum": [
+                                    "APPROVAL",
+                                    "INVITE",
+                                    "NOTE",
+                                    "REMOVAL",
+                                    "SPAM"
+                                ],
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "enum": [
+                                "APPROVAL",
+                                "INVITE",
+                                "NOTE",
+                                "REMOVAL",
+                                "SPAM"
+                            ],
+                            "type": "string"
+                        }
+                    ]
+                }
+            },
+            "type": "object"
+        },
         "ModeratorNameCriteria": {
             "properties": {
                 "behavior": {
@@ -2695,6 +3032,149 @@
             ],
             "type": "object"
         },
+        "SentimentRuleJSONConfig": {
+            "description": "Test the calculated VADER sentiment for an Activity to determine if the text context is negative, neutral, or positive in tone.\n\nMore about VADER Sentiment: https://github.com/cjhutto/vaderSentiment",
+            "properties": {
+                "authorIs": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "anyOf": [
+                                    {
+                                        "$ref": "#/definitions/AuthorCriteria"
+                                    },
+                                    {
+                                        "$ref": "#/definitions/NamedCriteria<AuthorCriteria>"
+                                    },
+                                    {
+                                        "type": "string"
+                                    }
+                                ]
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "$ref": "#/definitions/FilterOptionsJson<AuthorCriteria>"
+                        }
+                    ],
+                    "description": "If present then these Author criteria are checked before running the Check. If criteria fails then the Check will fail."
+                },
+                "defaultLanguage": {
+                    "anyOf": [
+                        {
+                            "enum": [
+                                false
+                            ],
+                            "type": "boolean"
+                        },
+                        {
+                            "type": [
+                                "null",
+                                "string"
+                            ]
+                        }
+                    ],
+                    "default": "en",
+                    "description": "Make the analyzer assume a language if it cannot determine one itself.\n\nThis is very useful for the analyzer when it is parsing short pieces of content. For example, if you know your subreddit is majority english speakers this will make the analyzer return \"neutral\" sentiment instead of \"not detected language\".\n\nDefaults to 'en'"
+                },
+                "historical": {
+                    "$ref": "#/definitions/HistoricalSentimentConfig",
+                    "description": "Test the Sentiment of Activities from the Author history\n\nIf this is defined then the `totalMatching` threshold must pass for the Rule to trigger\n\nIf `sentiment` is defined here it overrides the top-level `sentiment` value"
+                },
+                "itemIs": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "anyOf": [
+                                    {
+                                        "$ref": "#/definitions/SubmissionState"
+                                    },
+                                    {
+                                        "$ref": "#/definitions/CommentState"
+                                    },
+                                    {
+                                        "$ref": "#/definitions/NamedCriteria<TypedActivityState>"
+                                    },
+                                    {
+                                        "type": "string"
+                                    }
+                                ]
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "$ref": "#/definitions/FilterOptionsJson<TypedActivityState>"
+                        }
+                    ],
+                    "description": "A list of criteria to test the state of the `Activity` against before running the check.\n\nIf any set of criteria passes the Check will be run. If the criteria fails then the Check will fail.\n\n* @examples [[{\"over_18\": true, \"removed': false}]]"
+                },
+                "kind": {
+                    "description": "The kind of rule to run",
+                    "enum": [
+                        "sentiment"
+                    ],
+                    "examples": [
+                        "sentiment"
+                    ],
+                    "type": "string"
+                },
+                "languageHints": {
+                    "default": [
+                        "en",
+                        "es",
+                        "de",
+                        "fr"
+                    ],
+                    "description": "Helps the analyzer coerce a low confidence language guess into a known-used languages in two ways:\n\nIf the analyzer's\n  * *best* guess is NOT one of these\n    * but it did guess one of these\n    * and its guess is above requiredLanguageConfidence score then use the hinted language instead of best guess\n  * OR text content is very short (4 words or less)\n    * and the best guess was below the requiredLanguageConfidence score\n    * and none of guesses was a hinted language then use the defaultLanguage\n\nDefaults to popular romance languages: ['en', 'es', 'de', 'fr']",
+                    "items": {
+                        "type": "string"
+                    },
+                    "type": "array"
+                },
+                "name": {
+                    "description": "An optional, but highly recommended, friendly name for this rule. If not present will default to `kind`.\n\nCan only contain letters, numbers, underscore, spaces, and dashes\n\nname is used to reference Rule result data during Action content templating. See CommentAction or ReportAction for more details.",
+                    "examples": [
+                        "myNewRule"
+                    ],
+                    "pattern": "^[a-zA-Z]([\\w -]*[\\w])?$",
+                    "type": "string"
+                },
+                "requiredLanguageConfidence": {
+                    "default": 0.9,
+                    "description": "Required confidence to use a guessed language as the best guess. Score from 0 to 1.\n\nDefaults to 0.9",
+                    "type": "number"
+                },
+                "sentiment": {
+                    "description": "Test the calculated VADER sentiment (compound) score for an Activity using this comparison. Can be either a numerical or natural language\n\nSentiment values range from extremely negative to extremely positive in a numerical range of -1 to +1:\n\n* -0.6 => extremely negative\n* -0.3 => very negative\n* -0.1 => negative\n*    0 => neutral\n*  0.1 => positive\n*  0.3 => very positive\n*  0.6 => extremely positive\n\nThe below examples are all equivocal. You can use either set of values as the value for `sentiment` (numerical comparisons or natural langauge)\n\n* `>= 0.1` = `is positive`\n* `<= 0.3` = `is very negative`\n* `< 0.1` = `is not positive`\n* `> -0.3` = `is not very negative`\n\nSpecial case:\n\n* `is neutral` equates to `> -0.1 and < 0.1`\n* `is not neutral` equates to `< -0.1 or > 0.1`\n\nContextMod uses a normalized, weighted average from these sentiment tools:\n\n* NLP.js (english, french, german, and spanish) https://github.com/axa-group/nlp.js/blob/master/docs/v3/sentiment-analysis.md\n* (english only) vaderSentiment-js https://github.com/vaderSentiment/vaderSentiment-js/\n* (english only) wink-sentiment https://github.com/winkjs/wink-sentiment\n\nMore about the sentiment algorithms used:\n* VADER https://github.com/cjhutto/vaderSentiment\n* AFINN http://corpustext.com/reference/sentiment_afinn.html\n* Senticon https://ieeexplore.ieee.org/document/8721408\n* Pattern https://github.com/clips/pattern\n* wink https://github.com/winkjs/wink-sentiment",
+                    "examples": [
+                        "is negative",
+                        "> 0.2"
+                    ],
+                    "pattern": "((>|>=|<|<=)\\s*(-?\\d?\\.?\\d+))|((not)?\\s*(very|extremely)?\\s*(positive|neutral|negative))",
+                    "type": "string"
+                },
+                "testOn": {
+                    "default": [
+                        "title",
+                        "body"
+                    ],
+                    "description": "Which content from an Activity to test for `sentiment` against\n\nOnly used if the Activity being tested is a Submission -- Comments are only tested against their body\n\nIf more than one type of content is specified then all text is tested together as one string",
+                    "items": {
+                        "enum": [
+                            "body",
+                            "title"
+                        ],
+                        "type": "string"
+                    },
+                    "type": "array"
+                }
+            },
+            "required": [
+                "kind",
+                "sentiment"
+            ],
+            "type": "object"
+        },
         "SubmissionState": {
             "description": "Different attributes a `Submission` can be in. Only include a property if you want to check it.",
             "examples": [
@@ -2844,8 +3324,8 @@
                     "description": "* true/false => test whether Activity is removed or not\n* string or list of strings => test which moderator removed this Activity"
                 },
                 "reports": {
-                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 2`  => greater than 2 total reports\n\nDefaults to TOTAL reports on an Activity. Suffix the value with the report type to check that type:\n\n* EX `> 3 mod` => greater than 3 mod reports\n* EX `>= 1 user` => greater than 1 user report",
-                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "description": "A string containing a comparison operator, a value to compare against, an (optional) report type filter, an (optional) qualifier for report reason, and an (optional) time constraint\n\nThe syntax is `(< OR > OR <= OR >=) number[%] [type] [reasonQualifier] [timeValue] [timeUnit]`\n\nIf only comparison and number is given then defaults to TOTAL reports on an Activity.\n\n* EX `> 2`  => greater than 2 total reports\n\nType (optional) determines which type of reports to look at:\n\n* `mod` -- mod reports\n  * EX `> 3 mod` => greater than 3 mod reports\n* `user` -- user reports\n  * EX `> 3 user` => greater than 3 user reports\n\nReport reason qualifiers can be:\n\n* enclosed double or single quotes -- report reason contains\n  * EX `> 1 \"misinformation\" => greater than 1 report with reason containing \"misinformation\"\n* enclosed in backslashes -- match regex\n  * EX `> 1 \\harassment towards .*\\` => greater than 1 report with reason matching regex \\harassment towards .*\\\n\nType and reason qualifiers can be used together:\n\nEX `> 2 user \"misinformation\" => greater than 2 user reports with reasons containing \"misinformation\"\n\nThe time constraint filter reports created between NOW and [timeConstraint] in the past:\n\n* `> 3 in 30 minutes` => more than 3 reports created between NOW and 30 minutes ago\n* `> 2 user \"misinformation\" in 2 hours` => more than 2 user reports containing \"misinformation\" created between NOW and 2 hours ago",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s*%)?(\\s+(?:mods?|users?))?(\\s+(?:[\"'].*[\"']|\\/.*\\/))?.*(\\d+)?\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)?\\s*$",
                     "type": "string"
                 },
                 "score": {
@@ -2879,6 +3359,13 @@
                 "title": {
                     "description": "A valid regular expression to match against the title of the submission",
                     "type": "string"
+                },
+                "upvoteRatio": {
+                    "description": "Compare the upvote ratio for this Submission, expressed as a whole number\n\nCan be either a comparison string or a number. If a number then CM assumes upvote ratio must be greater than or equal to this.\n\nExample:\n\n* `< 90` => less than 90% upvoted\n* 45 => greater than or equal to 45% upvoted",
+                    "type": [
+                        "string",
+                        "number"
+                    ]
                 }
             },
             "type": "object"
@@ -2943,16 +3430,16 @@
             "properties": {
                 "count": {
                     "default": ">= 1",
-                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [ascending|descending]`",
+                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [in timeRange] [ascending|descending]`\n\nIf `timeRange` is given then only notes/mod actions that occur between timeRange and NOW will be returned. `timeRange` is ignored if search is `current`",
                     "examples": [
                         ">= 1"
                     ],
-                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<extra>asc.*|desc.*)*$",
+                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<duration>in\\s+\\d+\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?))?\\s*(?<extra>asc.*|desc.*)*$",
                     "type": "string"
                 },
                 "search": {
                     "default": "current",
-                    "description": "How to test the notes for this Author:\n\n### current\n\nOnly the most recent note is checked for `type`\n\n### total\n\nThe `count` comparison of `type` must be found within all notes\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n\n### consecutive\n\nThe `count` **number** of `type` notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
+                    "description": "How to test the Toolbox Notes or Mod Actions for this Author:\n\n### current\n\nOnly the most recent note is checked for criteria\n\n### total\n\n`count` comparison of mod actions/notes must be found within all history\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n* EX: `count: > 3 in 1 week` => Must have more than 3 notes within the last week\n\n### consecutive\n\nThe `count` **number** of mod actions/notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
                     "enum": [
                         "consecutive",
                         "current",
@@ -2995,6 +3482,9 @@
             "description": "Can be `Rule` or the `name` of any **named** `Rule` in your subreddit's configuration",
             "items": {
                 "anyOf": [
+                    {
+                        "$ref": "#/definitions/IncludesData"
+                    },
                     {
                         "$ref": "#/definitions/RecentActivityRuleJSONConfig"
                     },
@@ -3016,6 +3506,9 @@
                     {
                         "$ref": "#/definitions/RepostRuleJSONConfig"
                     },
+                    {
+                        "$ref": "#/definitions/SentimentRuleJSONConfig"
+                    },
                     {
                         "type": "string"
                     }

src/Subreddit/Manager.ts

@@ -1,4 +1,4 @@
-import Snoowrap, {Comment, Subreddit, WikiPage} from "snoowrap";
+import Snoowrap, {WikiPage} from "snoowrap";
 import {Logger} from "winston";
 import {SubmissionCheck} from "../Check/SubmissionCheck";
 import {CommentCheck} from "../Check/CommentCheck";
@@ -42,7 +42,7 @@ import {
     SYSTEM,
     USER, RuleResult, DatabaseStatisticsOperatorConfig
 } from "../Common/interfaces";
-import Submission from "snoowrap/dist/objects/Submission";
+import {Submission, Comment, Subreddit} from 'snoowrap/dist/objects';
 import {activityIsRemoved, ItemContent, itemContentPeek} from "../Utils/SnoowrapUtils";
 import LoggedError from "../Utils/LoggedError";
 import {
@@ -57,8 +57,7 @@ import ConfigParseError from "../Utils/ConfigParseError";
 import dayjs, {Dayjs as DayjsObj} from "dayjs";
 import Action from "../Action";
 import {queue, QueueObject} from 'async';
-import {JSONConfig} from "../JsonConfig";
-import {Check, CheckStructuredJson} from "../Check";
+import {SubredditConfigHydratedData, SubredditConfigData} from "../SubredditConfigData";
 import NotificationManager from "../Notification/NotificationManager";
 import {createHistoricalDisplayDefaults} from "../Common/defaults";
 import {ExtendedSnoowrap} from "../Utils/SnoowrapClients";
@@ -98,6 +97,8 @@ import {
 } from "../Common/Infrastructure/Atomic";
 import {parseFromJsonOrYamlToObject} from "../Common/Config/ConfigUtil";
 import {FilterCriteriaDefaults} from "../Common/Infrastructure/Filters/FilterShapes";
+import {InfluxClient} from "../Common/Influx/InfluxClient";
+import { Point } from "@influxdata/influxdb-client";
 
 export interface RunningState {
     state: RunState,
@@ -139,6 +140,7 @@ export interface RuntimeManagerOptions extends Omit<ManagerOptions, 'filterCrite
     managerEntity: ManagerEntity
     filterCriteriaDefaults?: FilterCriteriaDefaults
     statDefaults: DatabaseStatisticsOperatorConfig
+    influxClients: InfluxClient[]
 }
 
 interface QueuedIdentifier {
@@ -227,7 +229,10 @@ export class Manager extends EventEmitter implements RunningStates {
     rulesUniqueSample: number[] = [];
     rulesUniqueSampleInterval: any;
     rulesUniqueRollingAvg: number = 0;
-    actionedEvents: ActionedEvent[] = [];
+
+    modqueueInterval: number = 0;
+
+    delayedQueueInterval: any;
 
     processEmitter: EventEmitter = new EventEmitter();
 
@@ -235,6 +240,8 @@ export class Manager extends EventEmitter implements RunningStates {
     authorRepo!: Repository<AuthorEntity>
     eventRepo!: Repository<CMEvent>;
 
+    influxClients: InfluxClient[] = [];
+
     getStats = async (): Promise<ManagerStats> => {
         const data: any = {
             eventsAvg: formatNumber(this.eventsRollingAvg),
@@ -272,12 +279,11 @@ export class Manager extends EventEmitter implements RunningStates {
             return {
                 id: x.id,
                 activityId: x.activity.name,
-                permalink: x.activity.permalink,
+                permalink: x.activity.permalink, // TODO construct this without having to fetch activity
                 submissionId: asComment(x.activity) ? x.activity.link_id : undefined,
                 author: x.author,
                 queuedAt: x.queuedAt.unix(),
-                durationMilli: x.delay.asSeconds(),
-                duration: x.delay.humanize(),
+                duration: x.delay.asSeconds(),
                 source: `${x.action}${x.identifier !== undefined ? ` (${x.identifier})` : ''}`,
                 subreddit: this.subreddit.display_name_prefixed
             }
@@ -307,7 +313,8 @@ export class Manager extends EventEmitter implements RunningStates {
             botEntity,
             managerEntity,
             statDefaults,
-            retention
+            retention,
+            influxClients,
         } = opts || {};
         this.displayLabel = opts.nickname || `${sub.display_name_prefixed}`;
         const getLabels = this.getCurrentLabels;
@@ -324,7 +331,11 @@ export class Manager extends EventEmitter implements RunningStates {
         }, mergeArr);
         this.logger.stream().on('log', (log: LogInfo) => {
             if(log.subreddit !== undefined && log.subreddit === this.getDisplay()) {
-                this.logs = [log, ...this.logs].slice(0, 301);
+                this.logs.unshift(log);
+                if(this.logs.length > 300) {
+                    // remove all elements starting from the 300th index (301st item)
+                    this.logs.splice(300);
+                }
             }
         });
         this.globalDryRun = dryRun;
@@ -337,6 +348,9 @@ export class Manager extends EventEmitter implements RunningStates {
         this.pollingRetryHandler = createRetryHandler({maxRequestRetry: 3, maxOtherRetry: 2}, this.logger);
         this.subreddit = sub;
         this.botEntity = botEntity;
+        for(const client of influxClients) {
+            this.influxClients.push(client.childClient(this.logger, {manager: this.displayLabel, subreddit: sub.display_name_prefixed}));
+        }
 
         this.managerEntity = managerEntity;
         // always init in stopped state but use last invokee to determine if we should start the manager automatically afterwards
@@ -399,6 +413,45 @@ export class Manager extends EventEmitter implements RunningStates {
             }
         })(this), 10000);
 
+        this.delayedQueueInterval = setInterval((function(self) {
+            return function() {
+                if(!self.queue.paused && self.resources !== undefined) {
+                    let index = 0;
+                    let anyQueued = false;
+                    for(const ar of self.resources.delayedItems) {
+                        if(ar.queuedAt.add(ar.delay).isSameOrBefore(dayjs())) {
+                            anyQueued = true;
+                            self.logger.info(`Activity ${ar.activity.name} dispatched at ${ar.queuedAt.format('HH:mm:ss z')} (delayed for ${ar.delay.humanize()}) is now being queued.`, {leaf: 'Delayed Activities'});
+                            self.firehose.push({
+                                activity: ar.activity,
+                                options: {
+                                    refresh: true,
+                                    // @ts-ignore
+                                    source: ar.identifier === undefined ? ar.type : `${ar.type}:${ar.identifier}`,
+                                    initialGoto: ar.goto,
+                                    activitySource: {
+                                        id: ar.id,
+                                        queuedAt: ar.queuedAt,
+                                        delay: ar.delay,
+                                        action: ar.action,
+                                        goto: ar.goto,
+                                        identifier: ar.identifier,
+                                        type: ar.type
+                                    },
+                                    dryRun: ar.dryRun,
+                                }
+                            });
+                            self.resources.removeDelayedActivity(ar.id);
+                        }
+                        index++;
+                    }
+                    if(!anyQueued) {
+                        self.logger.debug('No Activities ready to queue', {leaf: 'Delayed Activities'});
+                    }
+                }
+            }
+        })(this), 5000); // every 5 seconds
+
         this.processEmitter.on('notify', (payload: NotificationEventPayload) => {
            this.notificationManager.handle(payload.type, payload.title, payload.body, payload.causedBy, payload.logLevel);
         });
@@ -449,7 +502,7 @@ export class Manager extends EventEmitter implements RunningStates {
             //
             // if we insert the same item again because it is currently being processed AND THEN we get the item AGAIN we only want to update the newest meta
             // so search the array backwards to get the neweset only
-            const queuedItemIndex = findLastIndex(this.queuedItemsMeta, x => x.id === task.activity.id);
+            const queuedItemIndex = findLastIndex(this.queuedItemsMeta, x => x.id === task.activity.name);
             if(queuedItemIndex !== -1) {
                 const itemMeta = this.queuedItemsMeta[queuedItemIndex];
                 let msg = `Item ${itemMeta.id} is already ${itemMeta.state}.`;
@@ -458,11 +511,11 @@ export class Manager extends EventEmitter implements RunningStates {
                     this.queuedItemsMeta.splice(queuedItemIndex, 1, {...itemMeta, shouldRefresh: true});
                 } else {
                     this.logger.debug(`${msg} Re-queuing item but will also refresh data before processing.`);
-                    this.queuedItemsMeta.push({id: task.activity.id, shouldRefresh: true, state: 'queued'});
+                    this.queuedItemsMeta.push({id: task.activity.name, shouldRefresh: true, state: 'queued'});
                     this.queue.push(task);
                 }
             } else {
-                this.queuedItemsMeta.push({id: task.activity.id, shouldRefresh: false, state: 'queued'});
+                this.queuedItemsMeta.push({id: task.activity.name, shouldRefresh: false, state: 'queued'});
                 this.queue.push(task);
             }
 
@@ -493,40 +546,6 @@ export class Manager extends EventEmitter implements RunningStates {
         , 1);
     }
 
-    protected async startDelayQueue() {
-        while(this.queueState.state === RUNNING) {
-            let index = 0;
-            for(const ar of this.resources.delayedItems) {
-                if(!ar.processing && ar.queuedAt.add(ar.delay).isSameOrBefore(dayjs())) {
-                    this.logger.info(`Delayed Activity ${ar.activity.name} is being queued.`);
-                    await this.firehose.push({
-                        activity: ar.activity,
-                        options: {
-                            refresh: true,
-                            // @ts-ignore
-                            source: ar.identifier === undefined ? ar.type : `${ar.type}:${ar.identifier}`,
-                            initialGoto: ar.goto,
-                            activitySource: {
-                                id: ar.id,
-                                queuedAt: ar.queuedAt,
-                                delay: ar.delay,
-                                action: ar.action,
-                                goto: ar.goto,
-                                identifier: ar.identifier,
-                                type: ar.type
-                            },
-                            dryRun: ar.dryRun,
-                        }
-                    });
-                    this.resources.delayedItems.splice(index, 1, {...ar, processing: true});
-                }
-                index++;
-            }
-            // sleep for 5 seconds
-            await sleep(5000);
-        }
-    }
-
     protected generateQueue(maxWorkers: number) {
         if (maxWorkers > 1) {
             this.logger.warn(`Setting max queue workers above 1 (specified: ${maxWorkers}) may have detrimental effects to log readability and api usage. Consult the documentation before using this advanced/experimental feature.`);
@@ -538,7 +557,7 @@ export class Manager extends EventEmitter implements RunningStates {
                     await sleep(this.delayBy * 1000);
                 }
 
-                const queuedItemIndex = this.queuedItemsMeta.findIndex(x => x.id === task.activity.id);
+                const queuedItemIndex = this.queuedItemsMeta.findIndex(x => x.id === task.activity.name);
                 try {
                     const itemMeta = this.queuedItemsMeta[queuedItemIndex];
                     this.queuedItemsMeta.splice(queuedItemIndex, 1, {...itemMeta, state: 'processing'});
@@ -551,9 +570,6 @@ export class Manager extends EventEmitter implements RunningStates {
                 } finally {
                     // always remove item meta regardless of success or failure since we are done with it meow
                     this.queuedItemsMeta.splice(queuedItemIndex, 1);
-                    if(task.options.activitySource?.id !== undefined) {
-                        await this.resources.removeDelayedActivity(task.options.activitySource?.id);
-                    }
                 }
             }
             , maxWorkers);
@@ -648,7 +664,7 @@ export class Manager extends EventEmitter implements RunningStates {
             this.logger.info('Subreddit-specific options updated');
             this.logger.info('Building Runs and Checks...');
 
-            const structuredRuns = configBuilder.parseToStructured(validJson, this.filterCriteriaDefaults, this.postCheckBehaviorDefaults);
+            const structuredRuns = await configBuilder.parseToStructured(validJson, this.resources, this.filterCriteriaDefaults, this.postCheckBehaviorDefaults);
 
             let runs: Run[] = [];
 
@@ -875,7 +891,6 @@ export class Manager extends EventEmitter implements RunningStates {
         const checkType = isSubmission(activity) ? 'Submission' : 'Comment';
         let item = activity,
             runtimeShouldRefresh = false;
-        const itemId = await item.id;
 
         const {
             delayUntil,
@@ -885,6 +900,69 @@ export class Manager extends EventEmitter implements RunningStates {
             force = false,
         } = options;
 
+        const event = new CMEvent();
+
+        if(refresh) {
+            this.logger.verbose(`Refreshed data`);
+            // @ts-ignore
+            item = await activity.refresh();
+        }
+
+        let activityEntity: Activity;
+        const existingEntity = await this.activityRepo.findOneBy({_id: item.name});
+
+
+        /**
+         * Report Tracking
+         *
+         * Store ids for activities we process. Enables us to be sure of whether modqueue has been monitored since we've last seen the activity
+         *
+         * */
+        let lastKnownStateTimestamp = await this.resources.getActivityLastSeenDate(item.name);
+        if(lastKnownStateTimestamp !== undefined && lastKnownStateTimestamp.isBefore(this.startedAt)) {
+            // if we last saw this activity BEFORE we started event polling (modqueue) then it's not useful to us
+            lastKnownStateTimestamp = undefined;
+        }
+        await this.resources.setActivityLastSeenDate(item.name);
+
+        // if modqueue is running then we know we are checking for new reports every X seconds
+        if(options.activitySource.identifier === 'modqueue') {
+            // if the activity is from modqueue and only has one report then we know that report was just created
+            if(item.num_reports === 1
+                // otherwise if it has more than one report AND we have seen it (its only seen if it has already been stored (in below block))
+                // then we are reasonably sure that any reports created were in the last X seconds
+                || (item.num_reports > 1 && lastKnownStateTimestamp !== undefined)) {
+
+                lastKnownStateTimestamp = dayjs().subtract(this.modqueueInterval, 'seconds');
+            }
+        }
+        // if activity is not from modqueue then known good timestamps for "time between last known report and now" is dependent on these things:
+        // 1) (most accurate) lastKnownStateTimestamp -- only available if activity either had 0 reports OR 1+ and existing reports have been stored (see below code)
+        // 2) last stored report time from Activity
+        // 3) create date of activity
+
+        let shouldPersistReports = false;
+
+        if (existingEntity === null) {
+            activityEntity = Activity.fromSnoowrapActivity(this.managerEntity.subreddit, activity, lastKnownStateTimestamp);
+            // always persist if activity is not already persisted and any reports exist
+            if (item.num_reports > 0) {
+                shouldPersistReports = true;
+            }
+        } else {
+            activityEntity = existingEntity;
+            // always persist if reports need to be updated
+            if (activityEntity.syncReports(item, lastKnownStateTimestamp)) {
+                shouldPersistReports = true;
+            }
+        }
+
+        if (shouldPersistReports) {
+            activityEntity = await this.activityRepo.save(activityEntity);
+        }
+
+        const itemId = await item.id;
+
         if(await this.resources.hasRecentSelf(item)) {
             let recentMsg = `Found in Activities recently (last ${this.resources.selfTTL} seconds) modified/created by this bot`;
             if(force) {
@@ -895,15 +973,6 @@ export class Manager extends EventEmitter implements RunningStates {
             }
         }
 
-        let activityEntity: Activity;
-        const existingEntity = await this.activityRepo.findOneBy({_id: item.name});
-        if(existingEntity === null) {
-            activityEntity = Activity.fromSnoowrapActivity(this.managerEntity.subreddit, activity);
-        } else {
-            activityEntity = existingEntity;
-        }
-
-        const event = new CMEvent();
         event.triggered = false;
         event.manager = this.managerEntity;
         event.activity = activityEntity;
@@ -918,7 +987,6 @@ export class Manager extends EventEmitter implements RunningStates {
         itemIdentifiers.push(`${checkType === 'Submission' ? 'SUB' : 'COM'} ${itemId}`);
         this.currentLabels = itemIdentifiers;
         let ePeek = '';
-        let peekParts: ItemContent;
         try {
             const [peek, { content: peekContent }] = await itemContentPeek(item);
             ePeek = peekContent;
@@ -963,7 +1031,6 @@ export class Manager extends EventEmitter implements RunningStates {
                             delay: dayjs.duration(remaining, 'seconds'),
                             id: 'notUsed',
                             queuedAt: dayjs(),
-                            processing: false,
                             activity,
                             author: getActivityAuthorName(activity.author),
                         });
@@ -977,7 +1044,7 @@ export class Manager extends EventEmitter implements RunningStates {
             }
             // refresh signal from firehose if activity was ingested multiple times before processing or re-queued while processing
             // want to make sure we have the most recent data
-            if(runtimeShouldRefresh || refresh) {
+            if(runtimeShouldRefresh) {
                 this.logger.verbose(`Refreshed data`);
                 // @ts-ignore
                 item = await activity.refresh();
@@ -1075,6 +1142,10 @@ export class Manager extends EventEmitter implements RunningStates {
                 event.runResults = runResults;
                 //actionedEvent.runResults = runResults;
 
+                const checksRun = actionedEvent.runResults.map(x => x.checkResults).flat().length;
+                let actionsRun = actionedEvent.runResults.map(x => x.checkResults?.map(y => y.actionResults)).flat().length;
+                let totalRulesRun = actionedEvent.runResults.map(x => x.checkResults?.map(y => y.ruleResults)).flat(5).length;
+
                 // determine if event should be recorded
                 const allOutputs = [...new Set(runResults.map(x => x.checkResults.map(y => y.recordOutputs ?? [])).flat(2).filter(x => recordOutputTypes.includes(x)))];
                 if(allOutputs.length > 0) {
@@ -1106,11 +1177,108 @@ export class Manager extends EventEmitter implements RunningStates {
                         }
                         await this.eventRepo.save(event);
                     }
-                }
+                    if (allOutputs.includes('influx') && this.influxClients.length > 0) {
+                        try {
+                            const time = dayjs().valueOf()
 
-                const checksRun = actionedEvent.runResults.map(x => x.checkResults).flat().length;
-                let actionsRun = actionedEvent.runResults.map(x => x.checkResults?.map(y => y.actionResults)).flat().length;
-                let totalRulesRun = actionedEvent.runResults.map(x => x.checkResults?.map(y => y.ruleResults)).flat().length;
+                            const measurements: Point[] = [];
+
+                            measurements.push(new Point('event')
+                                .timestamp(time)
+                                .tag('triggered', event.triggered ? '1' : '0')
+                                .tag('activityType', isSubmission(item) ? 'submission' : 'comment')
+                                .tag('sourceIdentifier', event.source.identifier ?? 'unknown')
+                                .tag('sourceType', event.source.type)
+                                .stringField('eventId', event.id)
+                                .stringField('activityId', event.activity.id)
+                                .stringField('author', actionedEvent.activity.author)
+                                .intField('processingTime', time - event.processedAt.valueOf())
+                                .intField('queuedTime', event.processedAt.valueOf() - event.queuedAt.valueOf())
+                                .intField('runsProcessed', actionedEvent.runResults.length)
+                                .intField('runsTriggered', actionedEvent.runResults.filter(x => x.triggered).length)
+                                .intField('checksProcessed', checksRun)
+                                .intField('checksTriggered', actionedEvent.runResults.map(x => x.checkResults).flat().filter(x => x.triggered).length)
+                                .intField('totalRulesProcessed', totalRulesRun)
+                                .intField('rulesTriggered', actionedEvent.runResults.map(x => x.checkResults?.map(y => y.ruleResults)).flat(5).filter((x: RuleResultEntity) => x.triggered === true).length)
+                                .intField('uniqueRulesProcessed', allRuleResults.length)
+                                .intField('cachedRulesProcessed', totalRulesRun - allRuleResults.length)
+                                .intField('actionsProcessed', actionsRun)
+                                .intField('apiUsage', startingApiLimit - this.client.ratelimitRemaining));
+
+                            const defaultPoint = () => new Point('triggeredEntity')
+                                .timestamp(time)
+                                .tag('activityType', isSubmission(item) ? 'submission' : 'comment')
+                                .tag('sourceIdentifier', event.source.identifier ?? 'unknown')
+                                .tag('sourceType', event.source.type)
+                                .stringField('activityId', event.activity.id)
+                                .stringField('author', actionedEvent.activity.author)
+                                .stringField('eventId', event.id);
+
+                            for (const r of event.runResults) {
+                                if (r.triggered) {
+                                    measurements.push(defaultPoint()
+                                        .tag('entityType', 'run')
+                                        .tag('name', r.run.name));
+                                    for (const c of r.checkResults) {
+                                        if (c.triggered) {
+                                            measurements.push(defaultPoint()
+                                                .tag('entityType', 'check')
+                                                .stringField('name', c.check.name)
+                                                .tag('fromCache', c.fromCache ? '1' : '0'));
+
+                                            if (c.ruleResults !== undefined) {
+                                                for (const ru of c.ruleResults) {
+                                                    if (ru.result.triggered) {
+                                                        measurements.push(defaultPoint()
+                                                            .tag('entityType', 'rule')
+                                                            .stringField('name', ru.result.premise.name)
+                                                            .tag('fromCache', ru.result.fromCache ? '1' : '0'))
+                                                    }
+                                                }
+                                            }
+                                            if (c.ruleSetResults !== undefined) {
+                                                for (const rs of c.ruleSetResults) {
+                                                    if (rs.result.triggered) {
+                                                        measurements.push(defaultPoint()
+                                                            .tag('entityType', 'ruleSet'));
+                                                        for (const ru of rs.result.results) {
+                                                            if (ru.triggered) {
+                                                                measurements.push(defaultPoint()
+                                                                    .tag('entityType', 'rule')
+                                                                    .stringField('name', ru.premise.name))
+                                                            }
+                                                        }
+                                                    }
+                                                }
+                                            }
+                                            if (c.actionResults !== undefined) {
+                                                for (const a of c.actionResults) {
+                                                    if (a.run) {
+                                                        measurements.push(defaultPoint()
+                                                            .tag('entityType', 'action')
+                                                            .stringField('name', a.premise.name)
+                                                            .tag('dryRun', a.dryRun ? '1' : '0')
+                                                            .tag('succes', a.success ? '1' : '0')
+                                                        )
+                                                    }
+                                                }
+                                            }
+                                        }
+                                    }
+                                }
+                            }
+
+                            for (const client of this.influxClients) {
+                                await client.writePoint(measurements);
+                            }
+                        } catch (e: any) {
+                            this.logger.error(new CMError('Error occurred while building or sending Influx data', {
+                                cause: e,
+                                isSerious: false
+                            }));
+                        }
+                    }
+                }
 
                 this.logger.verbose(`Run Stats:        Checks ${checksRun} | Rules => Total: ${totalRulesRun} Unique: ${allRuleResults.length} Cached: ${totalRulesRun - allRuleResults.length} Rolling Avg: ~${formatNumber(this.rulesUniqueRollingAvg)}/s | Actions ${actionsRun}`);
                 this.logger.verbose(`Reddit API Stats: Initial ${startingApiLimit} | Current ${this.client.ratelimitRemaining} | Used ~${startingApiLimit - this.client.ratelimitRemaining} | Events ~${formatNumber(this.eventsRollingAvg)}/s`);
@@ -1348,7 +1516,6 @@ export class Manager extends EventEmitter implements RunningStates {
                 state: RUNNING,
                 causedBy
             }
-            this.startDelayQueue();
             if(!suppressNotification) {
                 this.notificationManager.handle('runStateChanged', 'Queue Started', reason, causedBy);
             }
@@ -1466,6 +1633,11 @@ export class Manager extends EventEmitter implements RunningStates {
                 s.startInterval();
             }
             this.startedAt = dayjs();
+
+            const modQueuePollOpts = this.pollOptions.find(x => x.pollOn === 'modqueue');
+            if(modQueuePollOpts !== undefined) {
+                this.modqueueInterval = modQueuePollOpts.interval;
+            }
         }
 
         this.logger.info('Event polling STARTED');
@@ -1563,6 +1735,14 @@ export class Manager extends EventEmitter implements RunningStates {
         await this.syncRunningState('managerState');
     }
 
+    async destroy(causedBy: Invokee = 'system', options?: ManagerStateChangeOption) {
+        await this.stop(causedBy, options);
+        clearInterval(this.eventsSampleInterval);
+        clearInterval(this.delayedQueueInterval);
+        clearInterval(this.rulesUniqueSampleInterval)
+        await this.cacheManager.destroy(this.subreddit.display_name);
+    }
+
     setInitialRunningState(managerEntity: RunningStateEntities, type: RunningStateTypes): RunningState {
         if(managerEntity[type].runType.name === 'stopped' && managerEntity[type].invokee.name === 'user') {
             return {state: STOPPED, causedBy: 'user'};

src/Subreddit/ModNotes/ModAction.ts

@@ -0,0 +1,53 @@
+import {Submission, RedditUser, Comment, Subreddit, PrivateMessage} from "snoowrap/dist/objects"
+import {generateSnoowrapEntityFromRedditThing, parseRedditFullname} from "../../util"
+import Snoowrap from "snoowrap";
+
+//import {ExtendedSnoowrap} from "../../Utils/SnoowrapClients";
+
+export interface ModActionRaw {
+    action?: string | null
+    reddit_id?: string | null
+    details?: string | null
+    description?: string | null
+}
+
+export class ModAction {
+    action?: string
+    actedOn?: RedditUser | Submission | Comment | Subreddit | PrivateMessage
+    details?: string
+    description?: string
+
+    constructor(data: ModActionRaw | undefined, client: Snoowrap) {
+        const {
+            action,
+            reddit_id,
+            details,
+            description
+        } = data || {};
+        this.action = action !== null ? action : undefined;
+        this.details = details !== null ? details : undefined;
+        this.description = description !== null ? description : undefined;
+
+        if (reddit_id !== null && reddit_id !== undefined) {
+            const thing = parseRedditFullname(reddit_id);
+            if (thing !== undefined) {
+                this.actedOn = generateSnoowrapEntityFromRedditThing(thing, client);
+            }
+        }
+    }
+
+    toRaw(): ModActionRaw {
+        return {
+            action: this.action,
+            details: this.details,
+            reddit_id: this.actedOn !== undefined ? this.actedOn.id : undefined,
+            description: this.description
+        }
+    }
+
+    toJSON() {
+        return this.toRaw();
+    }
+}
+
+export default ModAction;

src/Subreddit/ModNotes/ModNote.ts

@@ -0,0 +1,119 @@
+import {ModAction, ModActionRaw} from "./ModAction";
+import {Submission, RedditUser, Comment, Subreddit} from "snoowrap/dist/objects"
+import {ModUserNote, ModUserNoteRaw} from "./ModUserNote";
+//import {ExtendedSnoowrap} from "../../Utils/SnoowrapClients";
+import dayjs, {Dayjs} from "dayjs";
+import {generateSnoowrapEntityFromRedditThing, parseRedditFullname} from "../../util";
+import Snoowrap from "snoowrap";
+import {ModActionType, ModUserNoteLabel} from "../../Common/Infrastructure/Atomic";
+import {RedditThing} from "../../Common/Infrastructure/Reddit";
+
+export interface ModNoteSnoowrapPopulated extends Omit<ModNoteRaw, 'subreddit' | 'user'> {
+    subreddit: Subreddit
+    user: RedditUser
+}
+
+export interface CreateModNoteData {
+    user: RedditUser
+    subreddit: Subreddit
+    activity?: Submission | Comment | RedditUser
+    label?: ModUserNoteLabel
+    note?: string
+}
+
+export const asCreateModNoteData = (val: any): val is CreateModNoteData => {
+    if(val !== null && typeof val === 'object') {
+        return val.user instanceof RedditUser && val.subreddit instanceof Subreddit && typeof val.note === 'string';
+    }
+    return false;
+}
+
+
+export interface ModNoteRaw {
+    subreddit: string
+    subreddit_id: string
+
+    user: string
+    user_id: string
+
+    operator: string
+    operator_id: string
+
+    id: string
+    created_at: number
+    cursor?: string
+    type: ModActionType | string
+    mod_action_data: ModActionRaw
+    user_note_data: ModUserNoteRaw
+}
+
+export class ModNote {
+
+    createdBy: RedditUser | Subreddit
+    createdByName?: string
+    createdAt: Dayjs
+    action: ModAction
+    note: ModUserNote
+    user: RedditUser
+    operatorVal: string
+    cursor?: string
+    id: string
+    subreddit: Subreddit
+    type: ModActionType | string
+
+
+    constructor(data: ModNoteRaw, client: Snoowrap) {
+
+        this.createdByName = data.operator;
+        this.createdAt = dayjs.unix(data.created_at);
+        this.id = data.id;
+        this.type = data.type;
+        this.cursor = data.cursor;
+
+        this.subreddit = new Subreddit({display_name: data.subreddit, id: data.subreddit_id}, client, false);
+        this.user = new RedditUser({name: data.user, id: data.user_id}, client, false);
+
+        this.operatorVal = data.operator;
+
+        const opThing = parseRedditFullname(data.operator_id) as RedditThing;
+        this.createdBy = generateSnoowrapEntityFromRedditThing(opThing, client) as RedditUser | Subreddit;
+        if (this.createdBy instanceof RedditUser) {
+            this.createdBy.name = data.operator;
+        }
+
+        this.action = new ModAction(data.mod_action_data, client);
+        if (this.action.actedOn instanceof RedditUser && this.action.actedOn.id === this.user.id) {
+            this.action.actedOn = this.user;
+        }
+
+        this.note = new ModUserNote(data.user_note_data, client);
+        if (this.note.actedOn instanceof RedditUser && this.note.actedOn.id === this.user.id) {
+            this.note.actedOn = this.user;
+        }
+    }
+
+    toRaw(): ModNoteRaw {
+        return {
+            subreddit: this.subreddit.display_name,
+            subreddit_id: this.subreddit.id,
+
+            user: this.user.name,
+            user_id: this.user.id,
+
+            operator: this.operatorVal,
+            operator_id: this.createdBy.id,
+
+            mod_action_data: this.action.toRaw(),
+
+            id: this.id,
+            user_note_data: this.note.toRaw(),
+            created_at: this.createdAt.unix(),
+            type: this.type,
+            cursor: this.cursor
+        }
+    }
+
+    toJSON() {
+        return this.toRaw();
+    }
+}

src/Subreddit/ModNotes/ModUserNote.ts

@@ -0,0 +1,48 @@
+import {Comment, PrivateMessage, RedditUser, Submission} from "snoowrap/dist/objects";
+import {ModUserNoteLabel} from "../../Common/Infrastructure/Atomic";
+//import {ExtendedSnoowrap} from "../../Utils/SnoowrapClients";
+import {generateSnoowrapEntityFromRedditThing, parseRedditFullname} from "../../util";
+import Snoowrap from "snoowrap";
+
+export interface ModUserNoteRaw {
+    note?: string | null
+    reddit_id?: string | null
+    label?: string | null
+}
+
+export class ModUserNote {
+    note?: string
+    actedOn?: RedditUser | Submission | Comment | PrivateMessage
+    label?: ModUserNoteLabel
+
+    constructor(data: ModUserNoteRaw | undefined, client: Snoowrap) {
+        const {
+            note,
+            reddit_id,
+            label
+        } = data || {};
+        this.note = note !== null ? note : undefined;
+        this.label = label !== null ? label as ModUserNoteLabel : undefined;
+
+        if (reddit_id !== null && reddit_id !== undefined) {
+            const thing = parseRedditFullname(reddit_id);
+            if (thing !== undefined) {
+                this.actedOn = generateSnoowrapEntityFromRedditThing(thing, client) as RedditUser | Submission | Comment;
+            }
+        }
+    }
+
+    toRaw(): ModUserNoteRaw {
+        return {
+            note: this.note,
+            reddit_id: this.actedOn !== undefined ? this.actedOn.id : undefined,
+            label: this.label
+        }
+    }
+
+    toJSON() {
+        return this.toRaw();
+    }
+}
+
+export default ModUserNote;

src/Subreddit/Streams.ts

@@ -1,11 +1,12 @@
 import {Poll, SnooStormOptions} from "snoostorm"
-import Snoowrap, {Listing} from "snoowrap";
+import Snoowrap, {Listing, RedditContent} from "snoowrap";
 import {EventEmitter} from "events";
 import {PollConfiguration} from "snoostorm/out/util/Poll";
 import {DEFAULT_POLLING_INTERVAL} from "../Common/interfaces";
 import {mergeArr, parseDuration, random} from "../util";
 import { Logger } from "winston";
 import {ErrorWithCause} from "pony-cause";
+import dayjs, {Dayjs as DayjsObj} from "dayjs";
 
 type Awaitable<T> = Promise<T> | T;
 
@@ -16,13 +17,15 @@ interface RCBPollingOptions<T> extends SnooStormOptions {
     name?: string,
     processed?: Set<T[keyof T]>
     label?: string
+    dateCutoff?: boolean
 }
 
 interface RCBPollConfiguration<T> extends PollConfiguration<T>,RCBPollingOptions<T> {
     get: () => Promise<Listing<T>>
+    dateCutoff: boolean
 }
 
-export class SPoll<T extends object> extends Poll<T> {
+export class SPoll<T extends RedditContent<object>> extends Poll<T> {
     identifier: keyof T;
     getter: () => Promise<Listing<T>>;
     frequency;
@@ -31,6 +34,8 @@ export class SPoll<T extends object> extends Poll<T> {
     // -- that is, we don't want to emit the items we immediately fetch on a fresh poll start since they existed "before" polling started
     newStart: boolean = true;
     enforceContinuity: boolean;
+    useDateCutoff: boolean;
+    dateCutoff?: DayjsObj;
     randInterval?: { clear: () => void };
     name: string = 'Reddit Stream';
     logger: Logger;
@@ -47,7 +52,8 @@ export class SPoll<T extends object> extends Poll<T> {
             name,
             subreddit,
             label = 'Polling',
-            processed
+            processed,
+            dateCutoff,
         } = options;
         this.subreddit = subreddit;
         this.name = name !== undefined ? name : this.name;
@@ -56,6 +62,7 @@ export class SPoll<T extends object> extends Poll<T> {
         this.getter = get;
         this.frequency = frequency;
         this.enforceContinuity = enforceContinuity;
+        this.useDateCutoff = dateCutoff;
 
         // if we pass in processed on init the intention is to "continue" from where the previous stream left off
         // WITHOUT new start behavior
@@ -80,7 +87,7 @@ export class SPoll<T extends object> extends Poll<T> {
                     // but only continue iterating if stream enforces continuity and we've only seen new items so far
                     while(page === 1 || (self.enforceContinuity && !self.newStart && !anyAlreadySeen)) {
                         if(page !== 1) {
-                            self.logger.debug(`Did not find any already seen activities and continuity is enforced. This probably means there were more new items than 1 api call can return. Fetching next page (${page})...`);
+                            self.logger.debug(`Did not find any already seen Activities and continuity is enforced. This probably means there were more new Activities than 1 api call can return. Fetching next page (page ${page})...`);
                             // @ts-ignore
                             batch = await batch.fetchMore({amount: 100});
                         }
@@ -95,24 +102,67 @@ export class SPoll<T extends object> extends Poll<T> {
                                 continue;
                             }
 
-                            // Emit for new items and add it to the list
+                            // add new item to list and set as processed
                             newItems.push(item);
                             self.processed.add(id);
-                            // but don't emit on new start since we are "buffering" already existing activities
-                            if(!self.newStart) {
-                                self.emit("item", item);
-                            }
                         }
                         page++;
                     }
-                    const newItemMsg = `Found ${newItems.length} new items out of ${batch.length} returned`;
+
                     if(self.newStart) {
-                        self.logger.debug(`${newItemMsg} but will ignore all on first start.`);
+
+                        self.logger.debug(`Found ${newItems.length} unseen Activities out of ${batch.length} returned, but will ignore all on first start.`);
                         self.emit("listing", []);
+
+                        if(self.useDateCutoff && self.dateCutoff === undefined) {
+                            self.logger.debug('Cutoff date should be used for filtering unseen Activities but none was set. Will determine date based on newest Activity returned from first polling results.');
+                            if(newItems.length === 0) {
+                                // no items found, cutoff is now
+                                self.dateCutoff = dayjs();
+                                self.logger.debug(`Cutoff date set to NOW (${self.dateCutoff.format('YYYY-MM-DD HH:mm:ssZ')}) since no unseen Activities returned. Unseen Activities will only be returned if newer than this date.`);
+                            } else {
+                                // set cutoff date for new items from the newest items found
+                                const sorted = [...newItems];
+                                sorted.sort((a, z) => z.created_utc - a.created_utc);
+                                self.dateCutoff = dayjs.unix(sorted[0].created_utc);
+                                self.logger.debug(`Cutoff date set to newest unseen Activity found, ${self.dateCutoff.format('YYYY-MM-DD HH:mm:ssZ')}. Unseen Activities will only be returned if newer than this date.`);
+                            }
+                        }
+
                     } else {
-                        self.logger.debug(newItemMsg);
+
+                        // applies mostly (only?) to 'unmoderated' polling
+                        //
+                        // scenario:
+                        // * polling unmoderated for many subreddits and unmoderated has not been clearing out for awhile so it has many (100's) of items
+                        // * a moderator, or CM, iterates through list and actions items so the list is shorter
+                        // * CM polling unmoderated and finds "unseen" items that don't appear in unprocessed list
+                        //
+                        // these "unseen" are OLDER than the "newest" seen items we have got from polling because CM only got the first page of unmoderated items
+                        // so now CM emits them as "new" and CM starts processing them. If it continues to process them then more and more 'unseen old' items continue to appear in stream,
+                        // creating a feedback loop where CM eventually processes the entire backlog of unmoderated items
+                        //
+                        // this is UNWANTED behavior. CM should only ever process items added to polling sources after it starts monitoring them.
+                        //
+                        // to address this we use a cutoff date determined from the newest activity returned from the first polling call (or current datetime if none returned)
+                        // then we make sure any 'new' items (unseen by CM) are newer than this cutoff date
+                        //
+                        // -- this is the default behavior for all polling sources except modqueue. See comments on that class below for why.
+                        const unixCutoff = self.useDateCutoff && self.dateCutoff !== undefined ? self.dateCutoff.unix() : undefined;
+                        const validNewItems = unixCutoff === undefined || newItems.length === 0 ? newItems : newItems.filter(x => x.created_utc >= unixCutoff);
+
+                        if(validNewItems.length !== newItems.length && self.dateCutoff !== undefined) {
+                            self.logger.warn(`${newItems.length - validNewItems.length} unseen Activities were created before cutoff date (${self.dateCutoff.format('YYYY-MM-DD HH:mm:ssZ')}) and have been filtered out.`);
+                        }
+                        self.logger.debug(`Found ${validNewItems.length} valid, unseen Activities out of ${batch.length} returned`);
+
+                        // only emit if not new start since we are "buffering" already existing activities
+                        for(const item of validNewItems) {
+                            self.emit('item', item);
+                        }
+
                         // Emit the new listing of all new items
-                        self.emit("listing", newItems);
+                        self.emit("listing", validNewItems);
                     }
                     // no longer new start on n+1 interval
                     self.newStart = false;
@@ -146,6 +196,7 @@ export class SPoll<T extends object> extends Poll<T> {
         this.logger.debug(msg);
         this.running = false;
         this.newStart = true;
+        this.dateCutoff = undefined;
         super.end();
     }
 }
@@ -159,6 +210,7 @@ export class UnmoderatedStream extends SPoll<Snoowrap.Submission | Snoowrap.Comm
             get: async () => client.getSubreddit(options.subreddit).getUnmoderated(options),
             identifier: "id",
             name: 'Unmoderated',
+            dateCutoff: true,
             ...options,
         });
     }
@@ -173,6 +225,9 @@ export class ModQueueStream extends SPoll<Snoowrap.Submission | Snoowrap.Comment
             get: async () => client.getSubreddit(options.subreddit).getModqueue(options),
             identifier: "id",
             name: 'Modqueue',
+            // cannot use cutoff date since 'new' items in this list are based on when they were reported, not when the item was created
+            // and unfortunately there is no way to use that "reported at" time since reddit doesn't include it in the returned items
+            dateCutoff: false,
             ...options,
         });
     }
@@ -187,6 +242,7 @@ export class SubmissionStream extends SPoll<Snoowrap.Submission | Snoowrap.Comme
             get: async () => client.getNew(options.subreddit, options),
             identifier: "id",
             name: 'Submission',
+            dateCutoff: true,
             ...options,
         });
     }
@@ -201,6 +257,7 @@ export class CommentStream extends SPoll<Snoowrap.Submission | Snoowrap.Comment>
             get: async () => client.getNewComments(options.subreddit, options),
             identifier: "id",
             name: 'Comment',
+            dateCutoff: true,
             ...options,
         });
     }

src/Subreddit/UserNotes.ts

@@ -15,6 +15,7 @@ import {RichContent} from "../Common/interfaces";
 import {Cache} from 'cache-manager';
 import {isScopeError} from "../Utils/Errors";
 import {ErrorWithCause} from "pony-cause";
+import {UserNoteType} from "../Common/Infrastructure/Atomic";
 
 interface RawUserNotesPayload {
     ver: number,
@@ -125,7 +126,7 @@ export class UserNotes {
         return this.mod as RedditUser;
     }
 
-    async addUserNote(item: (Submission|Comment), type: string | number, text: string = '', wikiEditReasonPrefix?: string): Promise<UserNote>
+    async addUserNote(item: (Submission|Comment), type: UserNoteType | number, text: string = '', wikiEditReasonPrefix?: string): Promise<UserNote>
     {
         const payload = await this.retrieveData();
         const userName = getActivityAuthorName(item.author);
@@ -235,7 +236,7 @@ export interface UserNoteJson extends RichContent {
      * User Note type key
      * @examples ["spamwarn"]
      * */
-    type: string,
+    type: UserNoteType,
 }
 
 export class UserNote {
@@ -246,7 +247,7 @@ export class UserNote {
     // noteType: string | null;
     // link: string;
 
-    constructor(public time: Dayjs, public text: string, public modIndex: number, public noteType: string | number, public link: (string | null) = null, public moderator?: RedditUser) {
+    constructor(public time: Dayjs, public text: string, public modIndex: number, public noteType: UserNoteType | number, public link: (string | null) = null, public moderator?: RedditUser) {
 
     }
 

src/SubredditConfigData.ts

@@ -0,0 +1,33 @@
+import {
+    ActivityCheckConfigValue,
+} from "./Check";
+import {ManagerOptions} from "./Common/interfaces";
+import {RunConfigHydratedData, RunConfigValue, RunConfigObject} from "./Run";
+
+export interface SubredditConfigData extends ManagerOptions {
+    /**
+     * A list of all the checks that should be run for a subreddit.
+     *
+     * Checks are split into two lists -- submission or comment -- based on kind and run independently.
+     *
+     * Checks in each list are run in the order found in the configuration.
+     *
+     * When a check "passes", and actions are performed, then all subsequent checks are skipped.
+     * @minItems 1
+     * */
+    checks?: ActivityCheckConfigValue[]
+
+    /**
+     * A list of sets of Checks to run
+     * @minItems 1
+     * */
+    runs?: RunConfigValue[]
+}
+
+export interface SubredditConfigHydratedData extends Omit<SubredditConfigData, 'checks'> {
+    runs?: RunConfigHydratedData[]
+}
+
+export interface SubredditConfigObject extends SubredditConfigHydratedData {
+    runs?: RunConfigObject[]
+}

src/Utils/InvalidRegexError.ts

@@ -1,9 +1,9 @@
 import ExtendableError from "es6-error";
 
 class InvalidRegexError extends ExtendableError {
-    constructor(regex: RegExp | RegExp[], val?: string, url?: string) {
+    constructor(regex: RegExp | RegExp[], val?: string, url?: string, message?: string) {
         const msgParts = [
-            'Regex(es) did not match the value given.',
+            message ?? 'Regex(es) did not match the value given.',
         ];
         let regArr = Array.isArray(regex) ? regex : [regex];
         for(const r of regArr) {

src/Utils/SnoowrapClients.ts

@@ -1,6 +1,9 @@
-import Snoowrap, {Listing} from "snoowrap";
-import {Subreddit} from "snoowrap/dist/objects";
+import Snoowrap, {Listing, RedditUser} from "snoowrap";
+import {Submission, Subreddit, Comment} from "snoowrap/dist/objects";
 import {parseSubredditName} from "../util";
+import {ModUserNoteLabel} from "../Common/Infrastructure/Atomic";
+import {CreateModNoteData, ModNote, ModNoteRaw, ModNoteSnoowrapPopulated} from "../Subreddit/ModNotes/ModNote";
+import {SimpleError} from "./Errors";
 
 // const proxyFactory = (endpoint: string) => {
 //     return class ProxiedSnoowrap extends Snoowrap {
@@ -14,6 +17,26 @@ import {parseSubredditName} from "../util";
 //     }
 // }
 
+export interface ModNoteGetOptions {
+    before?: string,
+    filter?: ModUserNoteLabel,
+    limit?: number
+}
+
+export interface ModNotesRaw {
+    mod_notes: ModNoteSnoowrapPopulated[]
+    start_cursor: string
+    end_cursor: string
+    has_next_page: boolean
+}
+
+export interface ModNotesResponse {
+    notes: ModNote[]
+    startCursor: string
+    endCursor: string
+    isFinished: boolean
+}
+
 export class ExtendedSnoowrap extends Snoowrap {
 
     constructor(args: any) {
@@ -53,6 +76,70 @@ export class ExtendedSnoowrap extends Snoowrap {
             }
         });
     }
+
+    async getModNotes(subreddit: Subreddit | string, user: RedditUser | string, options: ModNoteGetOptions = {limit: 100}): Promise<ModNotesResponse> {
+
+        const authorName = typeof user === 'string' ? user : user.name;
+        if(authorName === '[deleted]') {
+            throw new SimpleError(`User is '[deleted]', cannot retrieve`, {isSerious: false});
+        }
+        const subredditName = typeof subreddit === 'string' ? subreddit : subreddit.display_name;
+
+        const data: any = {
+            subreddit: subredditName,
+            user: authorName,
+            ...options
+        };
+        const response = await this.oauthRequest({
+            uri: `/api/mod/notes`,
+            method: 'get',
+            qs: data
+        }) as ModNotesRaw;
+
+        // TODO get all mod notes (iterate pages if has_next_page)
+        return {
+
+            // "undo" the _populate function snoowrap uses to replace user/subreddit keys with Proxies
+            // because we want to store the "raw" response data when caching (where user/subreddit keys are strings) so we can construct ModNote from either api response or cache using same data
+            notes: response.mod_notes.map(x => {
+                return new ModNote({
+                    ...x,
+                    subreddit: x.subreddit.display_name,
+                    user: x.user.name,
+                }, this);
+
+            }),
+            startCursor: response.start_cursor,
+            endCursor: response.end_cursor,
+            isFinished: !response.has_next_page
+        }
+    }
+
+    /**
+     * Add a Mod Note
+     *
+     * @see https://www.reddit.com/dev/api#POST_api_mod_notes
+     * */
+    async addModNote(data: CreateModNoteData): Promise<ModNote> {
+        const {note, label} = data;
+
+        const requestData: any = {
+            note,
+            label,
+            subreddit: data.subreddit.display_name,
+            user: data.user.name,
+        }
+        if(data.activity !== undefined) {
+            requestData.reddit_id = data.activity.name;
+        }
+
+        const response =await this.oauthRequest({
+            uri: `/api/mod/notes`,
+            method: 'post',
+            form: requestData
+        }) as { created: ModNoteRaw };
+        return new ModNote(response.created, this);
+    }
 }
 
 export class RequestTrackingSnoowrap extends ExtendedSnoowrap {

src/Utils/SnoowrapUtils.ts

@@ -17,7 +17,7 @@ import {
     convertSubredditsRawToStrong,
     getActivityAuthorName,
     getActivitySubredditName,
-    isStrongSubredditState,
+    isStrongSubredditState, isSubmission,
     mergeArr,
     normalizeName,
     parseDurationValToDuration,
@@ -117,46 +117,28 @@ export const isSubreddit = async (subreddit: Subreddit, stateCriteria: Subreddit
 }
 
 export const renderContent = async (template: string, data: (Submission | Comment), ruleResults: RuleResultEntity[] = [], usernotes: UserNotes) => {
+    const conditional: any = {};
+    if(data.can_mod_post) {
+        conditional.reports = data.num_reports;
+        conditional.modReports = data.mod_reports.length;
+        conditional.userReports = data.user_reports.length;
+    }
+    if(asSubmission(data)) {
+        conditional.nsfw = data.over_18;
+        conditional.spoiler = data.spoiler;
+        conditional.op = true;
+        conditional.upvoteRatio = `${data.upvote_ratio * 100}%`;
+    } else {
+        conditional.op = data.is_submitter;
+    }
     const templateData: any = {
         kind: data instanceof Submission ? 'submission' : 'comment',
         author: await data.author.name,
-        // make this a getter so that if we don't load notes (and api call) if we don't need to
-        // didn't work either for some reason
-        // tried to get too fancy :(
-        // get notes() {
-        //     return usernotes.getUserNotes(data.author).then((notesData) => {
-        //         // return usable notes data with some stats
-        //         const current = notesData.length > 0 ? notesData[notesData.length -1] : undefined;
-        //         // group by type
-        //         const grouped = notesData.reduce((acc: any, x) => {
-        //             const {[x.noteType]: nt = []} = acc;
-        //             return Object.assign(acc, {[x.noteType]: nt.concat(x)});
-        //         }, {});
-        //         return {
-        //             data: notesData,
-        //             current,
-        //             ...grouped,
-        //         };
-        //     });
-        // },
-        // when i was trying to use mustache-async (didn't work)
-        // notes: async () => {
-        //     const notesData = await usernotes.getUserNotes(data.author);
-        //     // return usable notes data with some stats
-        //     const current = notesData.length > 0 ? notesData[notesData.length -1] : undefined;
-        //     // group by type
-        //     const grouped = notesData.reduce((acc: any, x) => {
-        //         const {[x.noteType]: nt = []} = acc;
-        //         return Object.assign(acc, {[x.noteType]: nt.concat(x)});
-        //     }, {});
-        //     return {
-        //         data: notesData,
-        //         current,
-        //         ...grouped,
-        //     };
-        // },
+        votes: data.score,
+        age: dayjs.duration(dayjs().diff(dayjs.unix(data.created))).humanize(),
         permalink: `https://reddit.com${data.permalink}`,
         botLink: BOT_LINK,
+        ...conditional
     }
     if (template.includes('{{item.notes')) {
         // we need to get notes
@@ -344,7 +326,7 @@ export const getAttributionIdentifier = (sub: Submission, useParentMediaDomain =
 
 export const activityIsRemoved = (item: Submission | Comment): boolean => {
     if(item.can_mod_post) {
-        if (item instanceof Submission) {
+        if (asSubmission(item)) {
             // when automod filters a post it gets this category
             return item.banned_at_utc !== null && item.removed_by_category !== 'automod_filtered';
         }
@@ -352,7 +334,7 @@ export const activityIsRemoved = (item: Submission | Comment): boolean => {
         // so if we want to processing filtered comments we need to check for this
         return item.banned_at_utc !== null && item.removed;
     } else {
-        if (item instanceof Submission) {
+        if (asSubmission(item)) {
             return item.removed_by_category === 'moderator' || item.removed_by_category === 'deleted';
         }
         // in subreddits the bot does not mod it is not possible to tell the difference between a comment that was removed by the user and one that was removed by a mod
@@ -362,7 +344,7 @@ export const activityIsRemoved = (item: Submission | Comment): boolean => {
 
 export const activityIsFiltered = (item: Submission | Comment): boolean => {
     if(item.can_mod_post) {
-        if (item instanceof Submission) {
+        if (asSubmission(item)) {
             // when automod filters a post it gets this category
             return item.banned_at_utc !== null && item.removed_by_category === 'automod_filtered';
         }
@@ -375,7 +357,7 @@ export const activityIsFiltered = (item: Submission | Comment): boolean => {
 }
 
 export const activityIsDeleted = (item: Submission | Comment): boolean => {
-    if (item instanceof Submission) {
+    if (asSubmission(item)) {
         return item.removed_by_category === 'deleted';
     }
     return item.author.name === '[deleted]'

src/Utils/databaseUtils.ts

@@ -3,6 +3,7 @@ import {MysqlConnectionOptions} from "typeorm/driver/mysql/MysqlConnectionOption
 import {MongoConnectionOptions} from "typeorm/driver/mongodb/MongoConnectionOptions";
 import {PostgresConnectionOptions} from "typeorm/driver/postgres/PostgresConnectionOptions";
 import {resolve, parse as parsePath} from 'path';
+// https://stackoverflow.com/questions/49618719/why-does-typeorm-need-reflect-metadata
 import "reflect-metadata";
 import {DataSource} from "typeorm";
 import {castToBool, fileOrDirectoryIsWriteable, mergeArr, resolvePath} from "../util";

src/Web/Client/StorageProvider.ts

@@ -88,7 +88,8 @@ export class CacheStorageProvider extends StorageProvider {
 
     constructor(caching: CacheOptions & StorageProviderOptions) {
         super(caching);
-        this.cache = createCacheManager({...caching, prefix: buildCachePrefix(['web'])}) as Cache;
+        const {logger, invitesMaxAge, loggerLabels, ...restCache } = caching;
+        this.cache = createCacheManager({...restCache, prefix: buildCachePrefix(['web'])}) as Cache;
         this.logger.debug('Using CACHE');
         if (caching.store === 'none') {
             this.logger.warn(`Using 'none' as cache provider means no one will be able to access the interface since sessions will never be persisted!`);

src/Web/Client/index.ts

@@ -29,8 +29,6 @@ import session, {Session, SessionData} from "express-session";
 import Snoowrap, {Subreddit} from "snoowrap";
 import {getLogger} from "../../Utils/loggerFactory";
 import EventEmitter from "events";
-import stream, {Readable, Writable, Transform} from "stream";
-import winston from "winston";
 import tcpUsed from "tcp-port-used";
 import http from "http";
 import jwt from 'jsonwebtoken';
@@ -39,13 +37,6 @@ import got from 'got';
 import sharedSession from "express-socket.io-session";
 import dayjs from "dayjs";
 import httpProxy from 'http-proxy';
-import normalizeUrl from 'normalize-url';
-import GotRequest from "got/dist/source/core";
-import {prettyPrintJson} from "pretty-print-json";
-// @ts-ignore
-import DelimiterStream from 'delimiter-stream';
-import {pipeline} from 'stream/promises';
-import {defaultBotStatus} from "../Common/defaults";
 import {arrayMiddle, booleanMiddle} from "../Common/middleware";
 import {BotInstance, CMInstanceInterface} from "../interfaces";
 import { URL } from "url";
@@ -54,8 +45,6 @@ import Autolinker from "autolinker";
 import path from "path";
 import {ExtendedSnoowrap} from "../../Utils/SnoowrapClients";
 import ClientUser from "../Common/User/ClientUser";
-import {BotStatusResponse, InviteData} from "../Common/interfaces";
-import {TransformableInfo} from "logform";
 import {SimpleError} from "../../Utils/Errors";
 import {ErrorWithCause} from "pony-cause";
 import {CMInstance} from "./CMInstance";
@@ -65,8 +54,6 @@ import { ActionPremise } from "../../Common/Entities/ActionPremise";
 import {CacheStorageProvider, DatabaseStorageProvider} from "./StorageProvider";
 import {nanoid} from "nanoid";
 import {MigrationService} from "../../Common/MigrationService";
-import {WebSetting} from "../../Common/WebEntities/WebSetting";
-import {CheckResultEntity} from "../../Common/Entities/CheckResultEntity";
 import {RuleResultEntity} from "../../Common/Entities/RuleResultEntity";
 import {RuleSetResultEntity} from "../../Common/Entities/RuleSetResultEntity";
 import { PaginationAwareObject } from "../Common/util";
@@ -95,13 +82,26 @@ app.use((req, res, next) => {
     }
 });
 
+const staticHeaders = (res: express.Response, path: string, stat: object) => {
+    res.setHeader('X-Robots-Tag', 'noindex');
+}
+const staticOpts = {
+    setHeaders: staticHeaders
+}
+
 app.use(bodyParser.urlencoded({extended: false}));
 //app.use(cookieParser());
 app.set('views', `${__dirname}/../assets/views`);
 app.set('view engine', 'ejs');
-app.use('/public', express.static(`${__dirname}/../assets/public`));
-app.use('/monaco', express.static(`${__dirname}/../../../node_modules/monaco-editor/`));
-app.use('/schemas', express.static(`${__dirname}/../../Schema/`));
+app.use('/public', express.static(`${__dirname}/../assets/public`, staticOpts));
+app.use('/monaco', express.static(`${__dirname}/../../../node_modules/monaco-editor/`, staticOpts));
+app.use('/schemas', express.static(`${__dirname}/../../Schema/`, staticOpts));
+
+app.use((req, res, next) => {
+    // https://developers.google.com/search/docs/advanced/crawling/block-indexing#http-response-header
+    res.setHeader('X-Robots-Tag', 'noindex');
+    next();
+});
 
 const userAgent = `web:contextBot:web`;
 
@@ -205,7 +205,9 @@ const webClient = async (options: OperatorConfig) => {
     logger.stream().on('log', (log: LogInfo) => {
         emitter.emit('log', log);
         webLogs.unshift(log);
-        webLogs = webLogs.slice(0, 201);
+        if(webLogs.length > 200) {
+            webLogs.splice(200);
+        }
     });
 
     const migrationService = new MigrationService({
@@ -606,106 +608,10 @@ const webClient = async (options: OperatorConfig) => {
     const cmInstances: CMInstance[] = [];
     let init = false;
     const formatter = defaultFormat();
-    const formatTransform = formatter.transform as (info: TransformableInfo, opts?: any) => TransformableInfo;
 
     let server: http.Server,
         io: SocketServer;
 
-    const startLogStream = (sessionData: Session & Partial<SessionData>, user: Express.User) => {
-        // @ts-ignore
-        const sessionId = sessionData.id as string;
-        
-        if(connectedUsers[sessionId] !== undefined) {
-
-            const delim = new DelimiterStream({
-                delimiter: '\r\n',
-            });
-
-            const currInstance = cmInstances.find(x => x.getName() === sessionData.botId);
-            if(currInstance !== undefined) {
-                const ac = new AbortController();
-                const options = {
-                    signal: ac.signal,
-                };
-
-                const retryFn = (retryCount = 0, err: any = undefined) => {
-                    const delim = new DelimiterStream({
-                        delimiter: '\r\n',
-                    });
-
-                    if(err !== undefined) {
-                        // @ts-ignore
-                        currInstance.logger.warn(new ErrorWithCause(`Log streaming encountered an error, trying to reconnect (retries: ${retryCount})`, {cause: err}), {user: user.name});
-                    }
-                    const gotStream = got.stream.get(`${currInstance.normalUrl}/logs`, {
-                        retry: {
-                            limit: 5,
-                        },
-                        headers: {
-                            'Authorization': `Bearer ${createToken(currInstance, user)}`,
-                        },
-                        searchParams: {
-                            limit: sessionData.limit,
-                            sort: sessionData.sort,
-                            level: sessionData.level,
-                            stream: true,
-                            streamObjects: true,
-                            formatted: false,
-                        }
-                    });
-
-                    if(err !== undefined) {
-                        gotStream.once('data', () => {
-                            currInstance.logger.info('Streaming resumed', {instance: currInstance.getName(), user: user.name});
-                        });
-                    }
-
-                    gotStream.retryCount = retryCount;
-                    const s = pipeline(
-                        gotStream,
-                        delim,
-                        options
-                    ) as Promise<void>;
-
-                    // ECONNRESET
-                    s.catch((err) => {
-                        if(err.code !== 'ABORT_ERR' && err.code !== 'ERR_STREAM_PREMATURE_CLOSE') {
-                            // @ts-ignore
-                            currInstance.logger.error(new ErrorWithCause('Unexpected error, or too many retries, occurred while streaming logs', {cause: err}), {user: user.name});
-                        }
-                    });
-
-
-                    delim.on('data', (c: any) => {
-                        const logObj = JSON.parse(c) as LogInfo;
-                        let subredditMessage;
-                        let allMessage;
-                        if(logObj.subreddit !== undefined) {
-                            const {subreddit, bot, ...rest} = logObj
-                            // @ts-ignore
-                            subredditMessage = formatLogLineToHtml(formatter.transform(rest)[MESSAGE], rest.timestamp);
-                        }
-                        if(logObj.bot !== undefined) {
-                            const {bot, ...rest} = logObj
-                            // @ts-ignore
-                            allMessage = formatLogLineToHtml(formatter.transform(rest)[MESSAGE], rest.timestamp);
-                        }
-                        // @ts-ignore
-                        let formattedMessage = formatLogLineToHtml(formatter.transform(logObj)[MESSAGE], logObj.timestamp);
-                        io.to(sessionId).emit('log', {...logObj, subredditMessage, allMessage, formattedMessage});
-                    });
-
-                    gotStream.once('retry', retryFn);
-                }
-
-                retryFn();
-
-                return ac;
-            }
-            return undefined;
-        }
-    }
-
     try {
         server = await app.listen(port);
         io = new SocketServer(server);
@@ -965,29 +871,7 @@ const webClient = async (options: OperatorConfig) => {
 
         res.render('status', {
             instances: shownInstances,
-            bots: resp.bots.map((x: BotStatusResponse) => {
-                const {subreddits = []} = x;
-                const subredditsWithSimpleLogs = subreddits.map(y => {
-                    let transformedLogs: string[];
-                    if(y.name === 'All') {
-                        // only need to remove bot name here
-                        transformedLogs = (y.logs as LogInfo[]).map((z: LogInfo) => {
-                           const {bot, ...rest} = z;
-                           // @ts-ignore
-                           return formatLogLineToHtml(formatter.transform(rest)[MESSAGE] as string, rest.timestamp);
-                        });
-                    } else {
-                        transformedLogs = (y.logs as LogInfo[]).map((z: LogInfo) => {
-                            const {bot, subreddit, ...rest} = z;
-                            // @ts-ignore
-                            return formatLogLineToHtml(formatter.transform(rest)[MESSAGE] as string, rest.timestamp);
-                        });
-                    }
-                    y.logs = transformedLogs;
-                    return y;
-                });
-                return {...x, subreddits: subredditsWithSimpleLogs};
-            }),
+            bots: resp.bots,
             botId: (req.instance as CMInstance).getName(),
             instanceId: (req.instance as CMInstance).getName(),
             isOperator: isOp,
@@ -1022,7 +906,7 @@ const webClient = async (options: OperatorConfig) => {
 
     app.postAsync('/config', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions(true)], async (req: express.Request, res: express.Response) => {
         const {subreddit} = req.query as any;
-        const {location, data, create = false} = req.body as any;
+        const {location, data, reason = 'Updated through CM Web', create = false} = req.body as any;
 
         const client = new ExtendedSnoowrap({
             userAgent,
@@ -1036,7 +920,7 @@ const webClient = async (options: OperatorConfig) => {
             const wiki = await client.getSubreddit(subreddit).getWikiPage(location);
             await wiki.edit({
                 text: data,
-                reason: create ? 'Created Config through CM Web' : 'Updated through CM Web'
+                reason
             });
         } catch (err: any) {
             res.status(500);
@@ -1398,42 +1282,6 @@ const webClient = async (options: OperatorConfig) => {
             clearSockStreams(socket.id);
             socket.join(session.id);
 
-            socket.on('viewing', (data) => {
-                if(user !== undefined) {
-                    const {subreddit, bot: botVal} = data;
-                    const currBot = cmInstances.find(x => x.getName() === session.botId);
-                    if(currBot !== undefined) {
-
-                        if(liveInterval !== undefined) {
-                            clearInterval(liveInterval)
-                        }
-
-                        const liveEmit = async () => {
-                            try {
-                                const resp = await got.get(`${currBot.normalUrl}/liveStats`, {
-                                    headers: {
-                                        'Authorization': `Bearer ${createToken(currBot, user)}`,
-                                    },
-                                    searchParams: {
-                                        bot: botVal,
-                                        subreddit
-                                    }
-                                });
-                                const stats = JSON.parse(resp.body);
-                                io.to(session.id).emit('liveStats', stats);
-                            } catch (err: any) {
-                                currBot.logger.error(new ErrorWithCause('Could not retrieve live stats', {cause: err}));
-                            }
-                        }
-
-                        // do an initial get
-                        liveEmit();
-                        // and then every 5 seconds after that
-                        liveInterval = setInterval(async () => await liveEmit(), 5000);
-                    }
-                }
-            });
-
             if(session.botId !== undefined) {
                 const bot = cmInstances.find(x => x.getName() === session.botId);
                 if(bot !== undefined) {
@@ -1456,12 +1304,8 @@ const webClient = async (options: OperatorConfig) => {
 
                     // only setup streams if the user can actually access them (not just a web operator)
                     if(session.authBotId !== undefined) {
-                        // streaming logs and stats from client
+                        // streaming stats from client
                         const newStreams: (AbortController | NodeJS.Timeout)[] = [];
-                        const ac = startLogStream(session, user);
-                        if(ac !== undefined) {
-                            newStreams.push(ac);
-                        }
                         const interval = setInterval(async () => {
                             try {
                                 const resp = await got.get(`${bot.normalUrl}/stats`, {

src/Web/Server/routes/authenticated/user/addBot.ts

@@ -31,13 +31,13 @@ const addBot = () => {
             const {
                 bots: botsFromConfig = []
             } = req.botApp.fileConfig.document.toJS();
-            if(botsFromConfig.length === 0 || botsFromConfig.some(x => x.name !== botData.name)) {
+            if (botsFromConfig.length === 0 || !botsFromConfig.some(x => x.name === botData.name)) {
                 req.botApp.logger.warn('Overwriting existing bot with the same name BUT this bot does not exist in the operator CONFIG FILE. You should check how you have provided config before next start or else this bot may be started twice (once from file, once from arg/env)');
-
             }
 
             await existingBot.destroy('system');
-            req.botApp.bots.filter(x => x.botAccount !== botData.name);
+            const existingBotIndex = req.botApp.bots.findIndex(x => x.botAccount === botData.name);
+            req.botApp.bots.splice(existingBotIndex, 1);
         }
 
         req.botApp.fileConfig.document.addBot(botData);
@@ -54,14 +54,9 @@ const addBot = () => {
                 result.error = newBot.error;
                 return res.status(500).json(result);
             }
-            await newBot.testClient();
-            await newBot.buildManagers();
-            newBot.runManagers('system').catch((err) => {
-                req.botApp.logger.error(`Unexpected error occurred while running Bot ${newBot.botName}. Bot must be re-built to restart`);
-                if (!err.logged || !(err instanceof LoggedError)) {
-                    req.botApp.logger.error(err);
-                }
-            });
+            await newBot.init();
+            // return response early so client doesn't have to wait for all managers to be built
+            res.json(result);
         } catch (err: any) {
             result.success = false;
             if (newBot.error === undefined) {
@@ -73,7 +68,22 @@ const addBot = () => {
                 req.botApp.logger.error(err);
             }
         }
-        return res.json(result);
+
+        try {
+            await newBot.buildManagers();
+            newBot.runManagers('system').catch((err) => {
+                req.botApp.logger.error(`Unexpected error occurred while running Bot ${newBot.botName}. Bot must be re-built to restart`);
+                if (!err.logged || !(err instanceof LoggedError)) {
+                    req.botApp.logger.error(err);
+                }
+            });
+        } catch (err: any) {
+            req.botApp.logger.error(`Bot ${newBot.botName} cannot recover from this error and must be re-built`);
+            if (!err.logged || !(err instanceof LoggedError)) {
+                req.botApp.logger.error(err);
+            }
+        }
+        return;
     }
     return [...middleware, response];
 }

src/Web/Server/routes/authenticated/user/liveStats.ts

@@ -136,6 +136,9 @@ const liveStats = () => {
                 acc[curr].missPercent = `${formatNumber(per, {toFixed: 0})}%`;
                 acc[curr].identifierAverageHit = formatNumber(acc[curr].identifierAverageHit);
                 acc[curr].averageTimeBetweenHits = formatNumber(acc[curr].averageTimeBetweenHits)
+
+                delete acc[curr].requestTimestamps;
+                delete acc[curr].identifierRequestCount;
                 return acc;
             }, cumRaw);
             const cacheReq = subManagerData.reduce((acc, curr) => acc + curr.stats.cache.totalRequests, 0);

src/Web/Server/routes/authenticated/user/logs.ts

@@ -27,6 +27,15 @@ const logs = () => {
         booleanMiddle([{
             name: 'stream',
             defaultVal: false
+        }, {
+            name: 'formatted',
+            defaultVal: true,
+        }, {
+            name: 'transports',
+            defaultVal: false
+        }, {
+            name: 'streamObjects',
+            defaultVal: false
         }])
     ];
 
@@ -37,20 +46,56 @@ const logs = () => {
         const userName = req.user?.name as string;
         const isOperator = req.user?.isInstanceOperator(req.botApp);
         const realManagers = req.botApp.bots.map(x => req.user?.accessibleSubreddits(x).map(x => x.displayLabel)).flat() as string[];
-        const {level = 'verbose', stream, limit = 200, sort = 'descending', streamObjects = false, formatted = true} = req.query;
+        const {level = 'verbose', stream, limit = 200, sort = 'descending', streamObjects = false, formatted: formattedVal = true, transports: transportsVal = false} = req.query;
+
+        const formatted = formattedVal as boolean;
+        const transports = transportsVal as boolean;
+
+        let bots: Bot[] = [];
+        if(req.serverBot !== undefined) {
+            bots = [req.serverBot];
+        } else {
+            bots = req.user?.accessibleBots(req.botApp.bots) as Bot[];
+        }
+
+        let managers: Manager[] = [];
+
+        if(req.manager !== undefined) {
+            managers = [req.manager];
+        } else {
+            for(const b of bots) {
+                managers = managers.concat(req.user?.accessibleSubreddits(b) as Manager[]);
+            }
+        }
+
+        //const allReq = req.query.subreddit !== undefined && (req.query.subreddit as string).toLowerCase() === 'all';
+
         if (stream) {
+
+            const requestedManagers = managers.map(x => x.displayLabel);
+            const requestedBots = bots.map(x => x.botName);
+
             const origin = req.header('X-Forwarded-For') ?? req.header('host');
             try {
                 logger.stream().on('log', (log: LogInfo) => {
                     if (isLogLineMinLevel(log, level as string)) {
-                        const {subreddit: subName, user} = log;
-                        if (isOperator || (subName !== undefined && (realManagers.includes(subName) || (user !== undefined && user.includes(userName))))) {
+                        const {subreddit: subName, bot, user} = log;
+                        let canAccess = false;
+                        if(user !== undefined && user.includes(userName)) {
+                            canAccess = true;
+                        } else if(subName !== undefined || bot !== undefined) {
+                            if(subName === undefined) {
+                                canAccess = requestedBots.includes(bot);
+                            } else {
+                                canAccess = requestedManagers.includes(subName);
+                            }
+                        } else if(isOperator) {
+                            canAccess = true;
+                        }
+
+                        if (canAccess) {
                             if(streamObjects) {
-                                let obj: any = log;
-                                if(!formatted) {
-                                    const {[MESSAGE]: fMessage, ...rest} = log;
-                                    obj = rest;
-                                }
+                                let obj: any = transformLog(log, {formatted, transports});
                                 res.write(`${JSON.stringify(obj)}\r\n`);
                             } else if(formatted) {
                                 res.write(`${log[MESSAGE]}\r\n`)
@@ -74,12 +119,6 @@ const logs = () => {
                 res.destroy();
             }
         } else {
-            let bots: Bot[] = [];
-            if(req.serverBot !== undefined) {
-                bots = [req.serverBot];
-            } else {
-                bots = req.user?.accessibleBots(req.botApp.bots) as Bot[];
-            }
 
             const allReq = req.query.subreddit !== undefined && (req.query.subreddit as string).toLowerCase() === 'all';
 
@@ -114,15 +153,9 @@ const logs = () => {
                 botArr.push({
                     name: b.getBotName(),
                     system: systemLogs,
-                    all: formatted ? allLogs.map(x => {
-                        const {[MESSAGE]: fMessage, ...rest} = x;
-                        return {...rest, formatted: fMessage};
-                    }) : allLogs,
+                    all: allLogs.map(x => transformLog(x, {formatted, transports })),
                     subreddits: allReq ? [] : [...managerLogs.entries()].reduce((acc: any[], curr) => {
-                        const l = formatted ? curr[1].map(x => {
-                            const {[MESSAGE]: fMessage, ...rest} = x;
-                            return {...rest, formatted: fMessage};
-                            }) : curr[1];
+                        const l = curr[1].map(x => transformLog(x, {formatted, transports }));
                         acc.push({name: curr[0], logs: l});
                         return acc;
                     }, [])
@@ -135,4 +168,23 @@ const logs = () => {
     return [...middleware, response];
 }
 
+const transformLog = (obj: LogInfo, options: { formatted: boolean, transports: boolean }) => {
+    const {
+        [MESSAGE]: fMessage,
+        transport,
+        //@ts-ignore
+        name, // name is the name of the last transport
+        ...rest
+    } = obj;
+    const transformed: any = rest;
+    if (options.formatted) {
+        transformed.formatted = fMessage;
+    }
+    if (options.transports) {
+        transformed.transport = transport;
+        transformed.name = name;
+    }
+    return transformed;
+}
+
 export default logs;

src/Web/Server/routes/authenticated/user/status.ts

@@ -66,13 +66,13 @@ const status = () => {
 
         const subManagerData = [];
         for (const m of req.user?.accessibleSubreddits(bot) as Manager[]) {
-            const logs = req.manager === undefined || allReq || req.manager.getDisplay() === m.getDisplay() ? filterLogs(m.logs, {
-                    level: (level as string),
-                    // @ts-ignore
-                    sort,
-                    limit: limit as string,
-                    returnType: 'object'
-                }) as LogInfo[]: [];
+            // const logs = req.manager === undefined || allReq || req.manager.getDisplay() === m.getDisplay() ? filterLogs(m.logs, {
+            //         level: (level as string),
+            //         // @ts-ignore
+            //         sort,
+            //         limit: limit as string,
+            //         returnType: 'object'
+            //     }) as LogInfo[]: [];
 
             let retention = 'Unknown';
             if (m.resources !== undefined) {
@@ -89,7 +89,7 @@ const status = () => {
             const sd = {
                 name: m.displayLabel,
                 //linkName: s.replace(/\W/g, ''),
-                logs: logs || [], // provide a default empty value in case we truly have not logged anything for this subreddit yet
+                logs: [], // provide a default empty value in case we truly have not logged anything for this subreddit yet
                 botState: m.managerState,
                 eventsState: m.eventsState,
                 queueState: m.queueState,
@@ -237,14 +237,14 @@ const status = () => {
         const sharedSub = subManagerData.find(x => x.stats.cache.isShared);
         const sharedCount = sharedSub !== undefined ? sharedSub.stats.cache.currentKeyCount : 0;
         const scopes = req.user?.isInstanceOperator(bot) ? bot.client.scope : [];
-        const allSubLogs = subManagerData.map(x => x.logs).flat().sort(logSortFunc(sort as string)).slice(0, (limit as number) + 1);
-        const allLogs = filterLogs([...allSubLogs, ...(req.user?.isInstanceOperator(req.botApp) ? bot.logs : bot.logs.filter(x => x.user === req.user?.name))], {
-            level: (level as string),
-            // @ts-ignore
-            sort,
-            limit: limit as string,
-            returnType: 'object'
-        }) as LogInfo[];
+        //const allSubLogs = subManagerData.map(x => x.logs).flat().sort(logSortFunc(sort as string)).slice(0, (limit as number) + 1);
+        // const allLogs = filterLogs([...allSubLogs, ...(req.user?.isInstanceOperator(req.botApp) ? bot.logs : bot.logs.filter(x => x.user === req.user?.name))], {
+        //     level: (level as string),
+        //     // @ts-ignore
+        //     sort,
+        //     limit: limit as string,
+        //     returnType: 'object'
+        // }) as LogInfo[];
         let allManagerData: any = {
             name: 'All',
             status: bot.running ? 'RUNNING' : 'NOT RUNNING',
@@ -261,7 +261,7 @@ const status = () => {
                 causedBy: SYSTEM
             },
             dryRun: boolToString(bot.dryRun === true),
-            logs: allLogs,
+            logs: [],
             checks: checks,
             softLimit: bot.softLimit,
             hardLimit: bot.hardLimit,

src/Web/Server/server.ts

@@ -41,6 +41,12 @@ const server = addAsync(express());
 server.use(bodyParser.json());
 server.use(bodyParser.urlencoded({extended: false}));
 
+server.use((req, res, next) => {
+    // https://developers.google.com/search/docs/advanced/crawling/block-indexing#http-response-header
+    res.setHeader('X-Robots-Tag', 'noindex');
+    next();
+});
+
 declare module 'express-session' {
     interface SessionData {
         user: string,
@@ -79,7 +85,9 @@ const rcbServer = async function (options: OperatorConfigWithFileContext) {
 
         if(botName === undefined && subName === undefined) {
             sysLogs.unshift(log);
-            sysLogs = sysLogs.slice(0, 201);
+            if(sysLogs.length > 200) {
+                sysLogs.splice(200);
+            }
         }
     })
 

src/Web/assets/browser.js

@@ -0,0 +1,5 @@
+const logform = require('logform');
+const tripleBeam = require('triple-beam');
+
+window.format = logform.format;
+window.beam = tripleBeam;

src/Web/assets/public/TextEncoderStream.js

@@ -0,0 +1,160 @@
+// Copyright 2016 Google Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Polyfill for TextEncoderStream and TextDecoderStream
+
+(function() {
+    'use strict';
+
+    if (typeof self.TextEncoderStream === 'function' &&
+        typeof self.TextDecoderStream === 'function') {
+        // The constructors exist. Assume that they work and don't replace them.
+        return;
+    }
+
+    if (typeof self.TextEncoder !== 'function') {
+        throw new ReferenceError('TextEncoder implementation required');
+    }
+
+    if (typeof self.TextDecoder !== 'function') {
+        throw new ReferenceError('TextDecoder implementation required');
+    }
+
+    // These symbols end up being different for every realm, so mixing objects
+    // created in one realm with methods created in another fails.
+    const codec = Symbol('codec');
+    const transform = Symbol('transform');
+
+    class TextEncoderStream {
+        constructor() {
+            this[codec] = new TextEncoder();
+            this[transform] =
+                new TransformStream(new TextEncodeTransformer(this[codec]));
+        }
+    }
+
+    class TextDecoderStream {
+        constructor(label = undefined, options = undefined) {
+            this[codec] = new TextDecoder(label, options);
+            this[transform] = new TransformStream(
+                new TextDecodeTransformer(this[codec]));
+        }
+    }
+
+    // ECMAScript class syntax will create getters that are non-enumerable, but we
+    // need them to be enumerable in WebIDL-style, so we add them manually.
+    // "readable" and "writable" are always delegated to the TransformStream
+    // object. Properties specified in |properties| are delegated to the
+    // underlying TextEncoder or TextDecoder.
+    function addDelegatingProperties(prototype, properties) {
+        for (const transformProperty of ['readable', 'writable']) {
+            addGetter(prototype, transformProperty, function() {
+                return this[transform][transformProperty];
+            });
+        }
+
+        for (const codecProperty of properties) {
+            addGetter(prototype, codecProperty, function() {
+                return this[codec][codecProperty];
+            });
+        }
+    }
+
+    function addGetter(prototype, property, getter) {
+        Object.defineProperty(prototype, property,
+            {
+                configurable: true,
+                enumerable: true,
+                get: getter
+            });
+    }
+
+    addDelegatingProperties(TextEncoderStream.prototype, ['encoding']);
+    addDelegatingProperties(TextDecoderStream.prototype,
+        ['encoding', 'fatal', 'ignoreBOM']);
+
+    class TextEncodeTransformer {
+        constructor() {
+            this._encoder = new TextEncoder();
+            this._carry = undefined;
+        }
+
+        transform(chunk, controller) {
+            chunk = String(chunk);
+            if (this._carry !== undefined) {
+                chunk = this._carry + chunk;
+                this._carry = undefined;
+            }
+            const terminalCodeUnit = chunk.charCodeAt(chunk.length - 1);
+            if (terminalCodeUnit >= 0xD800 && terminalCodeUnit < 0xDC00) {
+                this._carry = chunk.substring(chunk.length - 1);
+                chunk = chunk.substring(0, chunk.length - 1);
+            }
+            const encoded = this._encoder.encode(chunk);
+            if (encoded.length > 0) {
+                controller.enqueue(encoded);
+            }
+        }
+
+        flush(controller) {
+            if (this._carry !== undefined) {
+                controller.enqueue(this._encoder.encode(this._carry));
+                this._carry = undefined;
+            }
+        }
+    }
+
+    class TextDecodeTransformer {
+        constructor(decoder) {
+            this._decoder = new TextDecoder(decoder.encoding, {
+                fatal: decoder.fatal,
+                ignoreBOM: decoder.ignoreBOM
+            });
+        }
+
+        transform(chunk, controller) {
+            const decoded = this._decoder.decode(chunk, {stream: true});
+            if (decoded != '') {
+                controller.enqueue(decoded);
+            }
+        }
+
+        flush(controller) {
+            // If {fatal: false} is in options (the default), then the final call to
+            // decode() can produce extra output (usually the unicode replacement
+            // character 0xFFFD). When fatal is true, this call is just used for its
+            // side-effect of throwing a TypeError exception if the input is
+            // incomplete.
+            var output = this._decoder.decode();
+            if (output !== '') {
+                controller.enqueue(output);
+            }
+        }
+    }
+
+    function exportAs(name, value) {
+        // Make it stringify as [object <name>] rather than [object Object].
+        value.prototype[Symbol.toStringTag] = name;
+        Object.defineProperty(self, name,
+            {
+                configurable: true,
+                enumerable: false,
+                writable: true,
+                value
+            });
+    }
+
+    exportAs('TextEncoderStream', TextEncoderStream);
+    exportAs('TextDecoderStream', TextDecoderStream);
+})();

src/Web/assets/public/logUtils.js

@@ -0,0 +1,120 @@
+const SPLAT = window.beam.SPLAT;
+const {combine, printf, timestamp, label, splat, errors} = window.format;
+
+window.formattedTime = (short, full) => `<span class="has-tooltip"><span style="margin-top:35px" class='tooltip rounded shadow-lg p-1 bg-gray-100 text-black space-y-3 p-2 text-left'>${full}</span><span>${short}</span></span>`;
+window.formatLogLineToHtml = (log, timestamp = undefined) => {
+    const val = typeof log === 'string' ? log : log[window.beam.MESSAGE];
+    const logContent = Autolinker.link(val, {
+        email: false,
+        phone: false,
+        mention: false,
+        hashtag: false,
+        stripPrefix: false,
+        sanitizeHtml: true,
+    })
+        .replace(/(\s*debug\s*):/i, '<span class="debug blue">$1</span>:')
+        .replace(/(\s*warn\s*):/i, '<span class="warn yellow">$1</span>:')
+        .replace(/(\s*info\s*):/i, '<span class="info green">$1</span>:')
+        .replace(/(\s*error\s*):/i, '<span class="error red">$1</span>:')
+        .replace(/(\s*verbose\s*):/i, '<span class="error purple">$1</span>:')
+        .replaceAll('\n', '<br />');
+    //.replace(HYPERLINK_REGEX, '<a target="_blank" href="$&">$&</a>');
+    let line;
+
+    let timestampString = timestamp;
+    if(timestamp === undefined && typeof log !== 'string') {
+        timestampString = log.timestamp;
+    }
+
+    if(timestampString !== undefined) {
+        const timeStampReplacement = formattedTime(dayjs(timestampString).format('HH:mm:ss z'), timestampString);
+        const splitLine = logContent.split(timestampString);
+        line = `<div class="logLine">${splitLine[0]}${timeStampReplacement}<span style="white-space: pre-wrap">${splitLine[1]}</span></div>`;
+    } else {
+        line = `<div style="white-space: pre-wrap" class="logLine">${logContent}</div>`
+    }
+    return line;
+}
+
+window.formatNumber = (val, options) => {
+    const {
+        toFixed = 2,
+        defaultVal = null,
+        prefix = '',
+        suffix = '',
+        round,
+    } = options || {};
+    let parsedVal = typeof val === 'number' ? val : Number.parseFloat(val);
+    if (Number.isNaN(parsedVal)) {
+        return defaultVal;
+    }
+    let prefixStr = prefix;
+    const {enable = false, indicate = true, type = 'round'} = round || {};
+    if (enable && !Number.isInteger(parsedVal)) {
+        switch (type) {
+            case 'round':
+                parsedVal = Math.round(parsedVal);
+                break;
+            case 'ceil':
+                parsedVal = Math.ceil(parsedVal);
+                break;
+            case 'floor':
+                parsedVal = Math.floor(parsedVal);
+        }
+        if (indicate) {
+            prefixStr = `~${prefix}`;
+        }
+    }
+    const localeString = parsedVal.toLocaleString(undefined, {
+        minimumFractionDigits: toFixed,
+        maximumFractionDigits: toFixed,
+    });
+    return `${prefixStr}${localeString}${suffix}`;
+};
+logFormatter = printf(({
+                                   level,
+                                   message,
+                                   labels = ['App'],
+                                   subreddit,
+                                   bot,
+                                   instance,
+                                   leaf,
+                                   itemId,
+                                   timestamp,
+                                   durationMs,
+                                   // @ts-ignore
+                                   [SPLAT]: splatObj,
+                                   stack,
+                                   ...rest
+                               }) => {
+    let stringifyValue = splatObj !== undefined ? JSON.stringify(splatObj) : '';
+    let msg = message;
+    let stackMsg = '';
+    if (stack !== undefined) {
+        const stackArr = stack.split('\n');
+        const stackTop = stackArr[0];
+        const cleanedStack = stackArr
+            .slice(1) // don't need actual error message since we are showing it as msg
+            .join('\n'); // rejoin with newline to preserve formatting
+        stackMsg = `\n${cleanedStack}`;
+        if (msg === undefined || msg === null || typeof message === 'object') {
+            msg = stackTop;
+        } else {
+            stackMsg = `\n${stackTop}${stackMsg}`
+        }
+    }
+
+    let nodes = labels;
+    if (leaf !== null && leaf !== undefined && !nodes.includes(leaf)) {
+        nodes.push(leaf);
+    }
+    const labelContent = `${nodes.map((x) => `[${x}]`).join(' ')}`;
+
+    return `${timestamp} ${level.padEnd(7)}: ${instance !== undefined ? `|${instance}| ` : ''}${bot !== undefined ? `~${bot}~ ` : ''}${subreddit !== undefined ? `{${subreddit}} ` : ''}${labelContent} ${msg}${durationMs !== undefined ? ` Elapsed: ${durationMs}ms (${window.formatNumber(durationMs/1000)}s) ` : ''}${stringifyValue !== '' ? ` ${stringifyValue}` : ''}${stackMsg}`;
+});
+
+window.formatLog = (logObj) => {
+    const formatted = logFormatter.transform(logObj);
+    const html = window.formatLogLineToHtml(formatted);
+    return {...formatted, html};
+}

src/Web/assets/views/config.ejs

@@ -57,7 +57,9 @@
                     </span>
                 </span>
                 | <input id="configUrl" class="text-black placeholder-gray-500 rounded mx-2" style="min-width:400px;" placeholder="URL of a config to load"/> <a href="#" id="loadConfig">Load</a>
-                <span id="saveTip">
+                <div id="editWrapper" class="my-2">
+                    <label style="display: none" for="reason">Edit Reason</label><input id="reason" class="text-black placeholder-gray-500 rounded mr-2" style="min-width:400px;" placeholder="Edit Reason: Updated through CM Web"/>
+                    <span id="saveTip">
                     <span style="margin-top:30px; z-index:100" class="tooltip rounded shadow-lg p-1 bg-gray-100 text-black -mt-2 space-y-3 p-2 text-left">
                         <div>In order to <strong id="configPageActionType">save</strong> a configuration to a subreddit's wiki page you must re-authorize ContextMod with Reddit to get the following permissions:</div>
                         <ul class="list-inside list-disc" id="reauthPermissions">
@@ -67,7 +69,7 @@
                         <div><b><a href="#" id="doAuthorize">Click Here to re-authorize</a></b></div>
                     </span>
                     <span>
-                        | <a id="doSave">Save</a>
+                        <a id="doSave">Save</a>
                         <svg id="saveQuestionIcon" xmlns="http://www.w3.org/2000/svg"
                              class="h-4 w-4 inline-block cursor-help"
                              fill="none"
@@ -76,6 +78,7 @@
                         </svg>
                     </span>
                 </span>
+                </div>
                 <div id="error" class="font-semibold"></div>
                 <select style="display:none;" id="schema-selection">
                     <option value="bot.yaml">Bot Configuration</option>
@@ -138,12 +141,14 @@
             const saveLink = document.querySelector('#doSave');
             saveLink.classList.remove('isDisabled');
             saveLink.href = '#';
+            document.querySelector('#reason').style.display = 'initial';
         } else {
             document.querySelector('#saveTip').classList.add('has-tooltip');
             document.querySelector('#saveQuestionIcon').style.display = 'initial';
             const saveLink = document.querySelector('#doSave');
             saveLink.classList.add('isDisabled');
             saveLink.href = '';
+            document.querySelector('#reason').style.display = 'none';
         }
     }
 
@@ -196,18 +201,24 @@
             return;
         }
 
+        const payload = {
+            location: window.wikiLocation,
+            create: window.creatingWikiPage,
+            data: window.ed.getModel().getValue(),
+        };
+
+        const reasonVal = document.querySelector('#reason').value;
+        if(reasonVal.trim() !== '') {
+            payload.reason = reasonVal;
+        }
+
         fetch(`${document.location.origin}/config${document.location.search}`, {
             method: 'POST',
             headers: {
                 'Accept': 'application/json',
                 'Content-Type': 'application/json'
             },
-            body: JSON.stringify({
-                //...data,
-                location: window.wikiLocation,
-                create: window.creatingWikiPage,
-                data: window.ed.getModel().getValue()
-            })
+            body: JSON.stringify(payload)
         }).then((resp) => {
             if (!resp.ok) {
                 resp.text().then(data => {
@@ -217,6 +228,9 @@
                 if(window.creatingWikiPage) {
                     window.isCreatingWikiPage(false);
                 }
+
+                document.querySelector('#reason').value = '';
+
                 document.querySelector('#error').innerHTML = `Wiki saved!`;
                 window.dirty = false;
                 setTimeout(() => {
@@ -228,39 +242,64 @@
 </script>
 <script defer="defer" src="/public/yaml/entry.js"></script>
 <script>
-    const gistReg = new RegExp(/.*gist.github.com\/.+\/(.+)/i)
+    const GIST_RAW_REGEX = new RegExp(/.*gist\.github\.com\/(.+)\/(.+)\/raw\/.+/i)
+    // match[1] = username
+    // match[2] = gistId
+    // match[3] = filename (optional)
+    const gistReg = new RegExp(/.*gist\.github\.com\/(.+)\/([^#\/]+)(?:#file-(.+))?/i)
     const blobReg = new RegExp(/.*github.com\/(.+)\/(.+)\/blob\/(.+)/i);
 
+    const normalizeGistFileKey = (val) => val.replaceAll(/[^\w\d]/g, '').toLowerCase().trim();
+
     function getPayloadUrl(url) {
         console.debug(`Attempting to detect resolvable URL for ${url}`);
-        let match = url.match(gistReg);
-        if (match !== null) {
-            const gistApiUrl = `https://api.github.com/gists/${match[1]}`;
-            console.debug(`Looks like a non-raw gist URL! Trying to resolve ${gistApiUrl}`);
-
+        let match = url.match(GIST_RAW_REGEX);
+        if(match !== null) {
             return new Promise((resolve, reject) => {
-                fetch(gistApiUrl).then((resp) => {
-                    if (!resp.ok) {
-                        console.error('Response was not OK from Gist API');
-                        resolve(url);
-                    } else {
-                        resp.json().then((data) => {
-                            // get first found file
-                            const fileKeys = Object.keys(data.files);
-                            if (fileKeys.length === 0) {
-                                console.error(`No files found in gist!`);
-                            } else {
-                                if (fileKeys.length > 1) {
-                                    console.warn(`More than one file found in gist! Using first found: ${fileKeys[0]}`);
+                // need to use CDN url or else we get a CORS policy error
+                resolve(url.replace('gist.github.com', 'gist.githubusercontent.com'));
+            })
+        } else {
+            match = url.match(gistReg);
+            if (match !== null) {
+                const gistApiUrl = `https://api.github.com/gists/${match[2]}`;
+                console.debug(`Looks like a non-raw gist URL! Trying to resolve ${gistApiUrl}`);
+
+                return new Promise((resolve, reject) => {
+                    fetch(gistApiUrl).then((resp) => {
+                        if (!resp.ok) {
+                            console.error('Response was not OK from Gist API');
+                            resolve(url);
+                        } else {
+                            resp.json().then((data) => {
+                                // get first found file
+                                const fileKeys = Object.keys(data.files);
+                                if (fileKeys.length === 0) {
+                                    console.error(`No files found in gist!`);
+                                } else {
+                                    let fileKey = fileKeys[0];
+                                    if (fileKeys.length > 1) {
+                                        if(match[3] !== undefined) {
+                                            //const normalizedFileName = normalizeGistFileKey(match.named.fileName.replace('/^file-/', ''));
+                                            const normalizedFileName = normalizeGistFileKey(match[3]);
+                                            const matchingKey = fileKeys.find(x => normalizeGistFileKey(x) === normalizedFileName);
+                                            if(matchingKey === undefined) {
+                                                console.error(`Found Gist ${match[2]} but it did not contain a file named ${match[3]}`);
+                                            }
+                                            fileKey = matchingKey;
+                                        } else {
+                                            console.warn(`More than one file found in gist but URL did not specify a filename! Using first found: ${fileKey}`);
+                                        }
+                                    }
+                                    const rawUrl = data.files[fileKey].raw_url;
+                                    console.debug(`Resolving raw gist url for file found (${fileKey}) to ${rawUrl}`);
+                                    resolve(rawUrl);
                                 }
-                                const rawUrl = data.files[fileKeys[0]].raw_url;
-                                console.debug(`Resolving raw gist url for first file found (${fileKeys[0]}) to ${rawUrl}`);
-                                resolve(rawUrl);
-                            }
-                        });
-                    }
-                })
-            });
+                            });
+                        }
+                    })
+                });
+            }
         }
 
         match = url.match(blobReg);

src/Web/assets/views/partials/head.ejs

@@ -8,5 +8,8 @@
     <meta charset="utf-8">
     <meta http-equiv="X-UA-Compatible" content="IE=edge">
     <meta name="viewport" content="width=device-width,initial-scale=1.0">
+    <!-- This is internal tool or at the very least a private site, should not be indexed -->
+    <!-- https://developers.google.com/search/docs/advanced/crawling/block-indexing#meta-tag -->
+    <meta name="robots" content="noindex">
     <!--icons from https://heroicons.com -->
 </head>

src/Web/assets/views/status.ejs

@@ -672,41 +672,10 @@
     dayjs.extend(window.dayjs_plugin_duration)
     dayjs.extend(window.dayjs_plugin_relativeTime)
     dayjs.extend(window.dayjs_plugin_isSameOrAfter)
-    window.formattedTime = (short, full) => `<span class="has-tooltip"><span style="margin-top:35px" class='tooltip rounded shadow-lg p-1 bg-gray-100 text-black space-y-3 p-2 text-left'>${full}</span><span>${short}</span></span>`;
-    window.formatLogLineToHtml = (log, timestamp = undefined) => {
-        const val = typeof log === 'string' ? log : log['MESSAGE'];
-        const logContent = Autolinker.link(val, {
-            email: false,
-            phone: false,
-            mention: false,
-            hashtag: false,
-            stripPrefix: false,
-            sanitizeHtml: true,
-        })
-            .replace(/(\s*debug\s*):/i, '<span class="debug blue">$1</span>:')
-            .replace(/(\s*warn\s*):/i, '<span class="warn yellow">$1</span>:')
-            .replace(/(\s*info\s*):/i, '<span class="info green">$1</span>:')
-            .replace(/(\s*error\s*):/i, '<span class="error red">$1</span>:')
-            .replace(/(\s*verbose\s*):/i, '<span class="error purple">$1</span>:')
-            .replaceAll('\n', '<br />');
-        //.replace(HYPERLINK_REGEX, '<a target="_blank" href="$&">$&</a>');
-        let line = '';
-
-        let timestampString = timestamp;
-        if(timestamp === undefined && typeof log !== 'string') {
-            timestampString = log.timestamp;
-        }
-
-        if(timestampString !== undefined) {
-            const timeStampReplacement = formattedTime(dayjs(timestampString).format('HH:mm:ss z'), timestampString);
-            const splitLine = logContent.split(timestampString);
-            line = `<div class="logLine">${splitLine[0]}${timeStampReplacement}<span style="white-space: pre-wrap">${splitLine[1]}</span></div>`;
-        } else {
-            line = `<div style="white-space: pre-wrap" class="logLine">${logContent}</div>`
-        }
-        return line;
-    }
 </script>
+<script src="public/TextEncoderStream.js"></script>
+<script src="public/browserBundle.js"></script>
+<script src="public/logUtils.js"></script>
 <script>
     window.sort = 'desc';
 
@@ -878,46 +847,370 @@
                 history.pushState(null, '', newRelativePathQuery);
             }
             const activeSub = document.querySelector(`[data-subreddit="${subreddit}"][data-bot="${bot}"].sub`);
-            if(!activeSub.classList.contains('seen')) {
-                //firstSub.classList.add('seen');
-
-                //subreddit = firstSub.dataset.subreddit;
-                //bot = subSection.dataset.bot;
-                level = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="level"]`).value;
-                sort = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="sort"]`).value;
-                limitSel = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="limit"]`).value;
-
-                fetch(`/api/logs?instance=<%= instanceId %>&bot=${bot}&subreddit=${subreddit}&level=${level}&sort=${sort}&limit=${limitSel}&stream=false&formatted=true`).then((resp) => {
-                    if (!resp.ok) {
-                        console.error('Response was not OK from logs GET');
-                    } else {
-                        resp.json().then((data) => {
-                            const logContainer = document.querySelector(`[data-subreddit="${subreddit}"] .logs`);
-                            const logLines = (subreddit.toLowerCase() === 'all' ? data[0].all : data[0].subreddits[0].logs).map(x => {
-                                let fString = x.formatted;
-                                if(x.bot !== undefined) {
-                                    fString = fString.replace(`~${x.bot}~ `, '');
-                                }
-                                if(x.subreddit !== undefined && subreddit !== 'All') {
-                                    fString = fString.replace(`{${x.subreddit}} `, '');
-                                }
-                                return window.formatLogLineToHtml(fString, x.timestamp)
-                            }).join('');
-                            logContainer.insertAdjacentHTML('afterbegin', logLines);
-                            activeSub.classList.add('seen');
-                        });
-                    }
-                }).catch((err) => {
-                    console.log(err);
-                });
-
-            }
             if(window.socket !== undefined) {
                 window.socket.emit('viewing', {bot, subreddit});
             }
         });
     });
 
+    let recentlySeen = new Map();
+
+
+    function getLogBlock(bot, subreddit) {
+
+        console.debug(`Getting initial logs for ${bot} ${subreddit}`);
+
+        level = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="level"]`).value;
+        sort = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="sort"]`).value;
+        limitSel = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="limit"]`).value;
+
+        return new Promise((resolve, reject) => {
+            fetch(`/api/logs?instance=<%= instanceId %>&bot=${bot}&subreddit=${subreddit}&level=${level}&sort=${sort}&limit=${limitSel}&stream=false&formatted=false`).then((resp) => {
+                if (!resp.ok) {
+                    console.error('Response was not OK from logs GET');
+                    reject('Response was not OK from logs GET');
+                } else {
+                    resp.json().then((data) => {
+                        const logContainer = document.querySelector(`.sub[data-bot="${bot}"] .logs[data-subreddit="${subreddit}"]`);
+                        const logLines = (subreddit.toLowerCase() === 'all' ? data[0].all : data[0].subreddits[0].logs).map(x => {
+                            const logObj = window.formatLog(x);
+                            let fString = logObj[window.beam.MESSAGE];
+                            if(x.bot !== undefined) {
+                                fString = fString.replace(`~${x.bot}~ `, '');
+                            }
+                            if(x.subreddit !== undefined && subreddit !== 'All') {
+                                fString = fString.replace(`{${x.subreddit}} `, '');
+                            }
+                            return window.formatLogLineToHtml(fString, x.timestamp)
+                        }).join('');
+                        logContainer.insertAdjacentHTML('afterbegin', logLines);
+                        console.debug(`Done with initial logs for ${bot} ${subreddit}`);
+                        resolve();
+                        //activeSub.classList.add('seen');
+                    });
+                }
+            }).catch((err) => {
+                console.log(err);
+                reject(err);
+            });
+        });
+    }
+
+    // https://stackoverflow.com/a/66394121/1469797
+    function onVisible(element, callback) {
+        new IntersectionObserver((entries, observer) => {
+            entries.forEach(entry => {
+                if(entry.intersectionRatio > 0) {
+                    callback(element);
+                    //observer.disconnect();
+                }
+            });
+        }).observe(element);
+    }
+
+    function getStreamingLogs(sub, bot) {
+
+        console.debug(`Getting stream for ${bot} ${sub}`);
+
+        level = document.querySelector(`[data-subreddit="${sub}"] [data-type="level"]`).value;
+        sort = document.querySelector(`[data-subreddit="${sub}"] [data-type="sort"]`).value;
+        limitSel = document.querySelector(`[data-subreddit="${sub}"] [data-type="limit"]`).value;
+
+        const logContainer = document.querySelector(`.sub[data-bot="${bot}"] .logs[data-subreddit="${sub}"]`);
+
+        let textBuffer = '';
+
+        var controller = new AbortController();
+        var signal = controller.signal;
+
+        let lastFlush;
+        let bufferTimeout;
+
+        let bufferedLogs = [];
+
+        const formattedMsg = (x) => {
+            const logObj = window.formatLog(x);
+            let fString = logObj[window.beam.MESSAGE];
+            if(x.bot !== undefined) {
+                fString = fString.replace(`~${x.bot}~ `, '');
+            }
+            if(x.subreddit !== undefined && sub !== 'All') {
+                fString = fString.replace(`{${x.subreddit}} `, '');
+            }
+            return window.formatLogLineToHtml(fString, x.timestamp);
+        }
+
+        const flushLogs = () => {
+            let existingLogs;
+
+            //const el = document.querySelector(`[data-subreddit="${sub}"][data-bot="${bot}"].sub`);
+            //const logContainer = el.querySelector(`.logs`);
+
+            if(window.sort === 'desc' || window.sort === 'descending') {
+                bufferedLogs.forEach((l) => {
+                    logContainer.insertAdjacentHTML('afterbegin', formattedMsg(l));
+                })
+                existingLogs = Array.from(logContainer.querySelectorAll(`.logLine`));
+                logContainer.replaceChildren(...existingLogs.slice(0, limitSel));
+            } else {
+                bufferedLogs.forEach((l) => {
+                    logContainer.insertAdjacentHTML('beforeend', formattedMsg(l));
+                    existingLogs = Array.from(logContainer.querySelectorAll(`.logLine`));
+                    const overLimit = limitSel - existingLogs.length;
+                    logContainer.replaceChildren(...existingLogs.slice(overLimit -1, limitSel));
+                })
+            }
+            lastFlush = Date.now();
+            bufferedLogs = [];
+        }
+
+        const fetchPromise = fetch(`/api/logs?instance=<%= instanceId %>&bot=${bot}&subreddit=${sub}&level=${level}&sort=${sort}&limit=${limitSel}&stream=true&streamObjects=true&formatted=false`, {signal})
+            .then(response => response.body)
+            .then(rs =>
+                rs.pipeThrough(new TextDecoderStream())
+                    .pipeThrough(new TransformStream({
+                        transform(chunk, controller) {
+                            textBuffer += chunk;
+                            const lines = textBuffer.split('\n');
+                            for (const line of lines.slice(0, -1)) {
+                                controller.enqueue(line);
+                            }
+                            textBuffer = lines.slice(-1)[0];
+                        },
+                        flush(controller) {
+                            if (textBuffer) {
+                                controller.enqueue(textBuffer);
+                            }
+                        }
+                    }))
+
+                    // Parse JSON objects
+                    .pipeThrough(new TransformStream({
+                        transform(line, controller) {
+                            if (line) {
+                                controller.enqueue(
+                                    JSON.parse(line)
+                                );
+                            }
+                        }
+                    }))
+            ).catch((e) => {
+                console.warn(e);
+            });
+
+        fetchPromise.then(async res => {
+            const reader = res.getReader();
+            let keepReading = true;
+            while(keepReading) {
+                const {done, value} = await reader.read();
+                if(done) {
+                    keepReading = false;
+                    console.debug('done');
+                }
+                if(value) {
+                    //console.log(`((Logged For ${bot} ${sub})) ${value.message}`);
+
+                    bufferedLogs.push(value);
+
+                    if(lastFlush !== undefined && bufferTimeout !== undefined && ((Date.now() - lastFlush)/1000) > 3) {
+                        //console.log('Immediate flush');
+                        clearTimeout(bufferTimeout);
+                        bufferTimeout = undefined;
+                        flushLogs();
+                    } else {
+                        //console.log('Using timeout');
+                        clearTimeout(bufferTimeout);
+                        bufferTimeout = setTimeout(() => {flushLogs();}, 1000);
+                    }
+                }
+            }
+            /*                function read() {
+                                reader.read().then(({done, value}) => {
+                                    if(done) {
+                                        console.log('done');
+                                        return;
+                                    }
+                                    if(value) {
+                                        console.log(value);
+                                        read();
+                                    }
+                                });
+                            }
+                            read();*/
+        }).catch((e) => {
+            if(e.name !== 'AbortError') {
+                console.error(e);
+            }
+        });
+
+        const existing = recentlySeen.get(`${bot}.${sub}`) || {};
+        recentlySeen.set(`${bot}.${sub}`, {...existing, fetch: fetchPromise, controller});
+    }
+
+    function updateLiveStats(resp) {
+        let el;
+        let isAll = resp.name.toLowerCase() === 'all';
+        if(isAll) {
+            // got all
+            el = document.querySelector(`[data-subreddit="All"][data-bot="${resp.bot}"].sub`);
+        } else {
+            // got subreddit
+            el = document.querySelector(`[data-subreddit="${resp.name}"].sub`);
+        }
+
+        if(resp.system.running && el.classList.contains('offline')) {
+            el.classList.remove('offline');
+        } else if(!resp.system.running && !el.classList.contains('offline')) {
+            el.classList.add('offline');
+        }
+
+        el.querySelector('.runningActivities').innerHTML = resp.runningActivities;
+        el.querySelector('.queuedActivities').innerHTML = resp.queuedActivities;
+        el.querySelector('.delayedItemsCount').innerHTML = resp.delayedItems.length;
+        el.querySelector('.delayedItemsList').innerHTML = 'No delayed Items!';
+        if(resp.delayedItems.length > 0) {
+            el.querySelector('.delayedItemsList').innerHTML = '';
+            const now = dayjs();
+            const sorted = resp.delayedItems.map(x => ({...x, queuedAtUnix: x.queuedAt, queuedAt: dayjs.unix(x.queuedAt), dispatchAt: dayjs.unix(x.queuedAt + x.duration)}));
+            sorted.sort((a, b) => {
+                return a.dispatchAt.isSameOrAfter(b.dispatchAt) ? 1 : -1
+            });
+            const delayedItemDivs = sorted.map(x => {
+                const diffUntilNow = x.dispatchAt.diff(now);
+                const durationUntilNow = dayjs.duration(diffUntilNow, 'ms');
+                const queuedAtDisplay = x.queuedAt.format('HH:mm:ss z');
+                const durationDayjs = dayjs.duration(x.duration, 'seconds');
+                const durationDisplay = durationDayjs.humanize();
+                const cancelLink = `<a href="#" data-id="${x.id}" data-subreddit="${x.subreddit}" class="delayCancel">CANCEL</a>`;
+                return `<div>A <a href="https://reddit.com${x.permalink}">${x.submissionId !== undefined ? 'Comment' : 'Submssion'}</a>${isAll ? ` in <a href="https://reddit.com${x.subreddit}">${x.subreddit}</a> ` : ''} by <a href="https://reddit.com/u/${x.author}">${x.author}</a> queued by ${x.source} at ${queuedAtDisplay} for ${durationDisplay} (dispatches ${durationUntilNow.humanize(true)}) -- ${cancelLink}</div>`;
+            });
+            el.querySelector('.delayedItemsList').insertAdjacentHTML('afterbegin', delayedItemDivs.join(''));
+            el.querySelectorAll('.delayedItemsList .delayCancel').forEach(elm => {
+                elm.addEventListener('click', e => {
+                    e.preventDefault();
+                    const id = e.target.dataset.id;
+                    const subreddit = e.target.dataset.subreddit;
+                    fetch(`/api/delayed?instance=<%= instanceId %>&bot=${resp.bot}&subreddit=${subreddit}&id=${id}`, {
+                        method: 'DELETE'
+                    }).then((resp) => {
+                        if (!resp.ok) {
+                            console.error('Response was not OK from delay cancel');
+                        } else {
+                            console.log('Removed ok');
+                        }
+                    });
+                });
+            });
+        }
+
+        el.querySelector('.allStats .eventsCount').innerHTML = resp.stats.historical.eventsCheckedTotal;
+        el.querySelector('.allStats .checksCount').innerHTML = resp.stats.historical.checksTriggeredTotal;
+        el.querySelector('.allStats .rulesCount').innerHTML = resp.stats.historical.rulesTriggeredTotal;
+        el.querySelector('.allStats .actionsCount').innerHTML = resp.stats.historical.actionsRunTotal;
+
+        if(isAll) {
+            for(const elm of ['apiAvg','apiLimit','apiDepletion','nextHeartbeat', 'nextHeartbeatHuman', 'limitReset', 'limitResetHuman', 'nannyMode', 'startedAtHuman']) {
+                el.querySelector(`#${elm}`).innerHTML = resp[elm];
+            }
+            el.querySelector(`.botStatus`).innerHTML = resp.system.running ? 'ONLINE' : 'OFFLINE';
+        } else {
+            if(el.querySelector('.modPermissionsCount').innerHTML != resp.permissions.length) {
+                el.querySelector('.modPermissionsCount').innerHTML = resp.permissions.length;
+                el.querySelector('.modPermissionsList').innerHTML = '';
+                el.querySelector('.modPermissionsList').insertAdjacentHTML('afterbegin', resp.permissions.map(x => `<li class="font-mono">${x}</li>`).join(''));
+            }
+
+            for(const elm of ['botState', 'queueState', 'eventsState']) {
+                const state = resp[elm];
+                el.querySelector(`.${elm}`).innerHTML = `${state.state}${state.causedBy === 'system' ? '' : ' (user)'}`;
+            }
+            for(const elm of ['startedAt', 'startedAtHuman', 'wikiLastCheck', 'wikiLastCheckHuman', 'wikiRevision', 'wikiRevisionHuman', 'validConfig', 'delayBy']) {
+                el.querySelector(`.${elm}`).innerHTML = resp[elm];
+            }
+            el.querySelector(`.commentCheckCount`).innerHTML = resp.checks.comments;
+            el.querySelector(`.submissionCheckCount`).innerHTML = resp.checks.submissions;
+
+            const newInner = resp.pollingInfo.map(x => `<li>${x}</li>`).join('');
+            if(el.querySelector(`.pollingInfo`).innerHTML !== newInner) {
+                el.querySelector(`.pollingInfo`).innerHTML = newInner;
+            }
+        }
+    }
+
+    function getLiveStats(bot, sub) {
+        console.debug(`Getting live stats for ${bot} ${sub}`)
+        return fetch(`/api/liveStats?instance=<%= instanceId %>&bot=${bot}&subreddit=${sub}`)
+            .then(response => response.json())
+            .then(resp => updateLiveStats(resp));
+    }
+
+    document.querySelectorAll('.sub').forEach(el => {
+        const sub = el.dataset.subreddit;
+        const bot = el.dataset.bot;
+        //console.log(`Focused on ${bot} ${sub}`);
+        onVisible(el, () => {
+            console.debug(`Focused on ${bot} ${sub}`);
+
+            const identifier = `${bot}.${sub}`;
+
+            recentlySeen.forEach((value, key) => {
+                const {timeout, liveStatsInt, ...rest} = value;
+                if(key === identifier && timeout !== undefined) {
+
+                    console.debug('Clearing timeout on own already set');
+                    clearTimeout(timeout);
+                    recentlySeen.set(key, rest);
+
+                } else if(key !== identifier) {
+
+                    // stop live stats for tabs we are not viewing
+                    clearInterval(liveStatsInt);
+
+                    // set timeout for logs we are not viewing
+                    if(timeout === undefined) {
+                        const t = setTimeout(() => {
+                            const k = key;
+                            const val = recentlySeen.get(k);
+                            if(val !== undefined) {
+                                const {controller} = val;
+                                console.debug(`timeout expired, stopping live data for ${k}`);
+                                if(controller !== undefined) {
+                                    console.debug('Stopping logs');
+                                    controller.abort();
+                                }
+                                // if(liveStatInt !== undefined) {
+                                //     console.log('Stopping live stats');
+                                //     clearInterval(liveStatInt);
+                                // }
+                                recentlySeen.delete(k);
+                            }
+                        },15000);
+                        recentlySeen.set(key, {timeout: t, liveStatsInt, ...rest});
+                    }
+                }
+            });
+
+            if(!recentlySeen.has(identifier)) {
+                getLogBlock(bot, sub).then(() => {
+                    getStreamingLogs(sub, bot);
+                });
+            }
+
+            // always get live stats for tab we just started viewing
+            getLiveStats(bot, sub).then(() => {
+                let liveStatsInt;
+                const liveStatFunc = () => {
+                    getLiveStats(bot, sub).catch(() => {
+                        // stop interval if live stat encounters an error
+                        clearInterval(liveStatsInt);
+                    })
+                };
+                liveStatsInt = setInterval(liveStatFunc, 5000);
+                recentlySeen.set(identifier, {liveStatsInt});
+            });
+        });
+    });
+
 
     var searchParams = new URLSearchParams(window.location.search);
     const shownSub = searchParams.get('sub') || 'All'
@@ -992,85 +1285,9 @@
         let shownBot = searchParams.get('bot');
         window.socket.emit('viewing', {bot: shownBot, subreddit: shownSub});
 
-        socket.on("log", data => {
-            const {
-                subreddit,
-                bot,
-                subredditMessage,
-                allMessage,
-                formattedMessage
-            } = data;
-            if(bot === undefined && subreddit === undefined) {
-                const sys = bufferedBot.get('system');
-                if(sys !== undefined) {
-                    sys.set('All', sys.get('All').concat(formattedMessage));
-                    bufferedBot.set('system', sys);
-                }
-            }
-            if(bot !== undefined) {
-                bufferedBot.set('All', bufferedBot.get('All').concat(allMessage));
+        // TODO web logging
+        // socket.on('log')
 
-                const buffBot = bufferedBot.get(bot) || newBufferedLogs();
-                buffBot.set('All', buffBot.get('All').concat(allMessage));
-                if (subreddit !== undefined) {
-                    buffBot.set(subreddit, (buffBot.get(subreddit) || []).concat(subredditMessage));
-                }
-                bufferedBot.set(bot, buffBot);
-            }
-
-
-            const flushLogs = () => {
-                bufferedBot.forEach((subLogs, botName) => {
-                    if(botName === 'All') {
-                        return;
-                    }
-                    subLogs.forEach((logs, subKey) => {
-                        // check sub exists -- may be a web log
-                        const el = document.querySelector(`[data-subreddit="${subKey}"][data-bot="${botName}"].sub.seen`);
-                        if(null !== el) {
-                            const limit = Number.parseInt(document.querySelector(`[data-subreddit="${subKey}"] [data-type="limit"]`).value);
-                            const logContainer = el.querySelector(`.logs`);
-                            let existingLogs;
-                            if(window.sort === 'desc' || window.sort === 'descending') {
-                                logs.forEach((l) => {
-                                    logContainer.insertAdjacentHTML('afterbegin', l);
-                                })
-                                existingLogs = Array.from(el.querySelectorAll(`.logs .logLine`));
-                                logContainer.replaceChildren(...existingLogs.slice(0, limit));
-                            } else {
-                                logs.forEach((l) => {
-                                    logContainer.insertAdjacentHTML('beforeend', l);
-                                    existingLogs = Array.from(el.querySelectorAll(`.logs .logLine`));
-                                    const overLimit = limit - existingLogs.length;
-                                    logContainer.replaceChildren(...existingLogs.slice(overLimit -1, limit));
-                                })
-                            }
-                        }
-                    });
-                });
-                lastFlush = Date.now();
-                bufferedBot = newBufferedBot();
-                //bufferedLogs = newBufferedLogs();
-                //console.log('Flushed Logs');
-            }
-
-            if(lastFlush !== undefined && bufferTimeout !== undefined && ((Date.now() - lastFlush)/1000) > 3) {
-                //console.log('Immediate flush');
-                clearTimeout(bufferTimeout);
-                bufferTimeout = undefined;
-                flushLogs();
-            } else {
-                //console.log('Using timeout');
-                clearTimeout(bufferTimeout);
-                bufferTimeout = setTimeout(() => {flushLogs();}, 1000);
-            }
-        });
-        socket.on("logClear", data => {
-            data.forEach((obj) => {
-                const n = obj.name === 'all' ? 'All' : obj.name;
-                document.querySelector(`[data-subreddit="${n}"].logs`).innerHTML = obj.logs;
-            })
-        });
         const subIndicators = ['red', 'green', 'yellow'];
         socket.on('opStats', (resp) => {
             for(const b of resp) {
@@ -1100,94 +1317,6 @@
             }
 
         });
-        socket.on('liveStats', (resp) => {
-            let el;
-            let isAll = resp.name.toLowerCase() === 'all';
-            if(isAll) {
-                // got all
-                el = document.querySelector(`[data-subreddit="All"][data-bot="${resp.bot}"].sub`);
-            } else {
-                // got subreddit
-                el = document.querySelector(`[data-subreddit="${resp.name}"].sub`);
-            }
-
-            if(resp.system.running && el.classList.contains('offline')) {
-                el.classList.remove('offline');
-            } else if(!resp.system.running && !el.classList.contains('offline')) {
-                el.classList.add('offline');
-            }
-
-            el.querySelector('.runningActivities').innerHTML = resp.runningActivities;
-            el.querySelector('.queuedActivities').innerHTML = resp.queuedActivities;
-            el.querySelector('.delayedItemsCount').innerHTML = resp.delayedItems.length;
-            el.querySelector('.delayedItemsList').innerHTML = 'No delayed Items!';
-            if(resp.delayedItems.length > 0) {
-                el.querySelector('.delayedItemsList').innerHTML = '';
-                const now = dayjs();
-                const sorted = resp.delayedItems.map(x => ({...x, dispatchAt: dayjs.unix(x.queuedAt + (x.durationMilli))}));
-                sorted.sort((a, b) => {
-                    return a.dispatchAt.isSameOrAfter(b.dispatchAt) ? 1 : -1
-                });
-                const delayedItemDivs = sorted.map(x => {
-                    const diffUntilNow = x.dispatchAt.diff(now)
-                    const durationUntilNow = dayjs.duration(diffUntilNow);
-                    const cancelLink = `<a href="#" data-id="${x.id}" data-subreddit="${x.subreddit}" class="delayCancel">CANCEL</a>`;
-                    return `<div>A <a href="https://reddit.com${x.permalink}">${x.submissionId !== undefined ? 'Comment' : 'Submssion'}</a>${isAll ? ` in <a href="https://reddit.com${x.subreddit}">${x.subreddit}</a> ` : ''} by <a href="https://reddit.com/u/${x.author}">${x.author}</a> queued by ${x.source} at ${dayjs.unix(x.queuedAt).format('HH:mm:ss z')} for ${x.duration} (dispatches in ${durationUntilNow.humanize()}) -- ${cancelLink}</div>`;
-                });
-                el.querySelector('.delayedItemsList').insertAdjacentHTML('afterbegin', delayedItemDivs.join(''));
-                el.querySelectorAll('.delayedItemsList .delayCancel').forEach(elm => {
-                    elm.addEventListener('click', e => {
-                        e.preventDefault();
-                        const id = e.target.dataset.id;
-                        const subreddit = e.target.dataset.subreddit;
-                        fetch(`/api/delayed?instance=<%= instanceId %>&bot=${resp.bot}&subreddit=${subreddit}&id=${id}`, {
-                            method: 'DELETE'
-                        }).then((resp) => {
-                            if (!resp.ok) {
-                                console.error('Response was not OK from delay cancel');
-                            } else {
-                                console.log('Removed ok');
-                            }
-                        });
-                    });
-                });
-            }
-
-            el.querySelector('.allStats .eventsCount').innerHTML = resp.stats.historical.eventsCheckedTotal;
-            el.querySelector('.allStats .checksCount').innerHTML = resp.stats.historical.checksTriggeredTotal;
-            el.querySelector('.allStats .rulesCount').innerHTML = resp.stats.historical.rulesTriggeredTotal;
-            el.querySelector('.allStats .actionsCount').innerHTML = resp.stats.historical.actionsRunTotal;
-
-            if(isAll) {
-                for(const elm of ['apiAvg','apiLimit','apiDepletion','nextHeartbeat', 'nextHeartbeatHuman', 'limitReset', 'limitResetHuman', 'nannyMode', 'startedAtHuman']) {
-                    el.querySelector(`#${elm}`).innerHTML = resp[elm];
-                }
-                el.querySelector(`.botStatus`).innerHTML = resp.system.running ? 'ONLINE' : 'OFFLINE';
-            } else {
-                if(el.querySelector('.modPermissionsCount').innerHTML != resp.permissions.length) {
-                    el.querySelector('.modPermissionsCount').innerHTML = resp.permissions.length;
-                    el.querySelector('.modPermissionsList').innerHTML = '';
-                    el.querySelector('.modPermissionsList').insertAdjacentHTML('afterbegin', resp.permissions.map(x => `<li class="font-mono">${x}</li>`).join(''));
-                }
-
-                for(const elm of ['botState', 'queueState', 'eventsState']) {
-                    const state = resp[elm];
-                    el.querySelector(`.${elm}`).innerHTML = `${state.state}${state.causedBy === 'system' ? '' : ' (user)'}`;
-                }
-                for(const elm of ['startedAt', 'startedAtHuman', 'wikiLastCheck', 'wikiLastCheckHuman', 'wikiRevision', 'wikiRevisionHuman', 'validConfig', 'delayBy']) {
-                    el.querySelector(`.${elm}`).innerHTML = resp[elm];
-                }
-                el.querySelector(`.commentCheckCount`).innerHTML = resp.checks.comments;
-                el.querySelector(`.submissionCheckCount`).innerHTML = resp.checks.submissions;
-
-                const newInner = resp.pollingInfo.map(x => `<li>${x}</li>`).join('');
-                if(el.querySelector(`.pollingInfo`).innerHTML !== newInner) {
-                    el.querySelector(`.pollingInfo`).innerHTML = newInner;
-                }
-            }
-
-            //console.log(resp);
-        });
     });
 
     socket.on('disconnect', () => {

src/util.ts

@@ -1,6 +1,5 @@
 import winston, {Logger} from "winston";
-import jsonStringify from 'safe-stable-stringify';
-import dayjs, {Dayjs, OpUnitType} from 'dayjs';
+import dayjs, {Dayjs} from 'dayjs';
 import {Duration} from 'dayjs/plugin/duration.js';
 import Ajv from "ajv";
 import {InvalidOptionArgumentError} from "commander";
@@ -8,6 +7,7 @@ import {deflateSync, inflateSync} from "zlib";
 import pixelmatch from 'pixelmatch';
 import os from 'os';
 import pathUtil from 'path';
+import {Response} from 'node-fetch';
 import crypto, {createHash} from 'crypto';
 import {
     ActionResult,
@@ -17,7 +17,7 @@ import {
     CheckSummary,
     ImageComparisonResult,
     ItemCritPropHelper,
-    LogInfo,
+    LogInfo, NamedGroup,
     PollingOptionsStrong,
     PostBehaviorOptionConfig,
     RegExResult,
@@ -39,7 +39,7 @@ import redisStore from "cache-manager-redis-store";
 import Autolinker from 'autolinker';
 import {create as createMemoryStore} from './Utils/memoryStore';
 import {LEVEL, MESSAGE} from "triple-beam";
-import {Comment, RedditUser, Submission} from "snoowrap/dist/objects";
+import {Comment, PrivateMessage, RedditUser, Submission, Subreddit} from "snoowrap/dist/objects";
 import reRegExp from '@stdlib/regexp-regexp';
 import fetch from "node-fetch";
 import ImageData from "./Common/ImageData";
@@ -55,10 +55,14 @@ import {RuleResultEntity as RuleResultEntity} from "./Common/Entities/RuleResult
 import {nanoid} from "nanoid";
 import {
     ActivityState,
+    asModLogCriteria,
+    asModNoteCriteria,
     AuthorCriteria,
     authorCriteriaProperties,
     CommentState,
     defaultStrongSubredditCriteriaOptions,
+    ModLogCriteria,
+    ModNoteCriteria,
     StrongSubredditCriteria,
     SubmissionState,
     SubredditCriteria,
@@ -70,15 +74,16 @@ import {
     ActivitySourceTypes,
     CacheProvider,
     ConfigFormat,
-    DurationVal,
+    DurationVal, ExternalUrlContext,
+    ModUserNoteLabel,
+    modUserNoteLabels,
     RedditEntity,
     RedditEntityType,
     statFrequencies,
     StatisticFrequency,
-    StatisticFrequencyOption,
-    StringOperator
+    StatisticFrequencyOption, UrlContext,
+    WikiContext
 } from "./Common/Infrastructure/Atomic";
-import {DurationComparison, GenericComparison} from "./Common/Infrastructure/Comparisons";
 import {
     AuthorOptions,
     FilterCriteriaDefaults,
@@ -110,6 +115,7 @@ import {
     HistoryFiltersOptions
 } from "./Common/Infrastructure/ActivityWindow";
 import {RunnableBaseJson} from "./Common/Infrastructure/Runnable";
+import Snoowrap from "snoowrap";
 
 
 //import {ResembleSingleCallbackComparisonResult} from "resemblejs";
@@ -239,7 +245,7 @@ export const defaultFormat = (defaultLabel = 'App') => printf(({
                                                                    stack,
                                                                    ...rest
                                                                }) => {
-    let stringifyValue = splatObj !== undefined ? jsonStringify(splatObj) : '';
+    let stringifyValue = splatObj !== undefined ? JSON.stringify(splatObj) : '';
     let msg = message;
     let stackMsg = '';
     if (stack !== undefined) {
@@ -258,7 +264,7 @@ export const defaultFormat = (defaultLabel = 'App') => printf(({
     }
 
     let nodes = labels;
-    if (leaf !== null && leaf !== undefined) {
+    if (leaf !== null && leaf !== undefined && !nodes.includes(leaf)) {
         nodes.push(leaf);
     }
     const labelContent = `${nodes.map((x: string) => `[${x}]`).join(' ')}`;
@@ -551,6 +557,8 @@ export const filterCriteriaPropertySummary = <T>(val: FilterCriteriaPropertyResu
         const expectedStrings = crit.map((x: any) => {
             if (asUserNoteCriteria(x)) {
                 return userNoteCriteriaSummary(x);
+            }  else if(asModNoteCriteria(x) || asModLogCriteria(x)) {
+                return modActionCriteriaSummary(x);
             }
             return x;
         }).join(' OR ');
@@ -599,6 +607,9 @@ export const formatNumber = (val: number | string, options?: numberFormatOptions
     if (Number.isNaN(parsedVal)) {
         return defaultVal;
     }
+    if(!Number.isFinite(val)) {
+        return 'Infinite';
+    }
     let prefixStr = prefix;
     const {enable = false, indicate = true, type = 'round'} = round || {};
     if (enable && !Number.isInteger(parsedVal)) {
@@ -709,118 +720,48 @@ export const isActivityWindowConfig = (val: any): val is FullActivityWindowConfi
     return false;
 }
 
-export const comparisonTextOp = (val1: number, strOp: string, val2: number): boolean => {
-    switch (strOp) {
-        case '>':
-            return val1 > val2;
-        case '>=':
-            return val1 >= val2;
-        case '<':
-            return val1 < val2;
-        case '<=':
-            return val1 <= val2;
-        default:
-            throw new Error(`${strOp} was not a recognized operator`);
-    }
-}
-
-const GENERIC_VALUE_COMPARISON = /^\s*(?<opStr>>|>=|<|<=)\s*(?<value>\d+)(?<extra>\s+.*)*$/
-const GENERIC_VALUE_COMPARISON_URL = 'https://regexr.com/60dq4';
-export const parseGenericValueComparison = (val: string): GenericComparison => {
-    const matches = val.match(GENERIC_VALUE_COMPARISON);
-    if (matches === null) {
-        throw new InvalidRegexError(GENERIC_VALUE_COMPARISON, val, GENERIC_VALUE_COMPARISON_URL)
-    }
-    const groups = matches.groups as any;
-
-    return {
-        operator: groups.opStr as StringOperator,
-        value: Number.parseFloat(groups.value),
-        isPercent: false,
-        extra: groups.extra,
-        displayText: `${groups.opStr} ${groups.value}`
-    }
-}
-
-const GENERIC_VALUE_PERCENT_COMPARISON = /^\s*(?<opStr>>|>=|<|<=)\s*(?<value>\d+)\s*(?<percent>%?)(?<extra>.*)$/
-const GENERIC_VALUE_PERCENT_COMPARISON_URL = 'https://regexr.com/60a16';
-export const parseGenericValueOrPercentComparison = (val: string): GenericComparison => {
-    const matches = val.match(GENERIC_VALUE_PERCENT_COMPARISON);
-    if (matches === null) {
-        throw new InvalidRegexError(GENERIC_VALUE_PERCENT_COMPARISON, val, GENERIC_VALUE_PERCENT_COMPARISON_URL)
-    }
-    const groups = matches.groups as any;
-
-    return {
-        operator: groups.opStr as StringOperator,
-        value: Number.parseFloat(groups.value),
-        isPercent: groups.percent !== '',
-        extra: groups.extra,
-        displayText: `${groups.opStr} ${groups.value}${groups.percent === undefined || groups.percent === '' ? '': '%'}`
-    }
-}
-
-export const dateComparisonTextOp = (val1: Dayjs, strOp: StringOperator, val2: Dayjs, granularity?: OpUnitType): boolean => {
-    switch (strOp) {
-        case '>':
-            return val1.isBefore(val2, granularity);
-        case '>=':
-            return val1.isSameOrBefore(val2, granularity);
-        case '<':
-            return val1.isAfter(val2, granularity);
-        case '<=':
-            return val1.isSameOrAfter(val2, granularity);
-        default:
-            throw new Error(`${strOp} was not a recognized operator`);
-    }
-}
-
-const ISO8601_REGEX: RegExp = /^(-?)P(?=\d|T\d)(?:(\d+)Y)?(?:(\d+)M)?(?:(\d+)([DW]))?(?:T(?:(\d+)H)?(?:(\d+)M)?(?:(\d+(?:\.\d+)?)S)?)?$/;
+// string must only contain ISO8601 optionally wrapped by whitespace
+const ISO8601_REGEX: RegExp = /^\s*((-?)P(?=\d|T\d)(?:(\d+)Y)?(?:(\d+)M)?(?:(\d+)([DW]))?(?:T(?:(\d+)H)?(?:(\d+)M)?(?:(\d+(?:\.\d+)?)S)?)?)\s*$/;
+// finds ISO8601 in any part of a string
+const ISO8601_SUBSTRING_REGEX: RegExp = /((-?)P(?=\d|T\d)(?:(\d+)Y)?(?:(\d+)M)?(?:(\d+)([DW]))?(?:T(?:(\d+)H)?(?:(\d+)M)?(?:(\d+(?:\.\d+)?)S)?)?)/g;
+// string must only duration optionally wrapped by whitespace
 const DURATION_REGEX: RegExp = /^\s*(?<time>\d+)\s*(?<unit>days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)\s*$/;
-export const parseDuration = (val: string): Duration => {
-    let matches = val.match(DURATION_REGEX);
-    if (matches !== null) {
-        const groups = matches.groups as any;
-        const dur: Duration = dayjs.duration(groups.time, groups.unit);
-        if (!dayjs.isDuration(dur)) {
-            throw new SimpleError(`Parsed value '${val}' did not result in a valid Dayjs Duration`);
-        }
-        return dur;
+// finds duration in any part of the string
+const DURATION_SUBSTRING_REGEX: RegExp = /(?<time>\d+)\s*(?<unit>days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)/g;
+
+export const parseDurationFromString = (val: string, strict = true): {duration: Duration, original: string}[] => {
+    let matches = parseRegex(strict ? DURATION_REGEX : DURATION_SUBSTRING_REGEX, val);
+    if (matches !== undefined) {
+        return matches.map(x => {
+            const groups = x.named as NamedGroup;
+            const dur: Duration = dayjs.duration(groups.time, groups.unit);
+            if (!dayjs.isDuration(dur)) {
+                throw new SimpleError(`Parsed value '${x.match}' did not result in a valid Dayjs Duration`);
+            }
+            return {duration: dur, original: `${groups.time} ${groups.unit}`};
+        });
     }
-    matches = val.match(ISO8601_REGEX);
-    if (matches !== null) {
-        const dur: Duration = dayjs.duration(val);
-        if (!dayjs.isDuration(dur)) {
-            throw new SimpleError(`Parsed value '${val}' did not result in a valid Dayjs Duration`);
-        }
-        return dur;
+
+    matches = parseRegex(strict ? ISO8601_REGEX : ISO8601_SUBSTRING_REGEX, val);
+    if (matches !== undefined) {
+        return matches.map(x => {
+            const dur: Duration = dayjs.duration(x.groups[0]);
+            if (!dayjs.isDuration(dur)) {
+                throw new SimpleError(`Parsed value '${x.groups[0]}' did not result in a valid Dayjs Duration`);
+            }
+            return {duration: dur, original: x.groups[0]};
+        });
     }
-    throw new InvalidRegexError([DURATION_REGEX, ISO8601_REGEX], val)
+
+    throw new InvalidRegexError([(strict ? DURATION_REGEX : DURATION_SUBSTRING_REGEX), (strict ? ISO8601_REGEX : ISO8601_SUBSTRING_REGEX)], val)
 }
 
-/**
- * Named groups: operator, time, unit
- * */
-const DURATION_COMPARISON_REGEX: RegExp = /^\s*(?<opStr>>|>=|<|<=)\s*(?<time>\d+)\s*(?<unit>days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)\s*$/;
-const DURATION_COMPARISON_REGEX_URL = 'https://regexr.com/609n8';
-export const parseDurationComparison = (val: string): DurationComparison => {
-    const matches = val.match(DURATION_COMPARISON_REGEX);
-    if (matches === null) {
-        throw new InvalidRegexError(DURATION_COMPARISON_REGEX, val, DURATION_COMPARISON_REGEX_URL)
+export const parseDuration = (val: string, strict = true): Duration => {
+    const res = parseDurationFromString(val, strict);
+    if(res.length > 1) {
+        throw new SimpleError(`Must only have one Duration value, found ${res.length} in: ${val}`);
     }
-    const groups = matches.groups as any;
-    const dur: Duration = dayjs.duration(groups.time, groups.unit);
-    if (!dayjs.isDuration(dur)) {
-        throw new SimpleError(`Parsed value '${val}' did not result in a valid Dayjs Duration`);
-    }
-    return {
-        operator: groups.opStr as StringOperator,
-        duration: dur
-    }
-}
-export const compareDurationValue = (comp: DurationComparison, date: Dayjs) => {
-    const dateToCompare = dayjs().subtract(comp.duration.asSeconds(), 'seconds');
-    return dateComparisonTextOp(date, comp.operator, dateToCompare);
+    return res[0].duration;
 }
 
 const SUBREDDIT_NAME_REGEX: RegExp = /^\s*(?:\/r\/|r\/)*(\w+)*\s*$/;
@@ -863,7 +804,7 @@ const WIKI_REGEX_URL = 'https://regexr.com/61bq1';
 const URL_REGEX: RegExp = /^\s*url:(?<url>[^\s]+)\s*$/;
 const URL_REGEX_URL = 'https://regexr.com/61bqd';
 
-export const parseWikiContext = (val: string) => {
+export const parseWikiContext = (val: string): WikiContext | undefined => {
     const matches = val.match(WIKI_REGEX);
     if (matches === null) {
         return undefined;
@@ -883,6 +824,34 @@ export const parseExternalUrl = (val: string) => {
     return (matches.groups as any).url as string;
 }
 
+export const parseUrlContext = (val: string): UrlContext | undefined => {
+    const wiki = parseWikiContext(val);
+    if(wiki !== undefined) {
+        return {
+            value: val,
+            context: wiki
+        }
+    }
+    const urlContext = parseExternalUrl(val);
+    if(urlContext !== undefined) {
+        return {
+            value: val,
+            context: {
+                url: urlContext
+            }
+        }
+    }
+    return undefined;
+}
+
+export const asWikiContext = (val: object): val is WikiContext => {
+    return val !== null && typeof val === 'object' && 'wiki' in val;
+}
+
+export const asExtUrlContext = (val: object): val is ExternalUrlContext => {
+    return val !== null && typeof val === 'object' && 'url' in val;
+}
+
 export const dummyLogger = {
     debug: (v: any) => null,
     error: (v: any) => null,
@@ -890,62 +859,81 @@ export const dummyLogger = {
     info: (v: any) => null
 }
 
-const GIST_REGEX = new RegExp(/.*gist\.github\.com\/.+\/(.+)/i)
-const GH_BLOB_REGEX = new RegExp(/.*github\.com\/(.+)\/(.+)\/blob\/(.+)/i);
-const REGEXR_REGEX = new RegExp(/^.*((regexr\.com)\/[\w\d]+).*$/i);
-const REGEXR_PAGE_REGEX = new RegExp(/(.|[\n\r])+"expression":"(.+)","text"/g);
-export const fetchExternalUrl = async (url: string, logger: (any) = dummyLogger): Promise<string> => {
+export const normalizeGistFileKey = (val: string) => val.replaceAll(/[^\w\d]/g, '').toLowerCase().trim();
+export const GIST_REGEX = new RegExp(/.*gist\.github\.com\/(?<user>.+)\/(?<gistId>[^#\/]+)(?:#file-(?<fileName>.+))?/i)
+export const GIST_RAW_REGEX = new RegExp(/.*gist\.github\.com\/(?<user>.+)\/(?<gistId>[^#\/]+)\/raw\/.+/i)
+export const GH_BLOB_REGEX = new RegExp(/.*github\.com\/(?<user>.+)\/(?<repo>.+)\/blob\/(?<path>.+)(?:#.+)?/i);
+export const REGEXR_REGEX = new RegExp(/^.*((regexr\.com)\/[\w\d]+).*$/i);
+export const REGEXR_PAGE_REGEX = new RegExp(/(.|[\n\r])+"expression":"(.+)","text"/g);
+export const fetchExternalResult = async (url: string, logger: (any) = dummyLogger): Promise<[string, Response]> => {
     let hadError = false;
     logger.debug(`Attempting to detect resolvable URL for ${url}`);
-    let match = url.match(GIST_REGEX);
-    if (match !== null) {
-        const gistApiUrl = `https://api.github.com/gists/${match[1]}`;
-        logger.debug(`Looks like a non-raw gist URL! Trying to resolve ${gistApiUrl}`);
+    let match = parseRegexSingleOrFail(GIST_RAW_REGEX, url); // check for raw gist url first and if found treat as normal URL
+    if(match === undefined) {
+        // if not raw then if its still a gist then we need to parse and use API
+        match = parseRegexSingleOrFail(GIST_REGEX, url);
 
-        try {
-            const response = await fetch(gistApiUrl);
-            if (!response.ok) {
-                logger.error(`Response was not OK from Gist API (${response.statusText}) -- will return response from original URL instead`);
-                if (response.size > 0) {
-                    logger.error(await response.text())
-                }
-                hadError = true;
-            } else {
-                const data = await response.json();
-                // get first found file
-                const fileKeys = Object.keys(data.files);
-                if (fileKeys.length === 0) {
-                    logger.error(`No files found in gist!`);
+        if (match !== undefined) {
+            const gistApiUrl = `https://api.github.com/gists/${match.named.gistId}`;
+            logger.debug(`Looks like a non-raw gist URL! Trying to resolve ${gistApiUrl} ${match.named.fileName !== undefined ? ` and find file ${match.named.fileName}` : ''}`);
+
+            try {
+                const response = await fetch(gistApiUrl);
+                if (!response.ok) {
+                    logger.warn(`Response was not OK from Gist API (${response.statusText}) -- will return response from original URL instead`);
+                    if (response.size > 0) {
+                        logger.warn(await response.text())
+                    }
+                    hadError = true;
                 } else {
-                    if (fileKeys.length > 1) {
-                        logger.warn(`More than one file found in gist! Using first found: ${fileKeys[0]}`);
+                    const data = await response.json();
+                    // get first found file
+                    const fileKeys = Object.keys(data.files);
+                    if (fileKeys.length === 0) {
+                        logger.error(`No files found in gist!`);
                     } else {
-                        logger.debug(`Using file ${fileKeys[0]}`);
-                    }
-                    const file = data.files[fileKeys[0]];
-                    if (file.truncated === false) {
-                        return file.content;
-                    }
-                    const rawUrl = file.raw_url;
-                    logger.debug(`File contents was truncated, retrieving full contents from ${rawUrl}`);
-                    try {
-                        const rawUrlResponse = await fetch(rawUrl);
-                        return await rawUrlResponse.text();
-                    } catch (err: any) {
-                        logger.error('Gist Raw URL Response returned an error, will return response from original URL instead');
-                        logger.error(err);
+                        let fileKey = fileKeys[0];
+                        if (fileKeys.length > 1) {
+                            if(match.named.fileName !== undefined) {
+                                //const normalizedFileName = normalizeGistFileKey(match.named.fileName.replace('/^file-/', ''));
+                                const normalizedFileName = normalizeGistFileKey(match.named.fileName);
+                                const matchingKey = fileKeys.find(x => normalizeGistFileKey(x) === normalizedFileName);
+                                if(matchingKey === undefined) {
+                                    throw new SimpleError(`Found Gist ${match.named.gistId} but it did not contain a file named ${match.named.fileName}`);
+                                }
+                                fileKey = matchingKey;
+                            } else {
+                                logger.warn(`More than one file found in gist but URL did not specify a filename! Using first found: ${fileKey}`);
+                            }
+                        } else {
+                            logger.debug(`Using file ${fileKey}`);
+                        }
+                        const file = data.files[fileKey];
+                        if (file.truncated === false) {
+                            return [file.content, response];
+                        }
+                        const rawUrl = file.raw_url;
+                        logger.debug(`File contents was truncated, retrieving full contents from ${rawUrl}`);
+                        try {
+                            const rawUrlResponse = await fetch(rawUrl);
+                            return [await rawUrlResponse.text(), rawUrlResponse];
+                        } catch (err: any) {
+                            logger.error('Gist Raw URL Response returned an error, will return response from original URL instead');
+                            logger.error(err);
+                        }
                     }
                 }
+            } catch (err: any) {
+                logger.error('Response returned an error, will return response from original URL instead');
+                logger.error(err);
             }
-        } catch (err: any) {
-            logger.error('Response returned an error, will return response from original URL instead');
-            logger.error(err);
         }
     }
-    match = url.match(GH_BLOB_REGEX);
 
-    if (match !== null) {
-        const rawUrl = `https://raw.githubusercontent.com/${match[1]}/${match[2]}/${match[3]}`
+    match = parseRegexSingleOrFail(GH_BLOB_REGEX, url)
+
+    if (match !== undefined) {
+        const rawUrl = `https://raw.githubusercontent.com/${match.named.user}/${match.named.repo}/${match.named.path}`
         logger.debug(`Looks like a single file github URL! Resolving to ${rawUrl}`);
         try {
             const response = await fetch(rawUrl);
@@ -956,7 +944,7 @@ export const fetchExternalUrl = async (url: string, logger: (any) = dummyLogger)
                 }
                 hadError = true;
             } else {
-                return await response.text();
+                return [await response.text(), response];
             }
         } catch (err: any) {
             logger.error('Response returned an error, will return response from original URL instead');
@@ -964,8 +952,8 @@ export const fetchExternalUrl = async (url: string, logger: (any) = dummyLogger)
         }
     }
 
-    match = url.match(REGEXR_REGEX);
-    if(match !== null) {
+    match = parseRegexSingleOrFail(REGEXR_REGEX, url);
+    if(match !== undefined) {
         logger.debug(`Looks like a Regexr URL! Trying to get expression from page HTML`);
         try {
             const response = await fetch(url);
@@ -1000,7 +988,7 @@ export const fetchExternalUrl = async (url: string, logger: (any) = dummyLogger)
         }
         throw new Error(`Response was not OK: ${response.statusText}`);
     }
-    return await response.text();
+    return [await response.text(), response];
 }
 
 export interface RetryOptions {
@@ -1449,30 +1437,56 @@ export const parseStringToRegex = (val: string, defaultFlags?: string): RegExp |
     return new RegExp(result[1], flags);
 }
 
-export const parseRegex = (reg: RegExp, val: string): RegExResult => {
+export const parseStringToRegexOrLiteralSearch = (val: string, defaultFlags: string = 'i'): RegExp => {
+    const maybeRegex = parseStringToRegex(val, defaultFlags);
+    if (maybeRegex !== undefined) {
+        return maybeRegex;
+    }
+    const literalSearchRegex = parseStringToRegex(`/${escapeRegex(val.trim())}/`, defaultFlags);
+    if (literalSearchRegex === undefined) {
+        throw new SimpleError(`Could not convert test value to a valid regex: ${val}`);
+    }
+    return literalSearchRegex;
+}
+
+export const parseRegex = (reg: RegExp, val: string): RegExResult[] | undefined => {
 
     if(reg.global) {
         const g = Array.from(val.matchAll(reg));
-        const global = g.map(x => {
+        if(g.length === 0) {
+            return undefined;
+        }
+        return g.map(x => {
             return {
                 match: x[0],
+                index: x.index,
                 groups: x.slice(1),
-                named: x.groups,
-            }
+                named: x.groups || {},
+            } as RegExResult;
         });
-        return {
-            matched: g.length > 0,
-            matches: g.length > 0 ? g.map(x => x[0]) : [],
-            global: g.length > 0 ? global : [],
-        };
     }
 
     const m = val.match(reg)
-    return {
-        matched: m !== null,
-        matches: m !== null ? m.slice(0).filter(x => x !== undefined) : [],
-        global: [],
+    if(m === null) {
+        return undefined;
     }
+    return [{
+        match: m[0],
+        index: m.index as number,
+        groups: m.slice(1),
+        named: m.groups || {}
+    }];
+}
+
+export const parseRegexSingleOrFail = (reg: RegExp, val: string): RegExResult | undefined => {
+    const results = parseRegex(reg, val);
+    if(results !== undefined) {
+        if(results.length > 1) {
+                throw new SimpleError(`Expected Regex to match once but got ${results.length} results. Either Regex must NOT be global (using 'g' flag) or parsed value must only match regex once. Given: ${val} || Regex: ${reg.toString()}`);
+        }
+        return results[0];
+    }
+    return undefined;
 }
 
 export const testMaybeStringRegex = (test: string, subject: string, defaultFlags: string = 'i'): [boolean, string] => {
@@ -1766,12 +1780,25 @@ export const difference = (a: Array<any>, b: Array<any>) => {
     return Array.from(setMinus(a, b));
 }
 
+// can use 'in' operator to check if object has a property with name WITHOUT TRIGGERING a snoowrap proxy to fetch
+export const isSubreddit = (value: any) => {
+    try {
+        return value !== null && typeof value === 'object' && (value instanceof Subreddit || ('id' in value && value.id !== undefined && value.id.includes('t5_')) || 'display_name' in value);
+    } catch (e) {
+        return false;
+    }
+}
+
+export const asSubreddit = (value: any): value is Subreddit => {
+    return isSubreddit(value);
+}
+
 /**
  * Cached activities lose type information when deserialized so need to check properties as well to see if the object is the shape of a Submission
  * */
 export const isSubmission = (value: any) => {
     try {
-        return value !== null && typeof value === 'object' && (value instanceof Submission || (value.name !== undefined && value.name.includes('t3_')));
+        return value !== null && typeof value === 'object' && (value instanceof Submission || ('name' in value && value.name !== undefined && value.name.includes('t3_')));
     } catch (e) {
         return false;
     }
@@ -1783,7 +1810,7 @@ export const asSubmission = (value: any): value is Submission => {
 
 export const isComment = (value: any) => {
     try {
-        return value !== null && typeof value === 'object' && (value instanceof Comment || value.name.includes('t1_'));
+        return value !== null && typeof value === 'object' && (value instanceof Comment || ('name' in value && value.name !== undefined && value.name.includes('t1_')));
     } catch (e) {
         return false;
     }
@@ -1799,7 +1826,7 @@ export const asActivity = (value: any): value is (Submission | Comment) => {
 
 export const isUser = (value: any) => {
     try {
-        return value !== null && typeof value === 'object' && (value instanceof RedditUser || value.name.includes('t2_'));
+        return value !== null && typeof value === 'object' && (value instanceof RedditUser || ('name' in value && value.name !== undefined && value.name.includes('t2_')));
     } catch(e) {
         return false;
     }
@@ -1821,6 +1848,20 @@ export const userNoteCriteriaSummary = (val: UserNoteCriteria): string => {
     return `${val.count === undefined ? '>= 1' : val.count} of ${val.search === undefined ? 'current' : val.search} notes is ${val.type}`;
 }
 
+export const modActionCriteriaSummary = (val: (ModNoteCriteria | ModLogCriteria)): string => {
+    const isNote = asModNoteCriteria(val);
+    const preamble = `${val.count === undefined ? '>= 1' : val.count} of ${val.search === undefined ? 'current' : val.search} ${isNote ? 'notes' : 'actions'} is`;
+    const filters = Object.entries(val).reduce((acc: string[], curr) => {
+        if(['count', 'search'].includes(curr[0])) {
+            return acc;
+        }
+        const vals = Array.isArray(curr[1]) ? curr[1] : [curr[1]];
+       acc.push(`${curr[0]}: ${vals.join(' ,')}`)
+        return acc;
+    }, []);
+    return `${preamble} ${filters.join(' || ')}`;
+}
+
 /**
  * Serialized activities store subreddit and user properties as their string representations (instead of proxy)
  * */
@@ -2489,6 +2530,30 @@ export const normalizeCriteria = <T extends AuthorCriteria | TypedActivityState
         if(criteria.description !== undefined) {
             criteria.description = Array.isArray(criteria.description) ? criteria.description : [criteria.description];
         }
+        if(criteria.modActions !== undefined) {
+            criteria.modActions.map((x, index) => {
+                const common = {
+                    ...x,
+                    type: x.type === undefined ? undefined : (Array.isArray(x.type) ? x.type : [x.type])
+                }
+                if(asModNoteCriteria(x)) {
+                    return {
+                        ...common,
+                        noteType: x.noteType === undefined ? undefined : (Array.isArray(x.noteType) ? x.noteType : [x.noteType]),
+                        note: x.note === undefined ? undefined : (Array.isArray(x.note) ? x.note : [x.note]),
+                    }
+                } else if(asModLogCriteria(x)) {
+                    return {
+                        ...common,
+                        action: x.action === undefined ? undefined : (Array.isArray(x.action) ? x.action : [x.action]),
+                        details: x.details === undefined ? undefined : (Array.isArray(x.details) ? x.details : [x.details]),
+                        description: x.description === undefined ? undefined : (Array.isArray(x.description) ? x.description : [x.description]),
+                        activityType: x.activityType === undefined ? undefined : (Array.isArray(x.activityType) ? x.activityType : [x.activityType]),
+                    }
+                }
+                return common;
+            })
+        }
     }
 
     return {
@@ -2652,6 +2717,22 @@ export const parseRedditFullname = (str: string): RedditThing | undefined => {
     }
 }
 
+export const generateSnoowrapEntityFromRedditThing = (data: RedditThing, client: Snoowrap) => {
+    switch(data.type) {
+        case 'comment':
+            return new Comment({id: data.val}, client, false);
+        case 'submission':
+            return new Submission({id: data.val}, client, false);
+        case 'user':
+            return new RedditUser({id: data.val}, client, false);
+        case 'subreddit':
+            return new Subreddit({id: data.val}, client, false);
+        case 'message':
+            return new PrivateMessage({id: data.val}, client, false)
+
+    }
+}
+
 export const activityDispatchConfigToDispatch = (config: ActivityDispatchConfig, activity: (Comment | Submission), type: ActivitySourceTypes, {action, dryRun}: {action?: string, dryRun?: boolean} = {}): ActivityDispatch => {
     let tolerantVal: boolean | Duration | undefined;
     if (config.tardyTolerant !== undefined) {
@@ -2666,7 +2747,6 @@ export const activityDispatchConfigToDispatch = (config: ActivityDispatchConfig,
         delay: parseDurationValToDuration(config.delay),
         tardyTolerant: tolerantVal,
         queuedAt: dayjs().utc(),
-        processing: false,
         id: nanoid(16),
         activity,
         action,
@@ -2793,3 +2873,52 @@ export const filterByTimeRequirement = (satisfiedEndtime: Dayjs, listSlice: Snoo
 
     return [truncatedItems.length !== listSlice.length, truncatedItems]
 }
+
+export const between = (val: number, a: number, b: number, inclusiveMin: boolean = false, inclusiveMax: boolean = false): boolean => {
+    var min = Math.min(a, b),
+        max = Math.max(a, b);
+
+    if(!inclusiveMin && !inclusiveMax) {
+        return val > min && val < max;
+    }
+    if(inclusiveMin && inclusiveMax) {
+        return val >= min && val <= max;
+    }
+    if(inclusiveMin) {
+        return val >= min && val < max;
+    }
+
+    // inclusive max
+    return val > min && val <= max;
+}
+
+export const toModNoteLabel = (val: string): ModUserNoteLabel => {
+    const cleanVal = val.trim().toUpperCase();
+    if (asModNoteLabel(cleanVal)) {
+        return cleanVal;
+    }
+    throw new Error(`${val} is not a valid mod note label. Must be one of: ${modUserNoteLabels.join(', ')}`);
+}
+
+
+export const asModNoteLabel = (val: string): val is ModUserNoteLabel => {
+    return modUserNoteLabels.includes(val);
+}
+
+/**
+ * Split an array into two based on a truthy function
+ *
+ * Returns arrays -> [[...passed],[...failed]]
+ *
+ * https://stackoverflow.com/a/42299191/1469797
+ * */
+export function partition<T>(array: T[], callback: (element: T, index: number, array: T[]) => boolean) {
+    return array.reduce(function (result: [T[], T[]], element, i) {
+            callback(element, i, array)
+                ? result[0].push(element)
+                : result[1].push(element);
+
+            return result;
+        }, [[], []]
+    );
+}

tests/languageProcessing.test.ts

@@ -0,0 +1,342 @@
+import {describe, it} from 'mocha';
+import chai,{assert} from 'chai';
+import chaiAsPromised from 'chai-as-promised';
+import {
+    getContentLanguage,
+    getLanguageTypeFromValue,
+    getStringSentiment, parseTextToNumberComparison,
+    testSentiment
+} from "../src/Common/LangaugeProcessing";
+import {GenericComparison, RangedComparison} from "../src/Common/Infrastructure/Comparisons";
+
+chai.use(chaiAsPromised);
+
+const longNeutralEnglish = "This is a normal english sentence without emotion";
+const longNeutralEnglish2 = 'I am neutral on the current subject';
+const longNeutralEnglish3 = 'The midterms were an election that happened';
+const longNegativeEnglish = "I hate when idiots drive their bad cars terribly. 😡";
+const longPositiveEnglish = "We love to be happy and laugh on this wonderful, amazing day";
+
+const shortIndistinctEnglish = "metal gear";
+const shortIndistinctEnglish2 = "idk hole ref";
+
+const shortPositiveEnglish = "haha fun";
+const shortNegativeEnglish = "fuck you";
+const shortSlangPositiveEnglish = "lol lmao";
+const shortSlangNegativeEnglish = "get fuked";
+
+const longIndonesian = "setiap kali scroll mesti nampak dia nie haih";
+const shortIndonesian = "Saya bangga saya rasis";
+const shortPolish = 'Dobry wieczór';
+const longRussian = 'Чит на золото для аватарии без скачивания бесплатно';
+const longItalian = 'Sembra ormai passato un secolo, visto che gli anime sono praticamente scomparsi dalla televisione.';
+
+const shortRomanian = 'Tu știi unde sta?';
+const longRomanian = 'Deci , daca aveti chef de un mic protest , va astept la aceste coordonate';
+
+const longFrench = "J’approuve et à ce moment là ça se soigne plus malheureusement";
+
+const longSpanish = "La segunda parece una mezcla entre una convención de fanáticos de los monster truck y un vertedero.";
+const longPositiveSpanish = 'me encanta esta hermosa cancion';
+const longPositiveSpanish2 = 'Increíble muy divertido gracias por compartir';
+
+const longGerman = "bin mir auch sicher, dass zb mein 65er halb so viel wiegt wie ein kasten Bier";
+
+const shortEmojiNegative = "France 😫 😞 :(";
+const shortEmojiPositive = "France 😂 😄 😁";
+
+describe('Language Detection', function () {
+
+    describe('Derives language from user input', async function () {
+        it('gets from valid, case-insensitive alpha2', async function () {
+            const lang = await getLanguageTypeFromValue('eN');
+            assert.equal(lang.alpha2, 'en');
+        });
+        it('gets from valid, case-insensitive alpha3', async function () {
+            const lang = await getLanguageTypeFromValue('eNg');
+            assert.equal(lang.alpha2, 'en');
+        });
+        it('gets from valid, case-insensitive language name', async function () {
+            const lang = await getLanguageTypeFromValue('EnGlIsH');
+            assert.equal(lang.alpha2, 'en');
+        });
+
+        it('throws on invalid value', function () {
+            assert.isRejected(getLanguageTypeFromValue('pofdsfa'))
+        });
+    })
+
+    describe('Recognizes the language in moderately long content well', function () {
+        it('should recognize english', async function () {
+            const lang = await getContentLanguage(longPositiveEnglish);
+            assert.equal(lang.language.alpha2, 'en');
+            assert.isFalse(lang.usedDefault);
+            assert.isAtLeast(lang.bestGuess.score, 0.9);
+        });
+        it('should recognize french', async function () {
+            const lang = await getContentLanguage(longFrench);
+            assert.equal(lang.language.alpha2, 'fr');
+            assert.isFalse(lang.usedDefault);
+            assert.isAtLeast(lang.bestGuess.score, 0.9);
+        });
+        it('should recognize spanish', async function () {
+            const lang = await getContentLanguage(longSpanish);
+            assert.equal(lang.language.alpha2, 'es');
+            assert.isFalse(lang.usedDefault);
+            assert.isAtLeast(lang.bestGuess.score, 0.9);
+        });
+        it('should recognize german', async function () {
+            const lang = await getContentLanguage(longGerman);
+            assert.equal(lang.language.alpha2, 'de');
+            assert.isFalse(lang.usedDefault);
+            assert.isAtLeast(lang.bestGuess.score, 0.9);
+        });
+        it('should recognize indonesian', async function () {
+            const lang = await getContentLanguage(longIndonesian);
+            assert.equal(lang.language.alpha2, 'id');
+            assert.isFalse(lang.usedDefault);
+            assert.isAtLeast(lang.bestGuess.score, 0.9);
+        });
+    });
+
+    describe('Correctly handles short content classification', function () {
+        it('uses default language', async function () {
+
+            for (const content of [shortIndistinctEnglish, shortIndistinctEnglish2, shortIndonesian]) {
+                const lang = await getContentLanguage(content);
+                assert.equal(lang.language.alpha2, 'en', content);
+                assert.isTrue(lang.usedDefault, content);
+            }
+        });
+
+        it('uses best guess when default language is not provided', async function () {
+
+            for (const content of [shortIndistinctEnglish, shortIndistinctEnglish2, shortIndonesian]) {
+                const lang = await getContentLanguage(content, {defaultLanguage: false});
+                assert.isFalse(lang.usedDefault);
+            }
+        });
+    });
+});
+
+describe('Sentiment', function() {
+
+    describe('Is conservative when no default language is used for short content', function() {
+
+        it('should return unusable result for short, ambiguous english content', async function() {
+            for(const content of [shortIndistinctEnglish, shortIndistinctEnglish2]) {
+                const res = await getStringSentiment(content, {defaultLanguage: false});
+                assert.isFalse(res.usableScore);
+            }
+        });
+
+        it('should return unusable result for short, non-english content', async function() {
+            for(const content of [shortIndonesian, shortPolish, shortRomanian]) {
+                const res = await getStringSentiment(content, {defaultLanguage: false});
+                assert.isFalse(res.usableScore);
+            }
+        });
+
+    });
+
+    describe('Is conservative when language confidence is high for unsupported languages', function() {
+
+        it('should return unusable result for long, non-english content', async function() {
+            for(const content of [longIndonesian, longRussian, longItalian, longRomanian]) {
+                const res = await getStringSentiment(content);
+                assert.isFalse(res.usableScore, content);
+            }
+        });
+    });
+
+    describe('vader/wink supersedes low confidence language guess', function() {
+
+        it('should return usable result when valid words found by vader/wink', async function() {
+            for(const content of [shortPositiveEnglish,shortNegativeEnglish]) {
+                const res = await getStringSentiment(content, {defaultLanguage: false});
+                assert.isTrue(res.usableScore);
+            }
+        });
+
+        it('should return usable result when valid slang found by vader/wink', async function() {
+            for(const content of [shortSlangPositiveEnglish,shortSlangNegativeEnglish]) {
+                const res = await getStringSentiment(content, {defaultLanguage: false});
+                assert.isTrue(res.usableScore);
+            }
+        });
+
+        it('should return usable result when valid emojis found by vader/wink', async function() {
+            for(const content of [shortEmojiPositive,shortEmojiNegative]) {
+                const res = await getStringSentiment(content, {defaultLanguage: false});
+                assert.isTrue(res.usableScore);
+            }
+        });
+    })
+
+    describe('Detects correct sentiment', function() {
+
+        describe('In English', function() {
+
+            it('should detect positive sentiment', async function() {
+                for(const content of [shortEmojiPositive,longPositiveEnglish, shortPositiveEnglish, shortSlangPositiveEnglish]) {
+                    const res = await getStringSentiment(content);
+                    assert.isTrue(res.usableScore);
+                    assert.isAtLeast(res.scoreWeighted, 0.1);
+                }
+            });
+
+            it('should detect negative sentiment', async function() {
+                for(const content of [shortEmojiNegative,longNegativeEnglish, shortNegativeEnglish, shortSlangNegativeEnglish]) {
+                    const res = await getStringSentiment(content);
+                    assert.isTrue(res.usableScore);
+                    assert.isAtMost(res.scoreWeighted, -0.1);
+                }
+            });
+
+            it('should detect neutral sentiment', async function() {
+                for(const content of [longNeutralEnglish, longNeutralEnglish2, longNeutralEnglish3]) {
+                    const res = await getStringSentiment(content);
+                    assert.isTrue(res.usableScore, content);
+                    assert.isAtMost(res.scoreWeighted, 0.1, content);
+                    assert.isAtLeast(res.scoreWeighted, -0.1, content);
+                }
+            });
+
+            it('should detect neutral sentiment for short content when english is default language', async function() {
+                for(const content of [shortIndistinctEnglish, shortIndistinctEnglish2, shortPolish]) {
+                    const res = await getStringSentiment(content);
+                    assert.isTrue(res.usableScore);
+                    assert.isAtMost(res.scoreWeighted, 0.1, content);
+                    assert.isAtLeast(res.scoreWeighted, -0.1, content);
+                }
+            });
+        });
+
+        describe('In Spanish', function() {
+            it('should detect positive ', async function() {
+                for(const content of [longPositiveSpanish, longPositiveSpanish2]) {
+                    const res = await getStringSentiment(content);
+                    assert.isTrue(res.usableScore, longPositiveSpanish2);
+                    assert.isAtLeast(res.scoreWeighted, 0.1, longPositiveSpanish2);
+                }
+            });
+        });
+    });
+
+    describe('Testing', function () {
+
+        describe('Parsing user input to comparison', function() {
+
+            it(`parses 'is neutral'`, function() {
+                const res = parseTextToNumberComparison('is neutral') as RangedComparison;
+                assert.deepEqual(res.range, [-0.1, 0.1]);
+                assert.isFalse(res.not);
+            });
+
+            it(`parses 'is not neutral'`, function() {
+                const res = parseTextToNumberComparison('is not neutral') as RangedComparison;
+                assert.deepEqual(res.range, [-0.1, 0.1]);
+                assert.isTrue(res.not);
+            });
+
+            it(`parses 'is positive'`, function() {
+                const res = parseTextToNumberComparison('is positive') as GenericComparison;
+                assert.equal(res.operator, '>=');
+                assert.equal(res.value, 0.1);
+            });
+
+            it(`parses 'is very positive'`, function() {
+                const res = parseTextToNumberComparison('is very positive') as GenericComparison;
+                assert.equal(res.operator, '>=');
+                assert.equal(res.value, 0.3);
+            });
+
+            it(`parses 'is extremely positive'`, function() {
+                const res = parseTextToNumberComparison('is extremely positive') as GenericComparison;
+                assert.equal(res.operator, '>=');
+                assert.equal(res.value, 0.6);
+            });
+
+            it(`parses 'is negative'`, function() {
+                const res = parseTextToNumberComparison('is negative') as GenericComparison;
+                assert.equal(res.operator, '<=');
+                assert.equal(res.value, -0.1);
+            });
+
+            it(`parses 'is very negative'`, function() {
+                const res = parseTextToNumberComparison('is very negative') as GenericComparison;
+                assert.equal(res.operator, '<=');
+                assert.equal(res.value, -0.3);
+            });
+
+            it(`parses 'is extremely negative'`, function() {
+                const res = parseTextToNumberComparison('is extremely negative') as GenericComparison;
+                assert.equal(res.operator, '<=');
+                assert.equal(res.value, -0.6);
+            });
+
+            it(`parses negative negations`, function() {
+                const res = parseTextToNumberComparison('is not extremely negative') as GenericComparison;
+                assert.equal(res.operator, '>');
+                assert.equal(res.value, -0.6);
+            });
+
+            it(`parses positive negations`, function() {
+                const res = parseTextToNumberComparison('is not positive') as GenericComparison;
+                assert.equal(res.operator, '<');
+                assert.equal(res.value, 0.1);
+            });
+
+        });
+
+        it('should fail test if score is unusable', async function() {
+
+            const comparison = parseTextToNumberComparison('is positive');
+
+            for(const content of [shortIndistinctEnglish, shortIndistinctEnglish2, shortPolish, longRomanian]) {
+                const sentimentResult = await getStringSentiment(content, {defaultLanguage: false});
+
+                const testResult = testSentiment(sentimentResult, comparison);
+                assert.isFalse(testResult.passes);
+            }
+        });
+
+        it('should handle generic comparisons', async function() {
+
+            const comparison = parseTextToNumberComparison('is positive');
+
+            for(const content of [shortEmojiPositive,longPositiveEnglish, shortPositiveEnglish, shortSlangPositiveEnglish]) {
+                const sentimentResult = await getStringSentiment(content, {defaultLanguage: false});
+
+                const testResult = testSentiment(sentimentResult, comparison);
+                assert.isTrue(testResult.passes);
+            }
+        });
+
+        it('should handle ranged comparisons', async function() {
+
+            const comparison = parseTextToNumberComparison('is neutral');
+
+            for(const content of [longNeutralEnglish, longNeutralEnglish2, longNeutralEnglish3]) {
+                const sentimentResult = await getStringSentiment(content, {defaultLanguage: false});
+
+                const testResult = testSentiment(sentimentResult, comparison);
+                assert.isTrue(testResult.passes);
+            }
+        });
+
+        it('should handle negated ranged comparisons', async function() {
+
+            const comparison = parseTextToNumberComparison('is not neutral');
+
+            for(const content of [longPositiveEnglish, longPositiveSpanish, longNegativeEnglish]) {
+                const sentimentResult = await getStringSentiment(content, {defaultLanguage: false});
+
+                const testResult = testSentiment(sentimentResult, comparison);
+                assert.isTrue(testResult.passes, content);
+            }
+        });
+    });
+});
+
+