Merge branch 'edge'

* Only return logs for "default viewed" subreddit/bot when fetching instance status, when specified from QS
  * Greatly reduces amount of data fetched and response time
* Return logs with formatted property for non-streaming response
* Implement server live stats endpoint to return subreddit/all stats based on QS
* Use client websocket connection to return stats for currently viewed subreddit

.github/FUNDING.yml

@@ -0,0 +1,2 @@
+github: [FoxxMD]
+custom: ["bitcoincash:qqmpsh365r8n9jhp4p8ks7f7qdr7203cws4kmkmr8q"]

.gitignore

@@ -383,3 +383,12 @@ dist
 **/src/**/*.js
 !src/Web/assets/public/yaml/*
 **/src/**/*.map
+/**/*.sqlite
+/**/*.bak
+*.yaml
+*.json5
+
+!src/Schema/*.json
+!docs/**/*.json5
+!docs/**/*.yaml
+!docs/**/*.json

README.md

@@ -1,6 +1,7 @@
-[![Latest Release](https://img.shields.io/github/v/release/foxxmd/context-mod)](https://github.com/FoxxMD/context-mod/releases)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![Docker Pulls](https://img.shields.io/docker/pulls/foxxmd/context-mod)](https://hub.docker.com/r/foxxmd/context-mod)
+# ContextMod [![Latest Release](https://img.shields.io/github/v/release/foxxmd/context-mod)](https://github.com/FoxxMD/context-mod/releases) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Docker Pulls](https://img.shields.io/docker/pulls/foxxmd/context-mod)](https://hub.docker.com/r/foxxmd/context-mod)
+
+<img src="/docs/logo.png" align="right"
+alt="ContextMod logo" width="180" height="176">
 
 **Context Mod** (CM) is an event-based, [reddit](https://reddit.com) moderation bot built on top of [snoowrap](https://github.com/not-an-aardvark/snoowrap) and written in [typescript](https://www.typescriptlang.org/).
 

docs/gettingStartedOperator.md

@@ -22,13 +22,14 @@ PROTIP: Using a container management tool like [Portainer.io CE](https://www.por
 
 ### [Dockerhub](https://hub.docker.com/r/foxxmd/context-mod)
 
-```
-foxxmd/context-mod:latest
-```
+An example of starting the container using the [minimum configuration](/docs/operatorConfiguration.md#minimum-config) with a [configuration file](/docs/operatorConfiguration.md#defining-configuration-via-file):
+
+* Bind the folder where the config is located on your host machine into the container `-v /host/path/folder:/config`
+* Tell CM where to find the config using an env `-e "OPERATOR_CONFIG=/config/myConfig.yaml"`
+* Expose the web interface using the container port `8085`
 
-Adding **environmental variables** to your `docker run` command will pass them through to the app EX:
 ```
-docker run -d -e "CLIENT_ID=myId" ... foxxmd/context-mod
+docker run -d -e "OPERATOR_CONFIG=/config/myConfig.yaml" -v /host/path/folder:/config -p 8085:8085 foxxmd/context-mod
 ```
 
 ### Locally
@@ -47,6 +48,12 @@ npm install
 tsc -p .
 ```
 
+An example of running CM using the [minimum configuration](/docs/operatorConfiguration.md#minimum-config) with a [configuration file](/docs/operatorConfiguration.md#defining-configuration-via-file):
+
+```bash
+node src/index.js run
+```
+
 ### [Heroku Quick Deploy](https://heroku.com/about)
 [![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://dashboard.heroku.com/new?template=https://github.com/FoxxMD/context-mod)
 

docs/operatorConfiguration.md

@@ -41,8 +41,10 @@ configuration.
 **Note:** When reading the **schema** if the variable is available at a level of configuration other than **FILE** it will be
 noted with the same symbol as above. The value shown is the default.
 
-* To load a JSON configuration (for **FILE**) **from the command line** use the `-c` cli argument EX: `node src/index.js -c /path/to/JSON/config.json`
-* To load a JSON configuration (for **FILE**) **using an environmental variable** use `OPERATOR_CONFIG` EX: `OPERATOR_CONFIG=/path/to/JSON/config.json`
+## Defining Configuration Via File
+
+* **from the command line**  use the `-c` cli argument EX: `node src/index.js -c /path/to/JSON/config.json`
+* **using an environmental variable**  use `OPERATOR_CONFIG` EX: `OPERATOR_CONFIG=/path/to/JSON/config.json`
 
 [**See the Operator Config Schema here**](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FOperatorConfig.json)
 
@@ -121,28 +123,41 @@ Below are examples of the minimum required config to run the application using a
 Using **FILE**
 <details>
 
+CM will look for a file configuration at `PROJECT_DIR/config.yaml` by default [or you can specify your own location.](#defining-configuration-via-file)
+
 YAML
 ```yaml
+operator:
+  name: YourRedditUsername
 bots:
   - credentials:
       clientId: f4b4df1c7b2
       clientSecret: 34v5q1c56ub
-      refreshToken: 34_f1w1v4
-      accessToken: p75_1c467b2
+web:
+  credentials:
+    clientId: f4b4df1c7b2
+    clientSecret: 34v5q1c56ub
 ```
 JSON
 ```json5
 {
+  "operator": {
+    "name": "YourRedditUsername"
+  },
   "bots": [
     {
       "credentials": {
         "clientId": "f4b4df1c7b2",
-        "clientSecret": "34v5q1c56ub",
-        "refreshToken": "34_f1w1v4",
-        "accessToken": "p75_1c467b2"
+        "clientSecret": "34v5q1c56ub"
       }
     }
-  ]
+  ],
+  "web": {
+    "credentials": {
+      "clientId": "f4b4df1c7b2",
+      "clientSecret": "34v5q1c56ub"
+    }
+  }
 }
 ```
 
@@ -153,10 +168,9 @@ Using **ENV** (`.env`)
 <details>
 
 ```
+OPERATOR=YourRedditUsername
 CLIENT_ID=f4b4df1c7b2
 CLIENT_SECRET=34v5q1c56ub
-REFRESH_TOKEN=34_f1w1v4
-ACCESS_TOKEN=p75_1c467b2
 ```
 
 </details>

docs/webInterface.md

@@ -0,0 +1,30 @@
+## Editing/Updating Your Config
+
+* Open the editor for your subreddit
+    * In the web dashboard \-> r/YourSubreddit \-> Config -> **View** [(here)](/docs/screenshots/config/config.jpg)
+* Follow the directions on the [link at the top of the window](/docs/screenshots/config/save.png) to enable config editing using your moderator account
+    * After enabling editing just click "save" at any time to save your config
+    * After you have added/edited your config the bot will detect changes within 5 minutes or you can manually trigger it by clicking **Update**
+
+## General Config (Editor) Tips
+
+* The editor will automatically validate your [syntax (formatting)](/docs/screenshots/config/syntax.png) and [config correctness](/docs/screenshots/config/correctness.png) (property names, required properties, etc.)
+    * These show up as squiggly lines like in Microsoft Word and as a [list at the bottom of the editor](/docs/screenshots/config/errors.png)
+* In your config all **Checks** and **Actions** have two properties that control how they behave:
+    * [**Enable**](/docs/screenshots/config/enable.png) (defaults to `enable: true`) -- Determines if the check or action is run, at all
+    * **Dryrun** (defaults to `dryRun: false`) -- When `true` the check or action will run but any **Actions** that may be triggered will "pretend" to execute but not actually talk to the Reddit API.
+      * Use `dryRun` to test your config without the bot making any changes on reddit
+    * When starting out with a new config it is recommended running the bot with remove/ban actions **disabled**
+        * Use `report` actions to get reports in your modqueue from the bot that describe what it detected and what it would do about it
+        * Once the bot is behaving as desired (no false positives or weird behavior) destructive actions can be enabled or turned off of dryrun
+
+## Web Dashboard Tips
+
+* Use the [**Overview** section](/docs/screenshots/botOperations.png) to control the bot at a high-level
+* You can **manually run** the bot on any activity (comment/submission) by pasting its permalink into the [input field below the Overview section](/docs/screenshots/runInput.png) and hitting one of the **run buttons**
+    * **Dry run** will make the bot run on the activity but it will only **pretend** to run actions, if triggered. This is super useful for testing your config without consequences
+    * **Run** will do everything
+* All of the bot's activity is shown in real-time in the [log section](/docs/screenshots/logs.png)
+    * This will output the results of all run checks/rules and any actions that run
+* You can view summaries of all activities that triggered a check (had actions run) by clicking on [Actioned Events](/docs/screenshots/actionsEvents.png)
+    * This includes activities run with dry run

package.json

@@ -35,6 +35,7 @@
     "cache-manager": "^3.4.4",
     "cache-manager-redis-store": "^2.0.0",
     "commander": "^8.0.0",
+    "comment-json": "^4.1.1",
     "cookie-parser": "^1.3.5",
     "dayjs": "^1.10.5",
     "deepmerge": "^4.2.2",
@@ -48,11 +49,11 @@
     "express-socket.io-session": "^1.3.5",
     "fast-deep-equal": "^3.1.3",
     "fuse.js": "^6.4.6",
+    "globrex": "^0.1.2",
     "got": "^11.8.2",
     "he": "^1.2.0",
     "http-proxy": "^1.18.1",
     "image-size": "^1.0.0",
-    "js-yaml": "^4.1.0",
     "json5": "^2.2.0",
     "jsonwebtoken": "^8.5.1",
     "leven": "^3.1.0",
@@ -68,6 +69,7 @@
     "passport-custom": "^1.1.1",
     "passport-jwt": "^4.0.0",
     "pixelmatch": "^5.2.1",
+    "pony-cause": "^1.1.1",
     "pretty-print-json": "^1.0.3",
     "safe-stable-stringify": "^1.1.1",
     "snoostorm": "^1.5.2",
@@ -82,6 +84,7 @@
     "winston-daily-rotate-file": "^4.5.5",
     "winston-duplex": "^0.1.1",
     "winston-transport": "^4.4.0",
+    "yaml": "2.0.0-10",
     "zlib": "^1.0.5"
   },
   "devDependencies": {
@@ -93,6 +96,7 @@
     "@types/express": "^4.17.13",
     "@types/express-session": "^1.17.4",
     "@types/express-socket.io-session": "^1.3.6",
+    "@types/globrex": "^0.1.1",
     "@types/he": "^1.1.1",
     "@types/http-proxy": "^1.17.7",
     "@types/js-yaml": "^4.0.1",
@@ -111,8 +115,9 @@
     "@types/string-similarity": "^4.0.0",
     "@types/tcp-port-used": "^1.0.0",
     "@types/triple-beam": "^1.3.2",
+    "ts-essentials": "^9.1.2",
     "ts-json-schema-generator": "^0.93.0",
-    "typescript-json-schema": "^0.50.1"
+    "typescript-json-schema": "~0.53"
   },
   "optionalDependencies": {
     "sharp": "^0.29.1"

src/Action/ApproveAction.ts

@@ -1,31 +1,72 @@
-import {ActionJson, ActionConfig} from "./index";
+import {ActionJson, ActionConfig, ActionOptions} from "./index";
 import Action from "./index";
-import Snoowrap, {Comment, Submission} from "snoowrap";
+import Snoowrap from "snoowrap";
 import {RuleResult} from "../Rule";
 import {ActionProcessResult} from "../Common/interfaces";
+import Submission from "snoowrap/dist/objects/Submission";
+import Comment from "snoowrap/dist/objects/Comment";
 
 export class ApproveAction extends Action {
+
+    targets: ApproveTarget[]
+
     getKind() {
         return 'Approve';
     }
 
+    constructor(options: ApproveOptions) {
+        super(options);
+        const {
+            targets = ['self']
+        } = options;
+
+        this.targets = targets;
+    }
+
     async process(item: Comment | Submission, ruleResults: RuleResult[], runtimeDryrun?: boolean): Promise<ActionProcessResult> {
         const dryRun = runtimeDryrun || this.dryRun;
         const touchedEntities = [];
-        //snoowrap typing issue, thinks comments can't be locked
-        // @ts-ignore
-        if (item.approved) {
-            this.logger.warn('Item is already approved');
-            return {
-                dryRun,
-                success: false,
-                result: 'Item is already approved'
+
+        const realTargets = item instanceof Submission ? ['self'] : this.targets;
+
+        for(const target of realTargets) {
+            let targetItem = item;
+            if(target !== 'self' && item instanceof Comment) {
+                targetItem = await this.resources.getActivity(this.client.getSubmission(item.link_id));
+            }
+
+            // @ts-ignore
+            if (targetItem.approved) {
+                const msg = `${target === 'self' ? 'Item' : 'Comment\'s parent Submission'} is already approved`;
+                this.logger.warn(msg);
+                return {
+                    dryRun,
+                    success: false,
+                    result: msg
+                }
+            }
+
+            if (!dryRun) {
+                // make sure we have an actual item and not just a plain object from cache
+                if(target !== 'self' && !(targetItem instanceof Submission)) {
+                    // @ts-ignore
+                    targetItem = await this.client.getSubmission((item as Comment).link_id).fetch();
+                }
+                // @ts-ignore
+                touchedEntities.push(await targetItem.approve());
+
+                if(target === 'self') {
+                    // @ts-ignore
+                    item.approved = true;
+                    await this.resources.resetCacheForItem(item);
+                } else if(await this.resources.hasActivity(targetItem)) {
+                    // @ts-ignore
+                    targetItem.approved = true;
+                    await this.resources.resetCacheForItem(targetItem);
+                }
             }
         }
-        if (!dryRun) {
-            // @ts-ignore
-            touchedEntities.push(await item.approve());
-        }
+
         return {
             dryRun,
             success: true,
@@ -34,8 +75,20 @@ export class ApproveAction extends Action {
     }
 }
 
-export interface ApproveActionConfig extends ActionConfig {
+export type ApproveTarget = 'self' | 'parent';
 
+export interface ApproveOptions extends ApproveActionConfig, ActionOptions {}
+
+export interface ApproveActionConfig extends ActionConfig {
+    /**
+     * Specify which Activities to approve
+     *
+     * This setting is only applicable if the Activity being acted on is a **comment**. On a **submission** the setting does nothing
+     *
+     * * self => approve activity being checked (comment)
+     * * parent => approve parent (submission) of activity being checked (comment)
+     * */
+    targets?: ApproveTarget[]
 }
 
 /**

src/Action/LockAction.ts

@@ -26,6 +26,9 @@ export class LockAction extends Action {
             //snoowrap typing issue, thinks comments can't be locked
             // @ts-ignore
             await item.lock();
+            // @ts-ignore
+            item.locked = true;
+            await this.resources.resetCacheForItem(item);
             touchedEntities.push(item);
         }
         return {

src/Action/MessageAction.ts

@@ -12,7 +12,8 @@ import {
     REDDIT_ENTITY_REGEX_URL,
     truncateStringToLength
 } from "../util";
-import SimpleError from "../Utils/SimpleError";
+import {SimpleError} from "../Utils/Errors";
+import {ErrorWithCause} from "pony-cause";
 
 export class MessageAction extends Action {
     content: string;
@@ -65,10 +66,7 @@ export class MessageAction extends Action {
                     recipient = `/r/${entityData.name}`;
                 }
             } catch (err: any) {
-                this.logger.error(`'to' field for message was not in a valid format. See ${REDDIT_ENTITY_REGEX_URL} for valid examples`);
-                this.logger.error(err);
-                err.logged = true;
-                throw err;
+                throw new ErrorWithCause(`'to' field for message was not in a valid format. See ${REDDIT_ENTITY_REGEX_URL} for valid examples`, {cause: err});
             }
             if(recipient.includes('/r/') && this.asSubreddit) {
                 throw new SimpleError(`Cannot send a message as a subreddit to another subreddit. Requested recipient: ${recipient}`);

src/Action/RemoveAction.ts

@@ -1,30 +1,48 @@
-import {ActionJson, ActionConfig} from "./index";
+import {ActionJson, ActionConfig, ActionOptions} from "./index";
 import Action from "./index";
 import Snoowrap, {Comment, Submission} from "snoowrap";
 import {RuleResult} from "../Rule";
 import {activityIsRemoved} from "../Utils/SnoowrapUtils";
 import {ActionProcessResult} from "../Common/interfaces";
+import dayjs from "dayjs";
+import {isSubmission} from "../util";
 
 export class RemoveAction extends Action {
+    spam: boolean;
+
     getKind() {
         return 'Remove';
     }
 
+    constructor(options: RemoveOptions) {
+        super(options);
+        const {
+            spam = false,
+        } = options;
+        this.spam = spam;
+    }
+
     async process(item: Comment | Submission, ruleResults: RuleResult[], runtimeDryrun?: boolean): Promise<ActionProcessResult> {
         const dryRun = runtimeDryrun || this.dryRun;
         const touchedEntities = [];
         // issue with snoowrap typings, doesn't think prop exists on Submission
         // @ts-ignore
         if (activityIsRemoved(item)) {
-            return {
-                dryRun,
-                success: false,
-                result: 'Item is already removed',
-            }
+            this.logger.warn('It looks like this Item is already removed!');
+        }
+        if (this.spam) {
+            this.logger.verbose('Marking as spam on removal');
         }
         if (!dryRun) {
             // @ts-ignore
-            await item.remove();
+            await item.remove({spam: this.spam});
+            item.banned_at_utc = dayjs().unix();
+            item.spam = this.spam;
+            if(!isSubmission(item)) {
+                // @ts-ignore
+                item.removed = true;
+            }
+            await this.resources.resetCacheForItem(item);
             touchedEntities.push(item);
         }
 
@@ -36,13 +54,16 @@ export class RemoveAction extends Action {
     }
 }
 
-export interface RemoveActionConfig extends ActionConfig {
+export interface RemoveOptions extends RemoveActionConfig, ActionOptions {
+}
 
+export interface RemoveActionConfig extends ActionConfig {
+    spam?: boolean
 }
 
 /**
  * Remove the Activity
  * */
 export interface RemoveActionJson extends RemoveActionConfig, ActionJson {
-kind: 'remove'
+    kind: 'remove'
 }

src/Action/ReportAction.ts

@@ -35,6 +35,7 @@ export class ReportAction extends Action {
             await item.report({reason: truncatedContent});
             // due to reddit not updating this in response (maybe)?? just increment stale activity
             item.num_reports++;
+            await this.resources.resetCacheForItem(item);
             touchedEntities.push(item);
         }
 

src/Action/SubmissionAction/FlairAction.ts

@@ -33,16 +33,26 @@ export class FlairAction extends Action {
         if(this.css !== '') {
             flairParts.push(`CSS: ${this.css}`);
         }
+        if(this.flair_template_id !== '') {
+            flairParts.push(`Template: ${this.flair_template_id}`);
+        }
         const flairSummary = flairParts.length === 0 ? 'No flair (unflaired)' : flairParts.join(' | ');
         this.logger.verbose(flairSummary);
         if (item instanceof Submission) {
             if(!this.dryRun) {
                 if (this.flair_template_id) {
-                    await item.selectFlair({flair_template_id: this.flair_template_id}).then(() => {});
+                    // typings are wrong for this function, flair_template_id should be accepted
+                    // assignFlair uses /api/flair (mod endpoint)
+                    // selectFlair uses /api/selectflair (self endpoint for user to choose their own flair for submission)
+                    // @ts-ignore
+                    await item.assignFlair({flair_template_id: this.flair_template_id}).then(() => {});
+                    item.link_flair_template_id = this.flair_template_id;
                 } else {
                     await item.assignFlair({text: this.text, cssClass: this.css}).then(() => {});
+                    item.link_flair_css_class = this.css;
+                    item.link_flair_text = this.text;
                 }
-
+                await this.resources.resetCacheForItem(item);
             }
         } else {
             this.logger.warn('Cannot flair Comment');

src/Action/UserFlairAction.ts

@@ -12,7 +12,7 @@ export class UserFlairAction extends Action {
     super(options);
 
     this.text = options.text === null || options.text === '' ? undefined : options.text;
-    this.css = options.css === null || options.text === '' ? undefined : options.text;
+    this.css = options.css === null || options.css === '' ? undefined : options.css;
     this.flair_template_id = options.flair_template_id === null || options.flair_template_id === '' ? undefined : options.flair_template_id;
   }
 
@@ -50,6 +50,7 @@ export class UserFlairAction extends Action {
             flairTemplateId: this.flair_template_id,
             username: item.author.name,
           });
+          item.author_flair_template_id = this.flair_template_id
         } catch (err: any) {
           this.logger.error('Either the flair template ID is incorrect or you do not have permission to access it.');
           throw err;
@@ -57,6 +58,9 @@ export class UserFlairAction extends Action {
       } else if (this.text === undefined && this.css === undefined) {
         // @ts-ignore
         await item.subreddit.deleteUserFlair(item.author.name);
+        item.author_flair_css_class = null;
+        item.author_flair_text = null;
+        item.author_flair_template_id = null;
       } else {
         // @ts-ignore
         await item.author.assignFlair({
@@ -64,7 +68,11 @@ export class UserFlairAction extends Action {
           cssClass: this.css,
           text: this.text,
         });
+        item.author_flair_text = this.text ?? null;
+        item.author_flair_css_class = this.css ?? null;
       }
+      await this.resources.resetCacheForItem(item);
+      await this.resources.resetCacheForItem(item.author);
     }
 
     return {

src/Action/index.ts

@@ -1,12 +1,13 @@
 import {Comment, Submission} from "snoowrap";
 import {Logger} from "winston";
 import {RuleResult} from "../Rule";
-import {SubredditResources} from "../Subreddit/SubredditResources";
+import {checkAuthorFilter, SubredditResources} from "../Subreddit/SubredditResources";
 import {ActionProcessResult, ActionResult, ChecksActivityState, TypedActivityStates} from "../Common/interfaces";
 import Author, {AuthorOptions} from "../Author/Author";
 import {mergeArr} from "../util";
 import LoggedError from "../Utils/LoggedError";
 import {ExtendedSnoowrap} from '../Utils/SnoowrapClients';
+import {ErrorWithCause} from "pony-cause";
 
 export abstract class Action {
     name?: string;
@@ -28,6 +29,7 @@ export abstract class Action {
             subredditName,
             dryRun = false,
             authorIs: {
+                excludeCondition = 'OR',
                 include = [],
                 exclude = [],
             } = {},
@@ -42,6 +44,7 @@ export abstract class Action {
         this.logger = logger.child({labels: [`Action ${this.getActionUniqueName()}`]}, mergeArr);
 
         this.authorIs = {
+            excludeCondition,
             exclude: exclude.map(x => new Author(x)),
             include: include.map(x => new Author(x)),
         }
@@ -72,27 +75,10 @@ export abstract class Action {
                 actRes.runReason = `Activity did not pass 'itemIs' test, Action not run`;
                 return actRes;
             }
-            if (this.authorIs.include !== undefined && this.authorIs.include.length > 0) {
-                for (const auth of this.authorIs.include) {
-                    if (await this.resources.testAuthorCriteria(item, auth)) {
-                        actRes.run = true;
-                        const results = await this.process(item, ruleResults, runtimeDryrun);
-                        return {...actRes, ...results};
-                    }
-                }
-                this.logger.verbose('Inclusive author criteria not matched, Action not run');
-                actRes.runReason = 'Inclusive author criteria not matched';
-                return actRes;
-            } else if (this.authorIs.exclude !== undefined && this.authorIs.exclude.length > 0) {
-                for (const auth of this.authorIs.exclude) {
-                    if (await this.resources.testAuthorCriteria(item, auth, false)) {
-                        actRes.run = true;
-                        const results = await this.process(item, ruleResults, runtimeDryrun);
-                        return {...actRes, ...results};
-                    }
-                }
-                this.logger.verbose('Exclusive author criteria not matched, Action not run');
-                actRes.runReason = 'Exclusive author criteria not matched';
+            const [authFilterResult, authFilterType] = await checkAuthorFilter(item, this.authorIs, this.resources, this.logger);
+            if(!authFilterResult) {
+                this.logger.verbose(`${authFilterType} author criteria not matched, Action not run`);
+                actRes.runReason = `${authFilterType} author criteria not matched`;
                 return actRes;
             }
 
@@ -101,7 +87,8 @@ export abstract class Action {
             return {...actRes, ...results};
         } catch (err: any) {
             if(!(err instanceof LoggedError)) {
-                this.logger.error(`Encountered error while running`, err);
+                const actionError = new ErrorWithCause('Action did not run successfully due to unexpected error', {cause: err});
+                this.logger.error(actionError);
             }
             actRes.success = false;
             actRes.result = err.message;

src/App.ts

@@ -1,7 +1,7 @@
 import winston, {Logger} from "winston";
 import dayjs, {Dayjs} from "dayjs";
 import {getLogger} from "./Utils/loggerFactory";
-import {Invokee, OperatorConfig} from "./Common/interfaces";
+import {Invokee, OperatorConfig, OperatorConfigWithFileContext, OperatorFileConfig} from "./Common/interfaces";
 import Bot from "./Bot";
 import LoggedError from "./Utils/LoggedError";
 import {sleep} from "./util";
@@ -14,7 +14,10 @@ export class App {
 
     error: any;
 
-    constructor(config: OperatorConfig) {
+    config: OperatorConfig;
+    fileConfig: OperatorFileConfig;
+
+    constructor(config: OperatorConfigWithFileContext) {
         const {
             operator: {
                 name,
@@ -23,6 +26,11 @@ export class App {
             bots = [],
         } = config;
 
+        const {fileConfig, ...rest} = config;
+
+        this.config = rest;
+        this.fileConfig = fileConfig;
+
         this.logger = getLogger(config.logging);
 
         this.logger.info(`Operators: ${name.length === 0 ? 'None Specified' : name.join(', ')}`)

src/Author/Author.ts

@@ -1,5 +1,5 @@
 import {UserNoteCriteria} from "../Rule";
-import {CompareValue, CompareValueOrPercent, DurationComparor} from "../Common/interfaces";
+import {CompareValue, CompareValueOrPercent, DurationComparor, JoinOperands} from "../Common/interfaces";
 import {parseStringToRegex} from "../util";
 
 /**
@@ -12,7 +12,17 @@ export interface AuthorOptions {
      * */
     include?: AuthorCriteria[];
     /**
-     * Only runs if `include` is not present. Will "pass" if any of set of the AuthorCriteria **does not** pass
+     * * OR => if ANY exclude condition "does not" pass then the exclude test passes
+     * * AND => if ALL exclude conditions "do not" pass then the exclude test passes
+     *
+     * Defaults to OR
+     * @default OR
+     * */
+    excludeCondition?: JoinOperands
+    /**
+     * Only runs if `include` is not present. Each AuthorCriteria is comprised of conditions that the Author being checked must "not" pass. See excludeCondition for set behavior
+     *
+     * EX: `isMod: true, name: Automoderator` => Will pass if the Author IS NOT a mod and IS NOT named Automoderator
      * */
     exclude?: AuthorCriteria[];
 }
@@ -36,15 +46,20 @@ export interface AuthorCriteria {
      * */
     name?: string[],
     /**
-     * A list of (user) flair css class values from the subreddit to match against
+     * A (user) flair css class (or list of) from the subreddit to match against
      * @examples ["red"]
      * */
-    flairCssClass?: string[],
+    flairCssClass?: string | string[],
     /**
-     * A list of (user) flair text values from the subreddit to match against
+     * A (user) flair text value (or list of) from the subreddit to match against
      * @examples ["Approved"]
      * */
-    flairText?: string[],
+    flairText?: string | string[],
+
+    /**
+     * A (user) flair template id (or list of) from the subreddit to match against
+     * */
+    flairTemplate?: string | string[]
     /**
      * Is the author a moderator?
      * */
@@ -136,8 +151,12 @@ export class Author implements AuthorCriteria {
 
     constructor(options: AuthorCriteria) {
         this.name = options.name;
-        this.flairCssClass = options.flairCssClass;
-        this.flairText = options.flairText;
+        if(options.flairCssClass !== undefined) {
+            this.flairCssClass = typeof options.flairCssClass === 'string' ? [options.flairCssClass] : options.flairCssClass;
+        }
+        if(options.flairText !== undefined) {
+            this.flairText = typeof options.flairText === 'string' ? [options.flairText] : options.flairText;
+        }
         this.isMod = options.isMod;
         this.userNotes = options.userNotes;
         this.age = options.age;

src/Bot/index.ts

@@ -3,36 +3,48 @@ import {Logger} from "winston";
 import dayjs, {Dayjs} from "dayjs";
 import {Duration} from "dayjs/plugin/duration";
 import EventEmitter from "events";
-import {BotInstanceConfig, Invokee, PAUSED, RUNNING, STOPPED, SYSTEM, USER} from "../Common/interfaces";
+import {
+    BotInstanceConfig,
+    FilterCriteriaDefaults,
+    Invokee, LogInfo,
+    PAUSED,
+    PollOn,
+    RUNNING,
+    STOPPED,
+    SYSTEM,
+    USER
+} from "../Common/interfaces";
 import {
     createRetryHandler,
-    formatNumber,
+    formatNumber, getExceptionMessage, getUserAgent,
     mergeArr,
     parseBool,
-    parseDuration,
-    parseSubredditName,
+    parseDuration, parseMatchMessage,
+    parseSubredditName, RetryOptions,
     sleep,
     snooLogWrapper
 } from "../util";
 import {Manager} from "../Subreddit/Manager";
 import {ExtendedSnoowrap, ProxiedSnoowrap} from "../Utils/SnoowrapClients";
-import {ModQueueStream, UnmoderatedStream} from "../Subreddit/Streams";
+import {CommentStream, ModQueueStream, SPoll, SubmissionStream, UnmoderatedStream} from "../Subreddit/Streams";
 import {BotResourcesManager} from "../Subreddit/SubredditResources";
 import LoggedError from "../Utils/LoggedError";
 import pEvent from "p-event";
-import SimpleError from "../Utils/SimpleError";
-import {isRateLimitError, isStatusError} from "../Utils/Errors";
+import {SimpleError, isRateLimitError, isRequestError, isScopeError, isStatusError, CMError} from "../Utils/Errors";
+import {ErrorWithCause} from "pony-cause";
 
 
 class Bot {
 
     client!: ExtendedSnoowrap;
     logger!: Logger;
+    logs: LogInfo[] = [];
     wikiLocation: string;
     dryRun?: true | undefined;
     running: boolean = false;
     subreddits: string[];
     excludeSubreddits: string[];
+    filterCriteriaDefaults?: FilterCriteriaDefaults
     subManagers: Manager[] = [];
     heartbeatInterval: number;
     nextHeartbeat: Dayjs = dayjs();
@@ -43,6 +55,7 @@ class Bot {
     nannyMode?: 'soft' | 'hard';
     nannyRunning: boolean = false;
     nextNannyCheck: Dayjs = dayjs().add(10, 'second');
+    sharedStreamRetryHandler: Function;
     nannyRetryHandler: Function;
     managerRetryHandler: Function;
     nextExpiration: Dayjs = dayjs();
@@ -51,7 +64,7 @@ class Bot {
     botAccount?: string;
     maxWorkers: number;
     startedAt: Dayjs = dayjs();
-    sharedModqueue: boolean = false;
+    sharedStreams: PollOn[] = [];
     streamListedOnce: string[] = [];
 
     stagger: number;
@@ -78,6 +91,7 @@ class Bot {
         const {
             notifications,
             name,
+            filterCriteriaDefaults,
             subreddits: {
                 names = [],
                 exclude = [],
@@ -85,6 +99,7 @@ class Bot {
                 dryRun,
                 heartbeatInterval,
             },
+            userAgent,
             credentials: {
                 reddit: {
                     clientId,
@@ -98,7 +113,7 @@ class Bot {
                 debug,
             },
             polling: {
-                sharedMod,
+                shared = [],
                 stagger = 2000,
             },
             queue: {
@@ -123,7 +138,8 @@ class Bot {
         this.hardLimit = hardLimit;
         this.wikiLocation = wikiConfig;
         this.heartbeatInterval = heartbeatInterval;
-        this.sharedModqueue = sharedMod;
+        this.filterCriteriaDefaults = filterCriteriaDefaults;
+        this.sharedStreams = shared;
         if(name !== undefined) {
             this.botName = name;
         }
@@ -137,6 +153,12 @@ class Bot {
             }
         }, mergeArr);
 
+        this.logger.stream().on('log', (log: LogInfo) => {
+            if(log.bot !== undefined && log.bot === this.getBotName() && log.subreddit === undefined) {
+                this.logs = [log, ...this.logs].slice(0, 301);
+            }
+        });
+
         let mw = maxWorkers;
         if(maxWorkers < 1) {
             this.logger.warn(`Max queue workers must be greater than or equal to 1 (Specified: ${maxWorkers})`);
@@ -152,7 +174,9 @@ class Bot {
         this.excludeSubreddits = exclude.map(parseSubredditName);
 
         let creds: any = {
-            get userAgent() { return getUserName() },
+            get userAgent() {
+                return getUserAgent(`web:contextBot:{VERSION}{FRAG}:BOT-${getBotName()}`, userAgent)
+            },
             clientId,
             clientSecret,
             refreshToken,
@@ -190,56 +214,12 @@ class Bot {
             }
         }
 
-        const retryHandler = createRetryHandler({maxRequestRetry: 8, maxOtherRetry: 2}, this.logger);
+        this.sharedStreamRetryHandler = createRetryHandler({maxRequestRetry: 8, maxOtherRetry: 2}, this.logger);
         this.nannyRetryHandler = createRetryHandler({maxRequestRetry: 5, maxOtherRetry: 1}, this.logger);
         this.managerRetryHandler = createRetryHandler({maxRequestRetry: 8, maxOtherRetry: 8, waitOnRetry: false, clearRetryCountAfter: 2}, this.logger);
 
         this.stagger = stagger ?? 2000;
 
-        const modStreamErrorListener = (name: string) => async (err: any) => {
-            this.logger.error(`Polling error occurred on stream ${name.toUpperCase()}`, err);
-            const shouldRetry = await retryHandler(err);
-            if(shouldRetry) {
-                defaultUnmoderatedStream.startInterval();
-            } else {
-                for(const m of this.subManagers) {
-                    if(m.modStreamCallbacks.size > 0) {
-                        m.notificationManager.handle('runStateChanged', `${name.toUpperCase()} Polling Stopped`, 'Encountered too many errors from Reddit while polling. Will try to restart on next heartbeat.');
-                    }
-                }
-                this.logger.error(`Mod stream ${name.toUpperCase()} encountered too many errors while polling. Will try to restart on next heartbeat.`);
-            }
-        }
-
-        const modStreamListingListener = (name: string) => async (listing: (Comment|Submission)[]) => {
-            // dole out in order they were received
-            if(!this.streamListedOnce.includes(name)) {
-                this.streamListedOnce.push(name);
-                return;
-            }
-            for(const i of listing) {
-                const foundManager = this.subManagers.find(x => x.subreddit.display_name === i.subreddit.display_name && x.modStreamCallbacks.get(name) !== undefined);
-                if(foundManager !== undefined) {
-                    foundManager.modStreamCallbacks.get(name)(i);
-                    if(stagger !== undefined) {
-                        await sleep(stagger);
-                    }
-                }
-            }
-
-        }
-
-        const defaultUnmoderatedStream = new UnmoderatedStream(this.client, {subreddit: 'mod', limit: 100, clearProcessed: { size: 100, retain: 100 }});
-        // @ts-ignore
-        defaultUnmoderatedStream.on('error', modStreamErrorListener('unmoderated'));
-        defaultUnmoderatedStream.on('listing', modStreamListingListener('unmoderated'));
-        const defaultModqueueStream = new ModQueueStream(this.client, {subreddit: 'mod', limit: 100, clearProcessed: { size: 100, retain: 100 }});
-        // @ts-ignore
-        defaultModqueueStream.on('error', modStreamErrorListener('modqueue'));
-        defaultModqueueStream.on('listing', modStreamListingListener('modqueue'));
-        this.cacheManager.modStreams.set('unmoderated', defaultUnmoderatedStream);
-        this.cacheManager.modStreams.set('modqueue', defaultModqueueStream);
-
         process.on('uncaughtException', (e) => {
             this.error = e;
         });
@@ -263,6 +243,37 @@ class Bot {
         });
     }
 
+    createSharedStreamErrorListener = (name: string) => async (err: any) => {
+        const shouldRetry = await this.sharedStreamRetryHandler(err);
+        if(shouldRetry) {
+            (this.cacheManager.modStreams.get(name) as SPoll<any>).startInterval(false, 'Within retry limits');
+        } else {
+            for(const m of this.subManagers) {
+                if(m.sharedStreamCallbacks.size > 0) {
+                    m.notificationManager.handle('runStateChanged', `${name.toUpperCase()} Polling Stopped`, 'Encountered too many errors from Reddit while polling. Will try to restart on next heartbeat.');
+                }
+            }
+            this.logger.error(`Mod stream ${name.toUpperCase()} encountered too many errors while polling. Will try to restart on next heartbeat.`);
+        }
+    }
+
+    createSharedStreamListingListener = (name: string) => async (listing: (Comment|Submission)[]) => {
+        // dole out in order they were received
+        if(!this.streamListedOnce.includes(name)) {
+            this.streamListedOnce.push(name);
+            return;
+        }
+        for(const i of listing) {
+            const foundManager = this.subManagers.find(x => x.subreddit.display_name === i.subreddit.display_name && x.sharedStreamCallbacks.get(name) !== undefined && x.eventsState.state === RUNNING);
+            if(foundManager !== undefined) {
+                foundManager.sharedStreamCallbacks.get(name)(i);
+                if(this.stagger !== undefined) {
+                    await sleep(this.stagger);
+                }
+            }
+        }
+    }
+
     async onTerminate(reason = 'The application was shutdown') {
         for(const m of this.subManagers) {
             await m.notificationManager.handle('runStateChanged', 'Application Shutdown', reason);
@@ -278,22 +289,16 @@ class Bot {
             if (initial) {
                 this.logger.error('An error occurred while trying to initialize the Reddit API Client which would prevent the entire application from running.');
             }
-            if (err.name === 'StatusCodeError') {
-                const authHeader = err.response.headers['www-authenticate'];
-                if (authHeader !== undefined && authHeader.includes('insufficient_scope')) {
-                    this.logger.error('Reddit responded with a 403 insufficient_scope. Please ensure you have chosen the correct scopes when authorizing your account.');
-                } else if (err.statusCode === 401) {
-                    this.logger.error('It is likely a credential is missing or incorrect. Check clientId, clientSecret, refreshToken, and accessToken');
-                } else if(err.statusCode === 400) {
-                    this.logger.error('Credentials may have been invalidated due to prior behavior. The error message may contain more information.');
-                }
-                this.logger.error(`Error Message: ${err.message}`);
-            } else {
-                this.logger.error(err);
-            }
-            this.error = `Error occurred while testing Reddit API client: ${err.message}`;
-            err.logged = true;
-            throw err;
+            const hint = getExceptionMessage(err, {
+                401: 'Likely a credential is missing or incorrect. Check clientId, clientSecret, refreshToken, and accessToken',
+                400: 'Credentials may have been invalidated manually or by reddit due to behavior',
+            });
+            let msg = `Error occurred while testing Reddit API client${hint !== undefined ? `: ${hint}` : ''}`;
+            this.error = msg;
+            const clientError = new CMError(msg, {cause: err});
+            clientError.logged = true;
+            this.logger.error(clientError);
+            throw clientError;
         }
     }
 
@@ -316,7 +321,7 @@ class Bot {
         while(!subListing.isFinished) {
             subListing = await subListing.fetchMore({amount: 100});
         }
-        availSubs = subListing;
+        availSubs = subListing.filter(x => x.display_name !== `u_${user.name}`);
 
         this.logger.info(`u/${user.name} is a moderator of these subreddits: ${availSubs.map(x => x.display_name_prefixed).join(', ')}`);
 
@@ -336,38 +341,166 @@ class Bot {
             }
         } else {
             if(this.excludeSubreddits.length > 0) {
-                this.logger.info(`Will run on all moderated subreddits but user-defined excluded: ${this.excludeSubreddits.join(', ')}`);
+                this.logger.info(`Will run on all moderated subreddits but own profile and user-defined excluded: ${this.excludeSubreddits.join(', ')}`);
                 const normalExcludes = this.excludeSubreddits.map(x => x.toLowerCase());
                 subsToRun = availSubs.filter(x => !normalExcludes.includes(x.display_name.toLowerCase()));
             } else {
                 this.logger.info(`No user-defined subreddit constraints detected, will run on all moderated subreddits EXCEPT own profile (${this.botAccount})`);
-                subsToRun = availSubs.filter(x => x.display_name_prefixed !== this.botAccount);
+                subsToRun = availSubs;
             }
         }
 
         // get configs for subs we want to run on and build/validate them
         for (const sub of subsToRun) {
             try {
-                this.subManagers.push(await this.createManager(sub));
+                this.subManagers.push(this.createManager(sub));
             } catch (err: any) {
 
             }
         }
-    }
+        for(const m of this.subManagers) {
+            try {
+                await this.initManager(m);
+            } catch (err: any) {
 
-    async createManager(sub: Subreddit): Promise<Manager> {
-        const manager = new Manager(sub, this.client, this.logger, this.cacheManager, {dryRun: this.dryRun, sharedModqueue: this.sharedModqueue, wikiLocation: this.wikiLocation, botName: this.botName as string, maxWorkers: this.maxWorkers});
-        try {
-            await manager.parseConfiguration('system', true, {suppressNotification: true});
-        } catch (err: any) {
-            if (!(err instanceof LoggedError)) {
-                this.logger.error(`Config was not valid:`, {subreddit: sub.display_name_prefixed});
-                this.logger.error(err, {subreddit: sub.display_name_prefixed});
-                err.logged = true;
             }
         }
+
+        this.parseSharedStreams();
+    }
+
+    parseSharedStreams() {
+
+        const sharedCommentsSubreddits = !this.sharedStreams.includes('newComm') ? [] : this.subManagers.filter(x => x.isPollingShared('newComm')).map(x => x.subreddit.display_name);
+        if (sharedCommentsSubreddits.length > 0) {
+            const stream = this.cacheManager.modStreams.get('newComm');
+            if (stream === undefined || stream.subreddit !== sharedCommentsSubreddits.join('+')) {
+                let processed;
+                if (stream !== undefined) {
+                    this.logger.info('Restarting SHARED COMMENT STREAM due to a subreddit config change');
+                    stream.end('Replacing with a new stream with updated subreddits');
+                    processed = stream.processed;
+                }
+                if (sharedCommentsSubreddits.length > 100) {
+                    this.logger.warn(`SHARED COMMENT STREAM => Reddit can only combine 100 subreddits for getting new Comments but this bot has ${sharedCommentsSubreddits.length}`);
+                }
+                const defaultCommentStream = new CommentStream(this.client, {
+                    subreddit: sharedCommentsSubreddits.join('+'),
+                    limit: 100,
+                    enforceContinuity: true,
+                    logger: this.logger,
+                    processed,
+                    label: 'Shared Polling'
+                });
+                // @ts-ignore
+                defaultCommentStream.on('error', this.createSharedStreamErrorListener('newComm'));
+                defaultCommentStream.on('listing', this.createSharedStreamListingListener('newComm'));
+                this.cacheManager.modStreams.set('newComm', defaultCommentStream);
+            }
+        } else {
+            const stream = this.cacheManager.modStreams.get('newComm');
+            if (stream !== undefined) {
+                stream.end('Determined no managers are listening on shared stream parsing');
+            }
+        }
+
+        const sharedSubmissionsSubreddits = !this.sharedStreams.includes('newSub') ? [] : this.subManagers.filter(x => x.isPollingShared('newSub')).map(x => x.subreddit.display_name);
+        if (sharedSubmissionsSubreddits.length > 0) {
+            const stream = this.cacheManager.modStreams.get('newSub');
+            if (stream === undefined || stream.subreddit !== sharedSubmissionsSubreddits.join('+')) {
+                let processed;
+                if (stream !== undefined) {
+                    this.logger.info('Restarting SHARED SUBMISSION STREAM due to a subreddit config change');
+                    stream.end('Replacing with a new stream with updated subreddits');
+                    processed = stream.processed;
+                }
+                if (sharedSubmissionsSubreddits.length > 100) {
+                    this.logger.warn(`SHARED SUBMISSION STREAM => Reddit can only combine 100 subreddits for getting new Submissions but this bot has ${sharedSubmissionsSubreddits.length}`);
+                }
+                const defaultSubStream = new SubmissionStream(this.client, {
+                    subreddit: sharedSubmissionsSubreddits.join('+'),
+                    limit: 100,
+                    enforceContinuity: true,
+                    logger: this.logger,
+                    processed,
+                    label: 'Shared Polling'
+                });
+                // @ts-ignore
+                defaultSubStream.on('error', this.createSharedStreamErrorListener('newSub'));
+                defaultSubStream.on('listing', this.createSharedStreamListingListener('newSub'));
+                this.cacheManager.modStreams.set('newSub', defaultSubStream);
+            }
+        } else {
+            const stream = this.cacheManager.modStreams.get('newSub');
+            if (stream !== undefined) {
+                stream.end('Determined no managers are listening on shared stream parsing');
+            }
+        }
+
+        const isUnmoderatedShared = !this.sharedStreams.includes('unmoderated') ? false : this.subManagers.some(x => x.isPollingShared('unmoderated'));
+        const unmoderatedstream = this.cacheManager.modStreams.get('unmoderated');
+        if (isUnmoderatedShared && unmoderatedstream === undefined) {
+            const defaultUnmoderatedStream = new UnmoderatedStream(this.client, {
+                subreddit: 'mod',
+                limit: 100,
+                logger: this.logger,
+                label: 'Shared Polling'
+            });
+            // @ts-ignore
+            defaultUnmoderatedStream.on('error', this.createSharedStreamErrorListener('unmoderated'));
+            defaultUnmoderatedStream.on('listing', this.createSharedStreamListingListener('unmoderated'));
+            this.cacheManager.modStreams.set('unmoderated', defaultUnmoderatedStream);
+        } else if (!isUnmoderatedShared && unmoderatedstream !== undefined) {
+            unmoderatedstream.end('Determined no managers are listening on shared stream parsing');
+        }
+
+        const isModqueueShared = !this.sharedStreams.includes('modqueue') ? false : this.subManagers.some(x => x.isPollingShared('modqueue'));
+        const modqueuestream = this.cacheManager.modStreams.get('modqueue');
+        if (isModqueueShared && modqueuestream === undefined) {
+            const defaultModqueueStream = new ModQueueStream(this.client, {
+                subreddit: 'mod',
+                limit: 100,
+                logger: this.logger,
+                label: 'Shared Polling'
+            });
+            // @ts-ignore
+            defaultModqueueStream.on('error', this.createSharedStreamErrorListener('modqueue'));
+            defaultModqueueStream.on('listing', this.createSharedStreamListingListener('modqueue'));
+            this.cacheManager.modStreams.set('modqueue', defaultModqueueStream);
+        } else if (isModqueueShared && modqueuestream !== undefined) {
+            modqueuestream.end('Determined no managers are listening on shared stream parsing');
+        }
+    }
+
+    async initManager(manager: Manager) {
+        try {
+            await manager.parseConfiguration('system', true, {suppressNotification: true, suppressChangeEvent: true});
+        } catch (err: any) {
+            if(err.logged !== true) {
+                const normalizedError = new ErrorWithCause(`Bot could not start manager because config was not valid`, {cause: err});
+                // @ts-ignore
+                this.logger.error(normalizedError, {subreddit: manager.subreddit.display_name_prefixed});
+            } else {
+                this.logger.error('Bot could not start manager because config was not valid', {subreddit: manager.subreddit.display_name_prefixed});
+            }
+        }
+    }
+
+    createManager(sub: Subreddit): Manager {
+        const manager = new Manager(sub, this.client, this.logger, this.cacheManager, {
+            dryRun: this.dryRun,
+            sharedStreams: this.sharedStreams,
+            wikiLocation: this.wikiLocation,
+            botName: this.botName as string,
+            maxWorkers: this.maxWorkers,
+            filterCriteriaDefaults: this.filterCriteriaDefaults,
+        });
         // all errors from managers will count towards bot-level retry count
         manager.on('error', async (err) => await this.panicOnRetries(err));
+        manager.on('configChange', async () => {
+           this.parseSharedStreams();
+           await this.runSharedStreams(false);
+        });
         return manager;
     }
 
@@ -404,11 +537,12 @@ class Bot {
                 const sub = await this.client.getSubreddit(name);
                 this.logger.info(`Attempting to add manager for r/${name}`);
                 try {
-                    const manager = await this.createManager(sub);
+                    const manager = this.createManager(sub);
                     this.logger.info(`Starting manager for r/${name}`);
                     this.subManagers.push(manager);
+                    await this.initManager(manager);
                     await manager.start('system', {reason: 'Caused by creation due to moderator invite'});
-                    await this.runModStreams();
+                    await this.runSharedStreams();
                 } catch (err: any) {
                     if (!(err instanceof LoggedError)) {
                         this.logger.error(err);
@@ -426,14 +560,14 @@ class Bot {
         }
     }
 
-    async runModStreams(notify = false) {
+    async runSharedStreams(notify = false) {
         for(const [k,v] of this.cacheManager.modStreams) {
-            if(!v.running && this.subManagers.some(x => x.modStreamCallbacks.get(k) !== undefined)) {
+            if(!v.running && this.subManagers.some(x => x.sharedStreamCallbacks.get(k) !== undefined)) {
                 v.startInterval();
-                this.logger.info(`Starting default ${k.toUpperCase()} mod stream`);
+                this.logger.info(`Starting ${k.toUpperCase()} shared polling`);
                 if(notify) {
                     for(const m of this.subManagers) {
-                        if(m.modStreamCallbacks.size > 0) {
+                        if(m.sharedStreamCallbacks.size > 0) {
                             await m.notificationManager.handle('runStateChanged', `${k.toUpperCase()} Polling Started`, 'Polling was successfully restarted on heartbeat.');
                         }
                     }
@@ -457,7 +591,7 @@ class Bot {
             }
         }
 
-        await this.runModStreams();
+        await this.runSharedStreams();
 
         this.nextNannyCheck = dayjs().add(10, 'second');
         this.nextHeartbeat = dayjs().add(this.heartbeatInterval, 'second');
@@ -537,9 +671,11 @@ class Bot {
                     }
                 }
             } catch (err: any) {
-                this.logger.info('Stopping event polling to prevent activity processing queue from backing up. Will be restarted when config update succeeds.')
-                await s.stopEvents('system', {reason: 'Invalid config will cause events to pile up in queue. Will be restarted when config update succeeds (next heartbeat).'});
-                if(!(err instanceof LoggedError)) {
+                if(s.eventsState.state === RUNNING) {
+                    this.logger.info('Stopping event polling to prevent activity processing queue from backing up. Will be restarted when config update succeeds.')
+                    await s.stopEvents('system', {reason: 'Invalid config will cause events to pile up in queue. Will be restarted when config update succeeds (next heartbeat).'});
+                }
+                if(err.logged !== true) {
                     this.logger.error(err, {subreddit: s.displayLabel});
                 }
                 if(this.nextHeartbeat !== undefined) {
@@ -547,7 +683,7 @@ class Bot {
                 }
             }
         }
-        await this.runModStreams(true);
+        await this.runSharedStreams(true);
     }
 
     async runApiNanny() {
@@ -615,6 +751,10 @@ class Bot {
                     m.notificationManager.handle('runStateChanged', 'Hard Limit Triggered', `Hard Limit of ${this.hardLimit} hit (API Remaining: ${this.client.ratelimitRemaining}). Subreddit event polling has been paused.`, 'system', 'warn');
                 }
 
+                for(const [k,v] of this.cacheManager.modStreams) {
+                    v.end('Hard limit cutoff');
+                }
+
                 this.nannyMode = 'hard';
                 return;
             }
@@ -681,6 +821,7 @@ class Bot {
                         await m.startEvents('system', {reason: 'API Nanny has been turned off due to better API conditions'});
                     }
                 }
+                await this.runSharedStreams(true);
                 this.nannyMode = undefined;
             }
 

src/Check/index.ts

@@ -28,9 +28,11 @@ import * as RuleSchema from '../Schema/Rule.json';
 import * as RuleSetSchema from '../Schema/RuleSet.json';
 import * as ActionSchema from '../Schema/Action.json';
 import {ActionObjectJson, RuleJson, RuleObjectJson, ActionJson as ActionTypeJson} from "../Common/types";
-import {SubredditResources} from "../Subreddit/SubredditResources";
+import {checkAuthorFilter, SubredditResources} from "../Subreddit/SubredditResources";
 import {Author, AuthorCriteria, AuthorOptions} from '..';
 import {ExtendedSnoowrap} from '../Utils/SnoowrapClients';
+import {isRateLimitError} from "../Utils/Errors";
+import {ErrorWithCause} from "pony-cause";
 
 const checkLogName = truncateStringToLength(25);
 
@@ -43,10 +45,7 @@ export abstract class Check implements ICheck {
     rules: Array<RuleSet | Rule> = [];
     logger: Logger;
     itemIs: TypedActivityStates;
-    authorIs: {
-        include: AuthorCriteria[],
-        exclude: AuthorCriteria[]
-    };
+    authorIs: AuthorOptions;
     cacheUserResult: Required<UserResultCacheOptions>;
     dryRun?: boolean;
     notifyOnTrigger: boolean;
@@ -69,6 +68,7 @@ export abstract class Check implements ICheck {
             itemIs = [],
             authorIs: {
                 include = [],
+                excludeCondition,
                 exclude = [],
             } = {},
             dryRun,
@@ -89,6 +89,7 @@ export abstract class Check implements ICheck {
         this.condition = condition;
         this.itemIs = itemIs;
         this.authorIs = {
+            excludeCondition,
             exclude: exclude.map(x => new Author(x)),
             include: include.map(x => new Author(x)),
         }
@@ -159,7 +160,7 @@ export abstract class Check implements ICheck {
         runStats.push(`${this.actions.length} Actions`);
         // not sure if this should be info or verbose
         this.logger.info(`=${this.enabled ? 'Enabled' : 'Disabled'}= ${type.toUpperCase()} (${this.condition})${this.notifyOnTrigger ? ' ||Notify on Trigger|| ' : ''} => ${runStats.join(' | ')}${this.description !== undefined ? ` => ${this.description}` : ''}`);
-        if (this.rules.length === 0 && this.itemIs.length === 0 && this.authorIs.exclude.length === 0 && this.authorIs.include.length === 0) {
+        if (this.rules.length === 0 && this.itemIs.length === 0 && this.authorIs.exclude?.length === 0 && this.authorIs.include?.length === 0) {
             this.logger.warn('No rules, item tests, or author test found -- this check will ALWAYS PASS!');
         }
         let ruleSetIndex = 1;
@@ -202,30 +203,9 @@ export abstract class Check implements ICheck {
                 this.logger.verbose(`${FAIL} => Item did not pass 'itemIs' test`);
                 return [false, allRuleResults];
             }
-            let authorPass = null;
-            if (this.authorIs.include !== undefined && this.authorIs.include.length > 0) {
-                for (const auth of this.authorIs.include) {
-                    if (await this.resources.testAuthorCriteria(item, auth)) {
-                        authorPass = true;
-                        break;
-                    }
-                }
-                if (!authorPass) {
-                    this.logger.verbose(`${FAIL} => Inclusive author criteria not matched`);
-                    return Promise.resolve([false, allRuleResults]);
-                }
-            }
-            if (authorPass === null && this.authorIs.exclude !== undefined && this.authorIs.exclude.length > 0) {
-                for (const auth of this.authorIs.exclude) {
-                    if (await this.resources.testAuthorCriteria(item, auth, false)) {
-                        authorPass = true;
-                        break;
-                    }
-                }
-                if (!authorPass) {
-                    this.logger.verbose(`${FAIL} =>  Exclusive author criteria not matched`);
-                    return Promise.resolve([false, allRuleResults]);
-                }
+            const [authFilterResult, authFilterType] = await checkAuthorFilter(item, this.authorIs, this.resources, this.logger);
+            if(!authFilterResult) {
+                return Promise.resolve([false, allRuleResults]);
             }
 
             if (this.rules.length === 0) {
@@ -270,9 +250,7 @@ export abstract class Check implements ICheck {
             this.logger.info(`${PASS} => Rules: ${resultsSummary(allResults, this.condition)}`);
             return [true, allRuleResults];
         } catch (e: any) {
-            e.logged = true;
-            this.logger.warn(`Running rules failed due to uncaught exception`, e);
-            throw e;
+            throw new ErrorWithCause('Running rules failed due to error', {cause: e});
         }
     }
 

src/Common/Config/AbstractConfigDocument.ts

@@ -0,0 +1,27 @@
+import {ConfigFormat} from "../types";
+
+export interface ConfigDocumentInterface<DocumentType> {
+    format: ConfigFormat;
+    parsed: DocumentType
+    //parsingError: Error | string;
+    raw: string;
+    location?: string;
+    toString(): string;
+    toJS(): object;
+}
+
+abstract class AbstractConfigDocument<DocumentType> implements ConfigDocumentInterface<DocumentType> {
+    public abstract format: ConfigFormat;
+    public abstract parsed: DocumentType;
+    //public abstract parsingError: Error | string;
+
+
+    constructor(public raw: string, public location?: string) {
+    }
+
+
+    public abstract toString(): string;
+    public abstract toJS(): object;
+}
+
+export default AbstractConfigDocument;

src/Common/Config/JsonConfigDocument.ts

@@ -0,0 +1,30 @@
+import AbstractConfigDocument from "./AbstractConfigDocument";
+import {stringify, parse} from 'comment-json';
+import JSON5 from 'json5';
+import {ConfigFormat} from "../types";
+import {OperatorJsonConfig} from "../interfaces";
+
+class JsonConfigDocument extends AbstractConfigDocument<OperatorJsonConfig> {
+
+    public parsed: OperatorJsonConfig;
+    protected cleanParsed: OperatorJsonConfig;
+    public format: ConfigFormat;
+
+    public constructor(raw: string, location?: string) {
+        super(raw, location);
+        this.parsed = parse(raw);
+        this.cleanParsed = JSON5.parse(raw);
+        this.format = 'json';
+    }
+
+    public toJS(): OperatorJsonConfig {
+        return this.cleanParsed;
+    }
+
+    public toString(): string {
+        return stringify(this.parsed, null, 1);
+    }
+
+}
+
+export default JsonConfigDocument;

src/Common/Config/Operator/index.ts

@@ -0,0 +1,54 @@
+import YamlConfigDocument from "../YamlConfigDocument";
+import JsonConfigDocument from "../JsonConfigDocument";
+import {YAMLMap, YAMLSeq} from "yaml";
+import {BotInstanceJsonConfig, OperatorJsonConfig} from "../../interfaces";
+import {assign} from 'comment-json';
+
+export interface OperatorConfigDocumentInterface {
+    addBot(botData: BotInstanceJsonConfig): void;
+    toJS(): OperatorJsonConfig;
+}
+
+export class YamlOperatorConfigDocument extends YamlConfigDocument implements OperatorConfigDocumentInterface {
+    addBot(botData: BotInstanceJsonConfig) {
+        const bots = this.parsed.get('bots') as YAMLSeq;
+        if (bots === undefined) {
+            this.parsed.add({key: 'bots', value: [botData]});
+        } else if (botData.name !== undefined) {
+            // overwrite if we find an existing
+            const existingIndex = bots.items.findIndex(x => (x as YAMLMap).get('name') === botData.name);
+            if (existingIndex !== -1) {
+                this.parsed.setIn(['bots', existingIndex], botData);
+            } else {
+                this.parsed.addIn(['bots'], botData);
+            }
+        } else {
+            this.parsed.addIn(['bots'], botData);
+        }
+    }
+
+    toJS(): OperatorJsonConfig  {
+        return super.toJS();
+    }
+}
+
+export class JsonOperatorConfigDocument extends JsonConfigDocument implements OperatorConfigDocumentInterface {
+    addBot(botData: BotInstanceJsonConfig) {
+        if (this.parsed.bots === undefined) {
+            this.parsed.bots = [botData];
+        } else if (botData.name !== undefined) {
+            const existingIndex = this.parsed.bots.findIndex(x => x.name === botData.name);
+            if (existingIndex !== -1) {
+                this.parsed.bots[existingIndex] = assign(this.parsed.bots[existingIndex], botData);
+            } else {
+                this.parsed.bots.push(botData);
+            }
+        } else {
+            this.parsed.bots.push(botData);
+        }
+    }
+
+    toJS(): OperatorJsonConfig {
+        return super.toJS();
+    }
+}

src/Common/Config/YamlConfigDocument.ts

@@ -0,0 +1,24 @@
+import AbstractConfigDocument from "./AbstractConfigDocument";
+import {Document, parseDocument} from 'yaml';
+import {ConfigFormat} from "../types";
+
+class YamlConfigDocument extends AbstractConfigDocument<Document> {
+
+    public parsed: Document;
+    public format: ConfigFormat;
+
+    public constructor(raw: string, location?: string) {
+        super(raw, location);
+        this.parsed = parseDocument(raw);
+        this.format = 'yaml';
+    }
+    public toJS(): object {
+        return this.parsed.toJS();
+    }
+
+    public toString(): string {
+        return this.parsed.toString();
+    }
+}
+
+export default YamlConfigDocument;

src/Common/ImageData.ts

@@ -3,9 +3,9 @@ import {Submission} from "snoowrap/dist/objects";
 import {URL} from "url";
 import {absPercentDifference, getSharpAsync, isValidImageURL} from "../util";
 import sizeOf from "image-size";
-import SimpleError from "../Utils/SimpleError";
 import {Sharp} from "sharp";
 import {blockhash} from "./blockhash/blockhash";
+import {SimpleError} from "../Utils/Errors";
 
 export interface ImageDataOptions {
     width?: number,

src/Common/defaults.ts

@@ -1,4 +1,4 @@
-import {HistoricalStats} from "./interfaces";
+import {HistoricalStats, FilterCriteriaDefaults} from "./interfaces";
 
 export const cacheOptDefaults = {ttl: 60, max: 500, checkPeriod: 600};
 export const cacheTTLDefaults = {authorTTL: 60, userNotesTTL: 300, wikiTTL: 300, submissionTTL: 60, commentTTL: 60, filterCriteriaTTL: 60, subredditTTL: 600, selfTTL: 60};
@@ -29,3 +29,15 @@ export const createHistoricalDefaults = (): HistoricalStats => {
         actionsRun: new Map(),
     };
 }
+
+export const filterCriteriaDefault: FilterCriteriaDefaults = {
+    authorIs: {
+        exclude: [
+            {
+                isMod: true
+            }
+        ]
+    }
+}
+
+export const VERSION = '0.10.12';

src/Common/interfaces.ts

@@ -8,6 +8,14 @@ import {IncomingMessage} from "http";
 import Submission from "snoowrap/dist/objects/Submission";
 import Comment from "snoowrap/dist/objects/Comment";
 import RedditUser from "snoowrap/dist/objects/RedditUser";
+import {AuthorCriteria, AuthorOptions} from "../Author/Author";
+import {ConfigFormat} from "./types";
+import AbstractConfigDocument, {ConfigDocumentInterface} from "./Config/AbstractConfigDocument";
+import {Document as YamlDocument} from 'yaml';
+import {JsonOperatorConfigDocument, YamlOperatorConfigDocument} from "./Config/Operator";
+import {ConsoleTransportOptions} from "winston/lib/winston/transports";
+import {DailyRotateFileTransportOptions} from "winston-daily-rotate-file";
+import {DuplexTransportOptions} from "winston-duplex/dist/DuplexTransport";
 
 /**
  * An ISO 8601 Duration
@@ -489,38 +497,6 @@ export type PollOn = 'unmoderated' | 'modqueue' | 'newSub' | 'newComm';
 export interface PollingOptionsStrong extends PollingOptions {
     limit: number,
     interval: number,
-    clearProcessed: ClearProcessedOptions
-}
-
-/**
- * For very long-running, high-volume subreddits clearing the list of processed activities helps manage memory bloat
- *
- * All of these options have default values based on the limit and/or interval set for polling options on each subreddit stream. They only need to modified if the defaults are not sufficient.
- *
- * If both `after` and `size` are defined whichever is hit first will trigger the list to clear. `after` will be reset after ever clear.
- * */
-export interface ClearProcessedOptions {
-    /**
-     * An interval the processed list should be cleared after.
-     *
-     * * EX `9 days`
-     * * EX `3 months`
-     * * EX `5 minutes`
-     * @pattern ^\s*(?<time>\d+)\s*(?<unit>days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)\s*$
-     * */
-    after?: string,
-    /**
-     * Number of activities found in processed list after which the list should be cleared.
-     *
-     * Defaults to the `limit` value from `PollingOptions`
-     * */
-    size?: number,
-    /**
-     * The number of activities to retain in processed list after clearing.
-     *
-     * Defaults to `limit` value from `PollingOptions`
-     * */
-    retain?: number,
 }
 
 export interface PollingDefaults {
@@ -594,8 +570,6 @@ export interface PollingOptions extends PollingDefaults {
      *
      * */
     pollOn: 'unmoderated' | 'modqueue' | 'newSub' | 'newComm'
-
-    clearProcessed?: ClearProcessedOptions
 }
 
 export interface TTLConfig {
@@ -855,6 +829,13 @@ export interface ManagerOptions {
     notifications?: NotificationConfig
 
     credentials?: ThirdPartyCredentialsJsonConfig
+
+    /**
+     * Set the default filter criteria for all checks. If this property is specified it will override any defaults passed from the bot's config
+     *
+     * Default behavior is to exclude all mods and automoderator from checks
+     * */
+    filterCriteriaDefaults?: FilterCriteriaDefaults
 }
 
 /**
@@ -965,8 +946,13 @@ export interface SubmissionState extends ActivityState {
      * */
     title?: string
 
-    link_flair_text?: string
-    link_flair_css_class?: string
+    link_flair_text?: string | string[]
+    link_flair_css_class?: string | string[]
+    flairTemplate?: string | string[]
+    /**
+     * Is the submission a reddit-hosted image or video?
+     * */
+    isRedditMediaDomain?: boolean
 }
 
 // properties calculated/derived by CM -- not provided as plain values by reddit
@@ -1092,6 +1078,92 @@ export interface RegExResult {
 }
 
 type LogLevel = "error" | "warn" | "info" | "verbose" | "debug";
+
+export type LogConsoleOptions = Pick<ConsoleTransportOptions, 'silent' | 'eol' | 'stderrLevels' | 'consoleWarnLevels'> & {
+    level?: LogLevel
+}
+
+export type LogFileOptions = Omit<DailyRotateFileTransportOptions, 'stream' | 'handleRejections' | 'options' | 'handleExceptions' | 'format' | 'log' | 'logv' | 'close' | 'dirname'> & {
+    level?: LogLevel
+    /**
+     * The absolute path to a directory where rotating log files should be stored.
+     *
+     * * If not present or `null` or `false` no log files will be created
+     * * If `true` logs will be stored at `[working directory]/logs`
+     *
+     * * ENV => `LOG_DIR`
+     * * ARG => `--logDir [dir]`
+     *
+     * @examples ["/var/log/contextmod"]
+     * */
+    dirname?: string | boolean | null
+}
+
+// export type StrongFileOptions = LogFileOptions & {
+//     dirname?: string
+// }
+
+export type LogStreamOptions = Omit<DuplexTransportOptions, 'name' | 'stream' | 'handleRejections' | 'handleExceptions' | 'format' | 'log' | 'logv' | 'close'> & {
+    level?: LogLevel
+}
+
+export interface LoggingOptions  {
+    /**
+     * The minimum log level to output. The log level set will output logs at its level **and all levels above it:**
+     *
+     *  * `error`
+     *  * `warn`
+     *  * `info`
+     *  * `verbose`
+     *  * `debug`
+     *
+     *  Note: `verbose` will display *a lot* of information on the status/result of run rules/checks/actions etc. which is very useful for testing configurations. Once your bot is stable changing the level to `info` will reduce log noise.
+     *
+     *  * ENV => `LOG_LEVEL`
+     *  * ARG => `--logLevel <level>`
+     *
+     *  @default "verbose"
+     *  @examples ["verbose"]
+     * */
+    level?: LogLevel,
+    /**
+     * **DEPRECATED** - Use `file.dirname` instead
+     * The absolute path to a directory where rotating log files should be stored.
+     *
+     * * If not present or `null` or `false` no log files will be created
+     * * If `true` logs will be stored at `[working directory]/logs`
+     *
+     * * ENV => `LOG_DIR`
+     * * ARG => `--logDir [dir]`
+     *
+     * @examples ["/var/log/contextmod"]
+     * @deprecated
+     * @see logging.file.dirname
+     * */
+    path?: string | boolean | null
+
+    /**
+     * Options for Rotating File logging
+     * */
+    file?: LogFileOptions
+    /**
+     * Options for logging to api/web
+     * */
+    stream?: LogStreamOptions
+    /**
+     * Options for logging to console
+     * */
+    console?: LogConsoleOptions
+}
+
+export type StrongLoggingOptions = Required<Pick<LoggingOptions, 'stream' | 'console' | 'file'>> & {
+    level?: LogLevel
+};
+
+export type LoggerFactoryOptions = StrongLoggingOptions & {
+    additionalTransports?: any[]
+    defaultLabel?: string
+}
 /**
  * Available cache providers
  * */
@@ -1220,6 +1292,7 @@ export interface Notifier {
 export interface ManagerStateChangeOption {
     reason?: string
     suppressNotification?: boolean
+    suppressChangeEvent?: boolean
 }
 
 /**
@@ -1351,6 +1424,27 @@ export interface SnoowrapOptions {
     debug?: boolean,
 }
 
+export type FilterCriteriaDefaultBehavior = 'replace' | 'merge';
+
+export interface FilterCriteriaDefaults {
+    itemIs?: TypedActivityStates
+    /**
+     * Determine how itemIs defaults behave when itemIs is present on the check
+     *
+     * * merge => adds defaults to check's itemIs
+     * * replace => check itemIs will replace defaults (no defaults used)
+     * */
+    itemIsBehavior?: FilterCriteriaDefaultBehavior
+    /**
+     * Determine how authorIs defaults behave when authorIs is present on the check
+     *
+     * * merge => merges defaults with check's authorIs
+     * * replace => check authorIs will replace defaults (no defaults used)
+     * */
+    authorIs?: AuthorOptions
+    authorIsBehavior?: FilterCriteriaDefaultBehavior
+}
+
 /**
  * The configuration for an **individual reddit account** ContextMod will run as a bot.
  *
@@ -1379,6 +1473,13 @@ export interface BotInstanceJsonConfig {
      * */
     snoowrap?: SnoowrapOptions
 
+    /**
+     * Define the default behavior for all filter criteria on all checks in all subreddits
+     *
+     * Defaults to exclude mods and automoderator from checks
+     * */
+    filterCriteriaDefaults?: FilterCriteriaDefaults
+
     /**
      * Settings related to bot behavior for subreddits it is managing
      * */
@@ -1446,18 +1547,31 @@ export interface BotInstanceJsonConfig {
      * */
     polling?: PollingDefaults & {
         /**
-         * If set to `true` all subreddits polling unmoderated/modqueue with default polling settings will share a request to "r/mod"
-         * otherwise each subreddit will poll its own mod view
+         * DEPRECATED: See `shared`
+         *
+         *  Using the ENV or ARG will sett `unmoderated` and `modqueue` on `shared`
          *
          * * ENV => `SHARE_MOD`
          * * ARG => `--shareMod`
          *
          * @default false
+         * @deprecated
          * */
         sharedMod?: boolean,
 
         /**
-         * If sharing a mod stream stagger pushing relevant Activities to individual subreddits.
+         * Set which polling sources should be shared among subreddits using default polling settings for that source
+         *
+         * * For `unmoderated and `modqueue` the bot will poll on **r/mod** for new activities
+         * * For `newSub` and `newComm` all subreddits sharing the source will be combined to poll like **r/subreddit1+subreddit2/new**
+         *
+         * If set to `true` all polling sources will be shared,  otherwise specify which sourcs should be shared as a list
+         *
+         * */
+        shared?: PollOn[] | true,
+
+        /**
+         * If sharing a stream staggers pushing relevant Activities to individual subreddits.
          *
          * Useful when running many subreddits and rules are potentially cpu/memory/traffic heavy -- allows spreading out load
          * */
@@ -1560,38 +1674,7 @@ export interface OperatorJsonConfig {
     /**
      * Settings to configure global logging defaults
      * */
-    logging?: {
-        /**
-         * The minimum log level to output. The log level set will output logs at its level **and all levels above it:**
-         *
-         *  * `error`
-         *  * `warn`
-         *  * `info`
-         *  * `verbose`
-         *  * `debug`
-         *
-         *  Note: `verbose` will display *a lot* of information on the status/result of run rules/checks/actions etc. which is very useful for testing configurations. Once your bot is stable changing the level to `info` will reduce log noise.
-         *
-         *  * ENV => `LOG_LEVEL`
-         *  * ARG => `--logLevel <level>`
-         *
-         *  @default "verbose"
-         *  @examples ["verbose"]
-         * */
-        level?: LogLevel,
-        /**
-         * The absolute path to a directory where rotating log files should be stored.
-         *
-         * * If not present or `null` no log files will be created
-         * * If `true` logs will be stored at `[working directory]/logs`
-         *
-         * * ENV => `LOG_DIR`
-         * * ARG => `--logDir [dir]`
-         *
-         * @examples ["/var/log/contextmod"]
-         * */
-        path?: string,
-    },
+    logging?: LoggingOptions,
 
     /**
      * Settings to configure the default caching behavior globally
@@ -1607,6 +1690,17 @@ export interface OperatorJsonConfig {
 
     bots?: BotInstanceJsonConfig[]
 
+    /**
+     * Added to the User-Agent information sent to reddit
+     *
+     * This string will be added BETWEEN version and your bot name.
+     *
+     * EX: `myBranch` => `web:contextMod:v1.0.0-myBranch:BOT-/u/MyBotUser`
+     *
+     * * ENV => `USER_AGENT`
+     * */
+    userAgent?: string
+
     /**
      * Settings for the web interface
      * */
@@ -1769,7 +1863,7 @@ export interface BotInstanceConfig extends BotInstanceJsonConfig {
         heartbeatInterval: number,
     },
     polling: {
-        sharedMod: boolean,
+        shared: PollOn[],
         stagger?: number,
         limit: number,
         interval: number,
@@ -1782,6 +1876,7 @@ export interface BotInstanceConfig extends BotInstanceJsonConfig {
         softLimit: number,
         hardLimit: number,
     }
+    userAgent?: string
 }
 
 export interface OperatorConfig extends OperatorJsonConfig {
@@ -1791,10 +1886,7 @@ export interface OperatorConfig extends OperatorJsonConfig {
         display?: string,
     },
     notifications?: NotificationConfig
-    logging: {
-        level: LogLevel,
-        path?: string,
-    },
+    logging: StrongLoggingOptions,
     caching: StrongCache,
     web: {
         port: number,
@@ -1821,6 +1913,15 @@ export interface OperatorConfig extends OperatorJsonConfig {
     credentials: ThirdPartyCredentialsJsonConfig
 }
 
+export interface OperatorFileConfig {
+    document: YamlOperatorConfigDocument | JsonOperatorConfigDocument
+    isWriteable?: boolean
+}
+
+export interface OperatorConfigWithFileContext extends OperatorConfig {
+    fileConfig: OperatorFileConfig
+}
+
 //export type OperatorConfig = Required<OperatorJsonConfig>;
 
 interface CacheTypeStat {
@@ -1846,6 +1947,7 @@ export interface LogInfo {
     instance?: string
     labels?: string[]
     bot?: string
+    user?: string
 }
 
 export interface ActionResult extends ActionProcessResult {
@@ -1888,22 +1990,6 @@ export interface RedditEntity {
     type: RedditEntityType
 }
 
-export interface StatusCodeError extends Error {
-    name: 'StatusCodeError',
-    statusCode: number,
-    message: string,
-    response: IncomingMessage,
-    error: Error
-}
-
-export interface RequestError extends Error {
-    name: 'RequestError',
-    statusCode: number,
-    message: string,
-    response: IncomingMessage,
-    error: Error
-}
-
 export interface HistoricalStatsDisplay extends HistoricalStats {
     checksRunTotal: number
     checksFromCacheTotal: number
@@ -2007,3 +2093,80 @@ export interface StringComparisonOptions {
     lengthWeight?: number,
     transforms?: ((str: string) => string)[]
 }
+
+export interface FilterCriteriaPropertyResult<T> {
+    property: keyof T
+    expected: (string | boolean | number)[]
+    found?: string | boolean | number | null
+    passed?: null | boolean
+    reason?: string
+    behavior: FilterBehavior
+}
+
+export interface FilterCriteriaResult<T> {
+    behavior: FilterBehavior
+    criteria: T//AuthorCriteria | TypedActivityStates
+    propertyResults: FilterCriteriaPropertyResult<T>[]
+    passed: boolean
+}
+
+export type FilterBehavior = 'include' | 'exclude'
+
+export interface FilterResult<T> {
+    criteriaResults: FilterCriteriaResult<T>[]
+    join: JoinOperands
+    passed: boolean
+}
+
+export interface TextTransformOptions {
+    /**
+     * A set of search-and-replace operations to perform on text values before performing a match. Transformations are performed in the order they are defined.
+     *
+     * * If `transformationsActivity` IS NOT defined then these transformations will be performed on BOTH the activity text (submission title or comment) AND the repost candidate text
+     * * If `transformationsActivity` IS defined then these transformations are only performed on repost candidate text
+     * */
+    transformations?: SearchAndReplaceRegExp[]
+
+    /**
+     * Specify a separate set of transformations for the activity text (submission title or comment)
+     *
+     * To perform no transformations when `transformations` is defined set this to an empty array (`[]`)
+     * */
+    transformationsActivity?: SearchAndReplaceRegExp[]
+}
+
+export interface TextMatchOptions {
+    /**
+     * The percentage, as a whole number, of a repost title/comment that must match the title/comment being checked in order to consider both a match
+     *
+     * Note: Setting to 0 will make every candidate considered a match -- useful if you want to match if the URL has been reposted anywhere
+     *
+     * Defaults to `85` (85%)
+     *
+     * @default 85
+     * @example [85]
+     * */
+    matchScore?: number
+
+    /**
+     * The minimum number of words in the activity being checked for which this rule will run on
+     *
+     * If the word count is below the minimum the rule fails
+     *
+     * Defaults to 2
+     *
+     * @default 2
+     * @example [2]
+     * */
+    minWordCount?: number
+
+    /**
+     * Should text matching be case sensitive?
+     *
+     * Defaults to false
+     *
+     * @default false
+     * @example [false]
+     **/
+    caseSensitive?: boolean
+}

src/Common/types.ts

@@ -28,3 +28,5 @@ export type SetRandomInterval = (
     minDelay: number,
     maxDelay: number,
 ) => { clear: () => void };
+
+export type ConfigFormat = 'json' | 'yaml';

src/Common/typings/support.d.ts

@@ -0,0 +1,26 @@
+declare module 'snoowrap/dist/errors' {
+
+    export interface InvalidUserError extends Error {
+
+    }
+    export interface NoCredentialsError extends Error {
+
+    }
+    export interface InvalidMethodCallError extends Error {
+
+    }
+
+    export interface RequestError extends Error {
+        statusCode: number,
+        response: http.IncomingMessage
+        error: Error
+    }
+
+    export interface StatusCodeError extends RequestError {
+        name: 'StatusCodeError',
+    }
+
+    export interface RateLimitError extends RequestError {
+        name: 'RateLimitError',
+    }
+}

src/ConfigBuilder.ts

@@ -1,12 +1,12 @@
 import {Logger} from "winston";
 import {
     buildCacheOptionsFromProvider, buildCachePrefix,
-    createAjvFactory,
+    createAjvFactory, fileOrDirectoryIsWriteable,
     mergeArr,
     normalizeName,
     overwriteMerge,
-    parseBool, randomId,
-    readConfigFile,
+    parseBool, parseFromJsonOrYamlToObject, randomId,
+    readConfigFile, removeFromSourceIfKeysExistsInDestination,
     removeUndefinedKeys
 } from "./util";
 import {CommentCheck} from "./Check/CommentCheck";
@@ -31,19 +31,34 @@ import {
     CacheOptions,
     BotInstanceJsonConfig,
     BotInstanceConfig,
-    RequiredWebRedditCredentials, RedditCredentials, BotCredentialsJsonConfig, BotCredentialsConfig
+    RequiredWebRedditCredentials,
+    RedditCredentials,
+    BotCredentialsJsonConfig,
+    BotCredentialsConfig,
+    FilterCriteriaDefaults, TypedActivityStates, OperatorFileConfig
 } from "./Common/interfaces";
 import {isRuleSetJSON, RuleSetJson, RuleSetObjectJson} from "./Rule/RuleSet";
 import deepEqual from "fast-deep-equal";
-import {ActionJson, ActionObjectJson, RuleJson, RuleObjectJson} from "./Common/types";
+import {ActionJson, ActionObjectJson, ConfigFormat, RuleJson, RuleObjectJson} from "./Common/types";
 import {isActionJson} from "./Action";
 import {getLogger} from "./Utils/loggerFactory";
 import {GetEnvVars} from 'env-cmd';
 import {operatorConfig} from "./Utils/CommandConfig";
 import merge from 'deepmerge';
 import * as process from "process";
-import {cacheOptDefaults, cacheTTLDefaults} from "./Common/defaults";
+import {cacheOptDefaults, cacheTTLDefaults, filterCriteriaDefault} from "./Common/defaults";
 import objectHash from "object-hash";
+import {AuthorCriteria, AuthorOptions} from "./Author/Author";
+import path from 'path';
+import {
+    JsonOperatorConfigDocument,
+    OperatorConfigDocumentInterface,
+    YamlOperatorConfigDocument
+} from "./Common/Config/Operator";
+import {ConfigDocumentInterface} from "./Common/Config/AbstractConfigDocument";
+import {Document as YamlDocument} from "yaml";
+import {SimpleError} from "./Utils/Errors";
+import {ErrorWithCause} from "pony-cause";
 
 export interface ConfigBuilderOptions {
     logger: Logger,
@@ -115,22 +130,51 @@ export class ConfigBuilder {
         return validConfig as JSONConfig;
     }
 
-    parseToStructured(config: JSONConfig): CheckStructuredJson[] {
+    parseToStructured(config: JSONConfig, filterCriteriaDefaultsFromBot?: FilterCriteriaDefaults): CheckStructuredJson[] {
         let namedRules: Map<string, RuleObjectJson> = new Map();
         let namedActions: Map<string, ActionObjectJson> = new Map();
-        const {checks = []} = config;
+        const {checks = [], filterCriteriaDefaults} = config;
         for (const c of checks) {
             const {rules = []} = c;
             namedRules = extractNamedRules(rules, namedRules);
             namedActions = extractNamedActions(c.actions, namedActions);
         }
 
+        const filterDefs = filterCriteriaDefaults ?? filterCriteriaDefaultsFromBot;
+        const {
+            authorIsBehavior = 'merge',
+            itemIsBehavior = 'merge',
+            authorIs: authorIsDefault = {},
+            itemIs: itemIsDefault = []
+        } = filterDefs || {};
+
         const structuredChecks: CheckStructuredJson[] = [];
         for (const c of checks) {
-            const {rules = []} = c;
+            const {rules = [], authorIs = {}, itemIs = []} = c;
             const strongRules = insertNamedRules(rules, namedRules);
             const strongActions = insertNamedActions(c.actions, namedActions);
-            const strongCheck = {...c, rules: strongRules, actions: strongActions} as CheckStructuredJson;
+
+            let derivedAuthorIs: AuthorOptions = authorIsDefault;
+            if (authorIsBehavior === 'merge') {
+                derivedAuthorIs = merge.all([authorIs, authorIsDefault], {arrayMerge: removeFromSourceIfKeysExistsInDestination});
+            } else if (Object.keys(authorIs).length > 0) {
+                derivedAuthorIs = authorIs;
+            }
+
+            let derivedItemIs: TypedActivityStates = itemIsDefault;
+            if (itemIsBehavior === 'merge') {
+                derivedItemIs = [...itemIs, ...itemIsDefault];
+            } else if (itemIs.length > 0) {
+                derivedItemIs = itemIs;
+            }
+
+            const strongCheck = {
+                ...c,
+                authorIs: derivedAuthorIs,
+                itemIs: derivedItemIs,
+                rules: strongRules,
+                actions: strongActions
+            } as CheckStructuredJson;
             structuredChecks.push(strongCheck);
         }
 
@@ -146,10 +190,6 @@ export const buildPollingOptions = (values: (string | PollingOptions)[]): Pollin
                 pollOn: v as PollOn,
                 interval: DEFAULT_POLLING_INTERVAL,
                 limit: DEFAULT_POLLING_LIMIT,
-                clearProcessed: {
-                    size: DEFAULT_POLLING_LIMIT,
-                    retain: DEFAULT_POLLING_LIMIT,
-                }
             });
         } else {
             const {
@@ -157,14 +197,12 @@ export const buildPollingOptions = (values: (string | PollingOptions)[]): Pollin
                 interval = DEFAULT_POLLING_INTERVAL,
                 limit = DEFAULT_POLLING_LIMIT,
                 delayUntil,
-                clearProcessed = {size: limit, retain: limit},
             } = v;
             opts.push({
                 pollOn: p as PollOn,
                 interval,
                 limit,
                 delayUntil,
-                clearProcessed
             });
         }
     }
@@ -299,7 +337,7 @@ export const parseDefaultBotInstanceFromArgs = (args: any): BotInstanceJsonConfi
             heartbeatInterval: heartbeat,
         },
         polling: {
-            sharedMod,
+            shared: sharedMod ? ['unmoderated', 'modqueue'] : undefined,
         },
         nanny: {
             softLimit,
@@ -336,7 +374,16 @@ export const parseOpConfigFromArgs = (args: any): OperatorJsonConfig => {
         },
         logging: {
             level: logLevel,
-            path: logDir === true ? `${process.cwd()}/logs` : undefined,
+            file: {
+                level: logLevel,
+                dirName: logDir,
+            },
+            stream: {
+                level: logLevel,
+            },
+            console: {
+                level: logLevel,
+            }
         },
         caching: {
             provider: caching,
@@ -402,7 +449,7 @@ export const parseDefaultBotInstanceFromEnv = (): BotInstanceJsonConfig => {
             heartbeatInterval: process.env.HEARTBEAT !== undefined ? parseInt(process.env.HEARTBEAT) : undefined,
         },
         polling: {
-            sharedMod: parseBool(process.env.SHARE_MOD),
+            shared: parseBool(process.env.SHARE_MOD) ? ['unmoderated', 'modqueue'] : undefined,
         },
         nanny: {
             softLimit: process.env.SOFT_LIMIT !== undefined ? parseInt(process.env.SOFT_LIMIT) : undefined,
@@ -420,9 +467,17 @@ export const parseOpConfigFromEnv = (): OperatorJsonConfig => {
             display: process.env.OPERATOR_DISPLAY
         },
         logging: {
-            // @ts-ignore
             level: process.env.LOG_LEVEL,
-            path: process.env.LOG_DIR === 'true' ? `${process.cwd()}/logs` : undefined,
+            file: {
+                level: process.env.LOG_LEVEL,
+                dirname: process.env.LOG_DIR,
+            },
+            stream: {
+                level: process.env.LOG_LEVEL,
+            },
+            console: {
+                level: process.env.LOG_LEVEL,
+            }
         },
         caching: {
             provider: {
@@ -448,9 +503,9 @@ export const parseOpConfigFromEnv = (): OperatorJsonConfig => {
             },
         },
         credentials: {
-          youtube: {
-              apiKey: process.env.YOUTUBE_API_KEY
-          }
+            youtube: {
+                apiKey: process.env.YOUTUBE_API_KEY
+            }
         }
     }
 
@@ -463,12 +518,26 @@ export const parseOpConfigFromEnv = (): OperatorJsonConfig => {
 // Actual ENVs (from environment)
 // json config
 // args from cli
-export const parseOperatorConfigFromSources = async (args: any): Promise<OperatorJsonConfig> => {
-    const {logLevel = process.env.LOG_LEVEL, logDir = process.env.LOG_DIR || false} = args || {};
+export const parseOperatorConfigFromSources = async (args: any): Promise<[OperatorJsonConfig, OperatorFileConfig]> => {
+    const {logLevel = process.env.LOG_LEVEL ?? 'debug', logDir = process.env.LOG_DIR} = args || {};
     const envPath = process.env.OPERATOR_ENV;
+    const initLoggerOptions = {
+        level: logLevel,
+        console: {
+            level: logLevel
+        },
+        file: {
+            level: logLevel,
+            dirname: logDir,
+        },
+        stream: {
+            level: logLevel
+        }
+    }
 
     // create a pre config logger to help with debugging
-    const initLogger = getLogger({logLevel, logDir: logDir === true ? `${process.cwd()}/logs` : logDir}, 'init');
+    // default to debug if nothing is provided
+    const initLogger = getLogger(initLoggerOptions, 'init');
 
     try {
         const vars = await GetEnvVars({
@@ -494,24 +563,90 @@ export const parseOperatorConfigFromSources = async (args: any): Promise<Operato
         //swallow silently for now 😬
     }
 
-    const {operatorConfig = process.env.OPERATOR_CONFIG} = args;
+    const {operatorConfig = (process.env.OPERATOR_CONFIG ?? path.resolve(__dirname, '../config.yaml'))} = args;
     let configFromFile: OperatorJsonConfig = {};
-    if (operatorConfig !== undefined) {
-        let rawConfig;
-        try {
-            rawConfig = await readConfigFile(operatorConfig, {log: initLogger}) as object;
-        } catch (err: any) {
-            initLogger.error('Cannot continue app startup because operator config file was not parseable.');
-            err.logged = true;
-            throw err;
+    let fileConfigFormat: ConfigFormat | undefined = undefined;
+    let fileConfig: object = {};
+    let rawConfig: string = '';
+    let configDoc: YamlOperatorConfigDocument | JsonOperatorConfigDocument;
+    let writeable = false;
+    try {
+        writeable = await fileOrDirectoryIsWriteable(operatorConfig);
+    } catch (e) {
+        initLogger.warn(`Issue while parsing operator config file location: ${e} \n This is only a problem if you do not have a config file but are planning on adding bots interactively.`);
+    }
+
+    try {
+        const [rawConfigValue, format] = await readConfigFile(operatorConfig, {log: initLogger});
+        rawConfig = rawConfigValue ?? '';
+        fileConfigFormat = format as ConfigFormat;
+    } catch (err: any) {
+        const {code} = err;
+        if (code === 'ENOENT') {
+            initLogger.warn('No operator config file found but will continue');
+            if (err.extension !== undefined) {
+                fileConfigFormat = err.extension
+            }
+        } else {
+            throw new ErrorWithCause('Cannot continue app startup because operator config file exists but was not parseable.', {cause: err});
         }
+    }
+    const [format, doc, jsonErr, yamlErr] = parseFromJsonOrYamlToObject(rawConfig, {
+        location: operatorConfig,
+        jsonDocFunc: (content, location) => new JsonOperatorConfigDocument(content, location),
+        yamlDocFunc: (content, location) => new YamlOperatorConfigDocument(content, location)
+    });
+
+
+    if (format !== undefined && fileConfigFormat === undefined) {
+        fileConfigFormat = 'yaml';
+    }
+
+    if (doc === undefined && rawConfig !== '') {
+        initLogger.error(`Could not parse file contents at ${operatorConfig} as JSON or YAML (likely it is ${fileConfigFormat}):`);
+        initLogger.error(jsonErr);
+        initLogger.error(yamlErr);
+        throw new SimpleError(`Could not parse file contents at ${operatorConfig} as JSON or YAML`);
+    } else if (doc === undefined && rawConfig === '') {
+        // create an empty doc
+        if(fileConfigFormat === 'json') {
+            configDoc = new JsonOperatorConfigDocument('{}', operatorConfig);
+        } else {
+            configDoc = new YamlOperatorConfigDocument('', operatorConfig);
+            configDoc.parsed = new YamlDocument({});
+        }
+        configFromFile = {};
+    } else {
+        configDoc = doc as (YamlOperatorConfigDocument | JsonOperatorConfigDocument);
+
         try {
-            configFromFile = validateJson(rawConfig, operatorSchema, initLogger) as OperatorJsonConfig;
+            configFromFile = validateJson(configDoc.toJS(), operatorSchema, initLogger) as OperatorJsonConfig;
+            const {
+                bots = [],
+                logging: {
+                    path = undefined
+                } = {}
+            } = configFromFile || {};
+            if(path !== undefined) {
+                initLogger.warn(`'path' property in top-level 'logging' object is DEPRECATED and will be removed in next minor version. Use 'logging.file.dirname' instead`);
+            }
+            for (const b of bots) {
+                const {
+                    polling: {
+                        sharedMod
+                    } = {}
+                } = b;
+                if (sharedMod !== undefined) {
+                    initLogger.warn(`'sharedMod' bot config property is DEPRECATED and will be removed in next minor version. Use 'shared' property instead (see docs)`);
+                    break;
+                }
+            }
         } catch (err: any) {
             initLogger.error('Cannot continue app startup because operator config file was not valid.');
             throw err;
         }
     }
+
     const opConfigFromArgs = parseOpConfigFromArgs(args);
     const opConfigFromEnv = parseOpConfigFromEnv();
 
@@ -531,14 +666,21 @@ export const parseOperatorConfigFromSources = async (args: any): Promise<Operato
         defaultBotInstance.caching = configFromFile.caching;
     }
 
-    let botInstances = [];
+    let botInstances: BotInstanceJsonConfig[] = [];
     if (botInstancesFromFile.length === 0) {
-        botInstances = [defaultBotInstance];
+        // only add default bot if user supplied any credentials
+        // otherwise its most likely just default, empty settings
+        if(defaultBotInstance.credentials !== undefined) {
+            botInstances = [defaultBotInstance];
+        }
     } else {
         botInstances = botInstancesFromFile.map(x => merge.all([defaultBotInstance, x], {arrayMerge: overwriteMerge}));
     }
 
-    return removeUndefinedKeys({...mergedConfig, bots: botInstances}) as OperatorJsonConfig;
+    return [removeUndefinedKeys({...mergedConfig, bots: botInstances}) as OperatorJsonConfig, {
+        document: configDoc,
+        isWriteable: writeable
+    }];
 }
 
 export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): OperatorConfig => {
@@ -551,8 +693,12 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
         logging: {
             level = 'verbose',
             path,
+            file = {},
+            console = {},
+            stream = {},
         } = {},
         caching: opCache,
+        userAgent,
         web: {
             port = 8085,
             maxLogs = 200,
@@ -624,156 +770,18 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
         }
     }
 
-    let hydratedBots: BotInstanceConfig[] = bots.map(x => {
-        const {
-            name: botName,
-            polling: {
-                sharedMod = false,
-                stagger,
-                limit = 100,
-                interval = 30,
-            } = {},
-            queue: {
-                maxWorkers = 1,
-            } = {},
-            caching,
-            nanny: {
-                softLimit = 250,
-                hardLimit = 50
-            } = {},
-            snoowrap = snoowrapOp,
-            credentials = {},
-            subreddits: {
-                names = [],
-                exclude = [],
-                wikiConfig = 'botconfig/contextbot',
-                dryRun,
-                heartbeatInterval = 300,
-            } = {},
-        } = x;
-
-        let botCache: StrongCache;
-        let botActionedEventsDefault: number;
-
-        if (caching === undefined) {
-
-            botCache = {
-                ...cacheTTLDefaults,
-                actionedEventsDefault: opActionedEventsDefault,
-                actionedEventsMax: opActionedEventsMax,
-                provider: {...defaultProvider}
-            };
-        } else {
-            const {
-                provider,
-                actionedEventsMax = opActionedEventsMax,
-                actionedEventsDefault = opActionedEventsDefault,
-                ...restConfig
-            } = caching;
-
-            botActionedEventsDefault = actionedEventsDefault;
-            if (actionedEventsMax !== undefined) {
-                botActionedEventsDefault = Math.min(actionedEventsDefault, actionedEventsMax);
-            }
-
-            if (typeof provider === 'string') {
-                botCache = {
-                    ...cacheTTLDefaults,
-                    ...restConfig,
-                    actionedEventsDefault: botActionedEventsDefault,
-                    provider: {
-                        store: provider as CacheProvider,
-                        ...cacheOptDefaults
-                    }
-                }
-            } else {
-                const {ttl = 60, max = 500, store = 'memory', ...rest} = provider || {};
-                botCache = {
-                    ...cacheTTLDefaults,
-                    ...restConfig,
-                    actionedEventsDefault: botActionedEventsDefault,
-                    actionedEventsMax,
-                    provider: {
-                        store,
-                        ...cacheOptDefaults,
-                        ...rest,
-                    },
-                }
-            }
-        }
-
-        let botCreds: BotCredentialsConfig;
-
-        if((credentials as any).clientId !== undefined) {
-            const creds = credentials as RedditCredentials;
-            const {
-                clientId: ci,
-                clientSecret: cs,
-                ...restCred
-            } = creds;
-            botCreds = {
-                reddit: {
-                        clientId: (ci as string),
-                        clientSecret: (cs as string),
-                        ...restCred,
-                }
-            }
-        } else {
-            const creds = credentials as BotCredentialsJsonConfig;
-            const {
-                reddit: {
-                    clientId: ci,
-                    clientSecret: cs,
-                    ...restRedditCreds
-                },
-                ...rest
-            } = creds;
-            botCreds = {
-                reddit: {
-                    clientId: (ci as string),
-                    clientSecret: (cs as string),
-                    ...restRedditCreds,
-                },
-                ...rest
-            }
-        }
-
-        if (botCache.provider.prefix === undefined || botCache.provider.prefix === defaultProvider.prefix) {
-            // need to provide unique prefix to bot
-            botCache.provider.prefix = buildCachePrefix([botCache.provider.prefix, 'bot', (botName || objectHash.sha1(botCreds))]);
-        }
-
-        return {
-            name: botName,
-            snoowrap,
-            subreddits: {
-                names,
-                exclude,
-                wikiConfig,
-                heartbeatInterval,
-                dryRun,
-            },
-            credentials: botCreds,
-            caching: botCache,
-            polling: {
-                sharedMod,
-                stagger,
-                limit,
-                interval,
-            },
-            queue: {
-                maxWorkers,
-            },
-            nanny: {
-                softLimit,
-                hardLimit
-            }
-        }
-
-    });
-
     const defaultOperators = typeof name === 'string' ? [name] : name;
 
+    const {
+        dirname = path,
+        ...fileRest
+    } = file;
+
+     const defaultWebCredentials = {
+         redirectUri: 'http://localhost:8085/callback'
+     };
+
+
     const config: OperatorConfig = {
         mode,
         operator: {
@@ -782,9 +790,22 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
         },
         logging: {
             level,
-            path
+            file: {
+                level: level,
+                dirname,
+                ...fileRest,
+            },
+            stream: {
+                level: level,
+                ...stream,
+            },
+            console: {
+                level: level,
+                ...console,
+            }
         },
         caching: cache,
+        userAgent,
         web: {
             port,
             caching: {
@@ -800,7 +821,7 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
             },
             maxLogs,
             clients: clients === undefined ? [{host: 'localhost:8095', secret: apiSecret}] : clients,
-            credentials: webCredentials as RequiredWebRedditCredentials,
+            credentials: {...defaultWebCredentials, ...webCredentials} as RequiredWebRedditCredentials,
             operators: operators || defaultOperators,
         },
         api: {
@@ -808,9 +829,177 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
             secret: apiSecret,
             friendly
         },
-        bots: hydratedBots,
+        bots: [],
         credentials,
     };
 
+    config.bots = bots.map(x => buildBotConfig(x, config));
+
     return config;
 }
+
+export const buildBotConfig = (data: BotInstanceJsonConfig, opConfig: OperatorConfig): BotInstanceConfig => {
+    const {
+        snoowrap: snoowrapOp,
+        caching: {
+            actionedEventsMax: opActionedEventsMax,
+            actionedEventsDefault: opActionedEventsDefault = 25,
+            provider: defaultProvider,
+        } = {},
+        userAgent,
+    } = opConfig;
+    const {
+        name: botName,
+        filterCriteriaDefaults = filterCriteriaDefault,
+        polling: {
+            sharedMod,
+            shared = [],
+            stagger,
+            limit = 100,
+            interval = 30,
+        } = {},
+        queue: {
+            maxWorkers = 1,
+        } = {},
+        caching,
+        nanny: {
+            softLimit = 250,
+            hardLimit = 50
+        } = {},
+        snoowrap = snoowrapOp,
+        credentials = {},
+        subreddits: {
+            names = [],
+            exclude = [],
+            wikiConfig = 'botconfig/contextbot',
+            dryRun,
+            heartbeatInterval = 300,
+        } = {},
+    } = data;
+
+    let botCache: StrongCache;
+    let botActionedEventsDefault: number;
+
+    if (caching === undefined) {
+
+        botCache = {
+            ...cacheTTLDefaults,
+            actionedEventsDefault: opActionedEventsDefault,
+            actionedEventsMax: opActionedEventsMax,
+            provider: {...defaultProvider as CacheOptions}
+        };
+    } else {
+        const {
+            provider,
+            actionedEventsMax = opActionedEventsMax,
+            actionedEventsDefault = opActionedEventsDefault,
+            ...restConfig
+        } = caching;
+
+        botActionedEventsDefault = actionedEventsDefault;
+        if (actionedEventsMax !== undefined) {
+            botActionedEventsDefault = Math.min(actionedEventsDefault, actionedEventsMax);
+        }
+
+        if (typeof provider === 'string') {
+            botCache = {
+                ...cacheTTLDefaults,
+                ...restConfig,
+                actionedEventsDefault: botActionedEventsDefault,
+                provider: {
+                    store: provider as CacheProvider,
+                    ...cacheOptDefaults
+                }
+            }
+        } else {
+            const {ttl = 60, max = 500, store = 'memory', ...rest} = provider || {};
+            botCache = {
+                ...cacheTTLDefaults,
+                ...restConfig,
+                actionedEventsDefault: botActionedEventsDefault,
+                actionedEventsMax,
+                provider: {
+                    store,
+                    ...cacheOptDefaults,
+                    ...rest,
+                },
+            }
+        }
+    }
+
+    let botCreds: BotCredentialsConfig;
+
+    if ((credentials as any).clientId !== undefined) {
+        const creds = credentials as RedditCredentials;
+        const {
+            clientId: ci,
+            clientSecret: cs,
+            ...restCred
+        } = creds;
+        botCreds = {
+            reddit: {
+                clientId: (ci as string),
+                clientSecret: (cs as string),
+                ...restCred,
+            }
+        }
+    } else {
+        const creds = credentials as BotCredentialsJsonConfig;
+        const {
+            reddit: {
+                clientId: ci,
+                clientSecret: cs,
+                ...restRedditCreds
+            },
+            ...rest
+        } = creds;
+        botCreds = {
+            reddit: {
+                clientId: (ci as string),
+                clientSecret: (cs as string),
+                ...restRedditCreds,
+            },
+            ...rest
+        }
+    }
+
+    if (botCache.provider.prefix === undefined || botCache.provider.prefix === (defaultProvider as CacheOptions).prefix) {
+        // need to provide unique prefix to bot
+        botCache.provider.prefix = buildCachePrefix([botCache.provider.prefix, 'bot', (botName || objectHash.sha1(botCreds))]);
+    }
+
+    let realShared = shared === true ? ['unmoderated', 'modqueue', 'newComm', 'newSub'] : shared;
+    if (sharedMod === true) {
+        realShared.push('unmoderated');
+        realShared.push('modqueue');
+    }
+
+    return {
+        name: botName,
+        snoowrap: snoowrap || {},
+        filterCriteriaDefaults,
+        subreddits: {
+            names,
+            exclude,
+            wikiConfig,
+            heartbeatInterval,
+            dryRun,
+        },
+        credentials: botCreds,
+        caching: botCache,
+        userAgent,
+        polling: {
+            shared: [...new Set(realShared)] as PollOn[],
+            stagger,
+            limit,
+            interval,
+        },
+        queue: {
+            maxWorkers,
+        },
+        nanny: {
+            softLimit,
+            hardLimit
+        }
+    }
+}

src/Rule/AttributionRule.ts

@@ -14,8 +14,8 @@ import {
     PASS
 } from "../util";
 import { Comment } from "snoowrap/dist/objects";
-import SimpleError from "../Utils/SimpleError";
 import as from "async";
+import {SimpleError} from "../Utils/Errors";
 
 
 export interface AttributionCriteria {

src/Rule/AuthorRule.ts

@@ -2,6 +2,7 @@ import {Rule, RuleJSONConfig, RuleOptions, RuleResult} from "./index";
 import {Comment} from "snoowrap";
 import Submission from "snoowrap/dist/objects/Submission";
 import {Author, AuthorCriteria} from "../Author/Author";
+import {checkAuthorFilter} from "../Subreddit/SubredditResources";
 
 /**
  * Checks the author of the Activity against AuthorCriteria. This differs from a Rule's AuthorOptions as this is a full Rule and will only pass/fail, not skip.
@@ -59,20 +60,8 @@ export class AuthorRule extends Rule {
     }
 
     protected async process(item: Comment | Submission): Promise<[boolean, RuleResult]> {
-        if (this.include.length > 0) {
-            for (const auth of this.include) {
-                if (await this.resources.testAuthorCriteria(item, auth)) {
-                    return Promise.resolve([true, this.getResult(true)]);
-                }
-            }
-            return Promise.resolve([false, this.getResult(false)]);
-        }
-        for (const auth of this.exclude) {
-            if (await this.resources.testAuthorCriteria(item, auth, false)) {
-                return Promise.resolve([true, this.getResult(true)]);
-            }
-        }
-        return Promise.resolve([false, this.getResult(false)]);
+        const [result, filterType] = await checkAuthorFilter(item, {include: this.include, exclude: this.exclude}, this.resources, this.logger);
+        return Promise.resolve([result, this.getResult(result)]);
     }
 }
 

src/Rule/RecentActivityRule.ts

@@ -1,6 +1,7 @@
 import {Rule, RuleJSONConfig, RuleOptions, RulePremise, RuleResult} from "./index";
-import {Comment, VoteableContent} from "snoowrap";
+import {VoteableContent} from "snoowrap";
 import Submission from "snoowrap/dist/objects/Submission";
+import Comment from "snoowrap/dist/objects/Comment";
 import as from 'async';
 import pMap from 'p-map';
 // @ts-ignore
@@ -23,7 +24,7 @@ import {
     parseSubredditName,
     parseUsableLinkIdentifier,
     PASS, sleep,
-    toStrongSubredditState
+    toStrongSubredditState, windowToActivityWindowCriteria
 } from "../util";
 import {
     ActivityWindow,
@@ -115,15 +116,42 @@ export class RecentActivityRule extends Rule {
     async process(item: Submission | Comment): Promise<[boolean, RuleResult]> {
         let activities;
 
+        // ACID is a bitch
+        // reddit may not return the activity being checked in the author's recent history due to availability/consistency issues or *something*
+        // so make sure we add it in if config is checking the same type and it isn't included
+        // TODO refactor this for SubredditState everywhere branch
+        let shouldIncludeSelf = true;
+        const strongWindow = windowToActivityWindowCriteria(this.window);
+        const {
+            subreddits: {
+                include = [],
+                exclude = []
+            } = {}
+        } = strongWindow;
+        if (include.length > 0 && !include.some(x => x.toLocaleLowerCase() === item.subreddit.display_name.toLocaleLowerCase())) {
+            shouldIncludeSelf = false;
+        } else if (exclude.length > 0 && exclude.some(x => x.toLocaleLowerCase() === item.subreddit.display_name.toLocaleLowerCase())) {
+            shouldIncludeSelf = false;
+        }
+
         switch (this.lookAt) {
             case 'comments':
                 activities = await this.resources.getAuthorComments(item.author, {window: this.window});
+                if (shouldIncludeSelf && item instanceof Comment && !activities.some(x => x.name === item.name)) {
+                    activities.unshift(item);
+                }
                 break;
             case 'submissions':
                 activities = await this.resources.getAuthorSubmissions(item.author, {window: this.window});
+                if (shouldIncludeSelf && item instanceof Submission && !activities.some(x => x.name === item.name)) {
+                    activities.unshift(item);
+                }
                 break;
             default:
                 activities = await this.resources.getAuthorActivities(item.author, {window: this.window});
+                if (shouldIncludeSelf && !activities.some(x => x.name === item.name)) {
+                    activities.unshift(item);
+                }
                 break;
         }
 
@@ -316,34 +344,6 @@ export class RecentActivityRule extends Rule {
                 }
             }
 
-            for (const activity of viableActivity) {
-                if (asSubmission(activity) && submissionState !== undefined) {
-                    if (!(await this.resources.testItemCriteria(activity, [submissionState]))) {
-                        continue;
-                    }
-                } else if (commentState !== undefined) {
-                    if (!(await this.resources.testItemCriteria(activity, [commentState]))) {
-                        continue;
-                    }
-                }
-                let inSubreddits = false;
-                for (const ss of subStates) {
-                    const res = await this.resources.testSubredditCriteria(activity, ss);
-                    if (res) {
-                        inSubreddits = true;
-                        break;
-                    }
-                }
-                if (inSubreddits) {
-                    currCount++;
-                    combinedKarma += activity.score;
-                    const pSub = getActivitySubredditName(activity);
-                    if (!presentSubs.includes(pSub)) {
-                        presentSubs.push(pSub);
-                    }
-                }
-            }
-
             const {operator, value, isPercent} = parseGenericValueOrPercentComparison(threshold);
             let sum = {
                 subsWithActivity: presentSubs,

src/Rule/RegexRule.ts

@@ -11,7 +11,7 @@ import {
     ActivityWindowType, JoinOperands,
 } from "../Common/interfaces";
 import dayjs from 'dayjs';
-import SimpleError from "../Utils/SimpleError";
+import {SimpleError} from "../Utils/Errors";
 
 export interface RegexCriteria {
     /**
@@ -95,6 +95,15 @@ export interface RegexCriteria {
      * */
     totalMatchThreshold?: string,
 
+    /**
+     * When `true` the Activity being checked MUST pass the `matchThreshold` before the Rule considers any history
+     *
+     * For use with `activityMatchThreshold`/`totalMatchThreshold` -- useful to conserve API calls
+     *
+     * @default false
+     * */
+    mustMatchCurrent?: boolean
+
     window?: ActivityWindowType
 }
 
@@ -140,6 +149,7 @@ export class RegexRule extends Rule {
                 matchThreshold = '> 0',
                 activityMatchThreshold = '> 0',
                 totalMatchThreshold = null,
+                mustMatchCurrent = false,
                 window,
             } = criteria;
 
@@ -152,7 +162,8 @@ export class RegexRule extends Rule {
             }, []);
 
             // check regex
-            const reg = parseStringToRegex(regex, 'g');
+            const regexContent = await this.resources.getContent(regex);
+            const reg = parseStringToRegex(regexContent, 'g');
             if(reg === undefined) {
                 throw new SimpleError(`Value given for regex on Criteria ${name} was not valid: ${regex}`);
             }
@@ -183,6 +194,8 @@ export class RegexRule extends Rule {
             if (singleMatched) {
                 activitiesMatchedCount++;
             }
+            const singleCriteriaPass = !mustMatchCurrent || (mustMatchCurrent && singleMatched);
+
             if (activityMatchComparison !== undefined) {
                 activityThresholdMet = !activityMatchComparison.isPercent && comparisonTextOp(activitiesMatchedCount, activityMatchComparison.operator, activityMatchComparison.value);
             }
@@ -191,7 +204,7 @@ export class RegexRule extends Rule {
             }
 
             let history: (Submission | Comment)[] = [];
-            if ((activityThresholdMet === false || totalThresholdMet === false) && window !== undefined) {
+            if ((activityThresholdMet === false || totalThresholdMet === false) && window !== undefined && singleCriteriaPass) {
                 // our checking activity didn't meet threshold requirements and criteria does define window
                 // leh go
 
@@ -257,12 +270,13 @@ export class RegexRule extends Rule {
             const critResults = {
                 criteria: {
                     name,
-                    regex,
+                    regex: regex !== regexContent ? `${regex} from ${regexContent}` : regex,
                     testOn,
                     matchThreshold,
                     activityMatchThreshold,
                     totalMatchThreshold,
-                    window: humanWindow
+                    window: humanWindow,
+                    mustMatchCurrent,
                 },
                 matches,
                 matchCount,

src/Rule/RepeatActivityRule.ts

@@ -1,17 +1,28 @@
 import {Rule, RuleJSONConfig, RuleOptions, RuleResult} from "./index";
 import {Comment} from "snoowrap";
 import {
-    activityWindowText, asSubmission,
-    comparisonTextOp, FAIL, getActivitySubredditName, isExternalUrlSubmission, isRedditMedia,
-    parseGenericValueComparison, parseSubredditName,
-    parseUsableLinkIdentifier as linkParser, PASS, subredditStateIsNameOnly, toStrongSubredditState
+    activityWindowText,
+    asSubmission,
+    comparisonTextOp,
+    FAIL,
+    getActivitySubredditName,
+    isExternalUrlSubmission,
+    isRedditMedia,
+    parseGenericValueComparison,
+    parseSubredditName,
+    parseUsableLinkIdentifier as linkParser,
+    PASS,
+    searchAndReplace,
+    stringSameness,
+    subredditStateIsNameOnly,
+    toStrongSubredditState
 } from "../util";
 import {
     ActivityWindow,
     ActivityWindowType,
-    ReferenceSubmission,
+    ReferenceSubmission, SearchAndReplaceRegExp,
     StrongSubredditState,
-    SubredditState
+    SubredditState, TextMatchOptions, TextTransformOptions
 } from "../Common/interfaces";
 import Submission from "snoowrap/dist/objects/Submission";
 import dayjs from "dayjs";
@@ -29,27 +40,6 @@ interface RepeatActivityReducer {
     allSets: RepeatActivityData[]
 }
 
-const getActivityIdentifier = (activity: (Submission | Comment), length = 200) => {
-    let identifier: string;
-    if (asSubmission(activity)) {
-        if (activity.is_self) {
-            identifier = `${activity.title}${activity.selftext.slice(0, length)}`;
-        } else if(isRedditMedia(activity)) {
-            identifier = activity.title;
-        } else {
-            identifier = parseUsableLinkIdentifier(activity.url) as string;
-        }
-    } else {
-        identifier = activity.body.slice(0, length);
-    }
-    return identifier;
-}
-
-const fuzzyOptions = {
-    includeScore: true,
-    distance: 15
-};
-
 export class RepeatActivityRule extends Rule {
     threshold: string;
     window: ActivityWindowType;
@@ -62,6 +52,9 @@ export class RepeatActivityRule extends Rule {
     activityFilterFunc: (x: Submission|Comment) => Promise<boolean> = async (x) => true;
     keepRemoved: boolean;
     minWordCount: number;
+    transformations: SearchAndReplaceRegExp[]
+    caseSensitive: boolean
+    matchScore: number
 
     constructor(options: RepeatActivityOptions) {
         super(options);
@@ -75,7 +68,13 @@ export class RepeatActivityRule extends Rule {
             include = [],
             exclude = [],
             keepRemoved = false,
+            transformations = [],
+            caseSensitive = true,
+            matchScore = 85,
         } = options;
+        this.matchScore = matchScore;
+        this.transformations = transformations;
+        this.caseSensitive = caseSensitive;
         this.minWordCount = minWordCount;
         this.keepRemoved = keepRemoved;
         this.threshold = threshold;
@@ -136,6 +135,37 @@ export class RepeatActivityRule extends Rule {
         }
     }
 
+    getActivityIdentifier(activity: (Submission | Comment), length = 200, transform = true) {
+        let identifier: string;
+        if (asSubmission(activity)) {
+            if (activity.is_self) {
+                identifier = `${activity.title}${activity.selftext.slice(0, length)}`;
+            } else if(isRedditMedia(activity)) {
+                identifier = activity.title;
+            } else {
+                identifier = parseUsableLinkIdentifier(activity.url) as string;
+            }
+        } else {
+            identifier = activity.body.slice(0, length);
+        }
+
+        if(!transform) {
+            return identifier;
+        }
+
+        // apply any transforms
+        if (this.transformations.length > 0) {
+            identifier = searchAndReplace(identifier, this.transformations);
+        }
+
+        // perform after transformations so as not to mess up regex's depending on case
+        if(!this.caseSensitive) {
+            identifier = identifier.toLowerCase();
+        }
+
+        return identifier;
+    }
+
     async process(item: Submission|Comment): Promise<[boolean, RuleResult]> {
         let referenceUrl;
         if(asSubmission(item) && this.useSubmissionAsReference) {
@@ -162,9 +192,10 @@ export class RepeatActivityRule extends Rule {
             const acc = await accProm;
             const {openSets = [], allSets = []} = acc;
 
-            let identifier = getActivityIdentifier(activity);
+            let identifier = this.getActivityIdentifier(activity);
+
             const isUrl = isExternalUrlSubmission(activity);
-            let fu = new Fuse([identifier], !isUrl ? fuzzyOptions : {...fuzzyOptions, distance: 5});
+            //let fu = new Fuse([identifier], !isUrl ? fuzzyOptions : {...fuzzyOptions, distance: 5});
             const validSub = await this.activityFilterFunc(activity);
             let minMet = identifier.length >= this.minWordCount;
 
@@ -174,12 +205,15 @@ export class RepeatActivityRule extends Rule {
             let currIdentifierInOpen = false;
             const bufferedActivities = this.gapAllowance === undefined || this.gapAllowance === 0 ? [] : activities.slice(Math.max(0, index - this.gapAllowance), Math.max(0, index));
             for (const o of openSets) {
-                const res = fu.search(o.identifier);
-                const match = res.length > 0;
-                if (match && validSub && minMet) {
+                const strMatchResults = stringSameness(o.identifier, identifier);
+                if (strMatchResults.highScoreWeighted >= this.matchScore && minMet) {
                     updatedOpenSets.push({...o, sets: [...o.sets, activity]});
                     currIdentifierInOpen = true;
-                } else if (bufferedActivities.some(x => fu.search(getActivityIdentifier(x)).length > 0) && validSub && minMet) {
+                } else if (bufferedActivities.some(x => {
+                    let buffIdentifier = this.getActivityIdentifier(x);
+                    const buffMatch = stringSameness(identifier, buffIdentifier);
+                    return buffMatch.highScoreWeighted >= this.matchScore;
+                }) && validSub && minMet) {
                     updatedOpenSets.push(o);
                 } else if(!currIdentifierInOpen && !isUrl) {
                     updatedAllSets.push(o);
@@ -193,15 +227,18 @@ export class RepeatActivityRule extends Rule {
                     // could be that a spammer is using different URLs for each submission but similar submission titles so search by title as well
                     const sub = activity as Submission;
                     identifier = sub.title;
-                    fu = new Fuse([identifier], !isUrl ? fuzzyOptions : {...fuzzyOptions, distance: 5});
+                    //fu = new Fuse([identifier], !isUrl ? fuzzyOptions : {...fuzzyOptions, distance: 5});
                     minMet = identifier.length >= this.minWordCount;
                     for (const o of openSets) {
-                        const res = fu.search(o.identifier);
-                        const match = res.length > 0;
-                        if (match && validSub && minMet) {
+                        const strMatchResults = stringSameness(o.identifier, identifier);
+                        if (strMatchResults.highScoreWeighted >= this.matchScore && minMet) {
                             updatedOpenSets.push({...o, sets: [...o.sets, activity]});
                             currIdentifierInOpen = true;
-                        } else if (bufferedActivities.some(x => fu.search(getActivityIdentifier(x)).length > 0) && validSub && minMet && !updatedOpenSets.includes(o)) {
+                        } else if (bufferedActivities.some(x => {
+                            let buffIdentifier = this.getActivityIdentifier(x);
+                            const buffMatch = stringSameness(identifier, buffIdentifier);
+                            return buffMatch.highScoreWeighted >= this.matchScore;
+                        }) && validSub && minMet && !updatedOpenSets.includes(o)) {
                             updatedOpenSets.push(o);
                         } else if(!updatedAllSets.includes(o)) {
                             updatedAllSets.push(o);
@@ -232,7 +269,7 @@ export class RepeatActivityRule extends Rule {
         let applicableGroupedActivities = identifierGroupedActivities;
         if (this.useSubmissionAsReference) {
             applicableGroupedActivities = new Map();
-            let identifier = getActivityIdentifier(item);
+            let identifier = this.getActivityIdentifier(item);
             let referenceSubmissions = identifierGroupedActivities.get(identifier);
             if(referenceSubmissions === undefined && isExternalUrlSubmission(item)) {
                 // if external url sub then try by title
@@ -240,7 +277,7 @@ export class RepeatActivityRule extends Rule {
                 referenceSubmissions = identifierGroupedActivities.get(identifier);
                 if(referenceSubmissions === undefined) {
                     // didn't get by title so go back to url since that's the default
-                    identifier = getActivityIdentifier(item);
+                    identifier = this.getActivityIdentifier(item);
                 }
             }
 
@@ -265,7 +302,7 @@ export class RepeatActivityRule extends Rule {
             };
             for (let set of value) {
                 const test = comparisonTextOp(set.length, operator, thresholdValue);
-                const md = set.map((x: (Comment | Submission)) => `[${asSubmission(x) ? x.title : getActivityIdentifier(x, 50)}](https://reddit.com${x.permalink}) in ${x.subreddit_name_prefixed} on ${dayjs(x.created_utc * 1000).utc().format()}`);
+                const md = set.map((x: (Comment | Submission)) => `[${asSubmission(x) ? x.title : this.getActivityIdentifier(x, 50)}](https://reddit.com${x.permalink}) in ${x.subreddit_name_prefixed} on ${dayjs(x.created_utc * 1000).utc().format()}`);
 
                 summaryData.sets.push(set);
                 summaryData.largestTrigger = Math.max(summaryData.largestTrigger, set.length);
@@ -325,7 +362,7 @@ interface SummaryData {
     triggeringSetsMarkdown: string[]
 }
 
-interface RepeatActivityConfig extends ActivityWindow, ReferenceSubmission {
+interface RepeatActivityConfig extends ActivityWindow, ReferenceSubmission, TextMatchOptions {
     /**
      * The number of repeat submissions that will trigger the rule
      * @default ">= 5"
@@ -383,18 +420,9 @@ interface RepeatActivityConfig extends ActivityWindow, ReferenceSubmission {
     keepRemoved?: boolean
 
     /**
-     * For activities that are text-based this is the minimum number of words required for the activity to be considered for a repeat
-     *
-     * EX if `minimumWordCount=5` and a comment is `what about you` then it is ignored because `3 is less than 5`
-     *
-     * **For self-text submissions** -- title + body text
-     *
-     * **For comments* -- body text
-     *
-     * @default 1
-     * @example [1]
+     * A set of search-and-replace operations to perform on text values before performing a match. Transformations are performed in the order they are defined.
      * */
-    minWordCount?: number,
+    transformations?: SearchAndReplaceRegExp[]
 }
 
 export interface RepeatActivityOptions extends RepeatActivityConfig, RuleOptions {

src/Rule/RepostRule.ts

@@ -18,7 +18,7 @@ import {
     RepostItem,
     RepostItemResult,
     SearchAndReplaceRegExp,
-    SearchFacetType,
+    SearchFacetType, TextMatchOptions, TextTransformOptions,
 } from "../Common/interfaces";
 import objectHash from "object-hash";
 import {getActivities, getAttributionIdentifier} from "../Utils/SnoowrapUtils";
@@ -30,59 +30,6 @@ import {rest} from "lodash";
 
 const parseYtIdentifier = parseUsableLinkIdentifier();
 
-export interface TextMatchOptions {
-    /**
-     * The percentage, as a whole number, of a repost title/comment that must match the title/comment being checked in order to consider both a match
-     *
-     * Note: Setting to 0 will make every candidate considered a match -- useful if you want to match if the URL has been reposted anywhere
-     *
-     * Defaults to `85` (85%)
-     *
-     * @default 85
-     * @example [85]
-     * */
-    matchScore?: number
-
-    /**
-     * The minimum number of words in the activity being checked for which this rule will run on
-     *
-     * If the word count is below the minimum the rule fails
-     *
-     * Defaults to 2
-     *
-     * @default 2
-     * @example [2]
-     * */
-    minWordCount?: number
-
-    /**
-     * Should text matching be case sensitive?
-     *
-     * Defaults to false
-     *
-     * @default false
-     * @example [false]
-     **/
-    caseSensitive?: boolean
-}
-
-export interface TextTransformOptions {
-    /**
-     * A set of search-and-replace operations to perform on text values before performing a match. Transformations are performed in the order they are defined.
-     *
-     * * If `transformationsActivity` IS NOT defined then these transformations will be performed on BOTH the activity text (submission title or comment) AND the repost candidate text
-     * * If `transformationsActivity` IS defined then these transformations are only performed on repost candidate text
-     * */
-    transformations?: SearchAndReplaceRegExp[]
-
-    /**
-     * Specify a separate set of transformations for the activity text (submission title or comment)
-     *
-     * To perform no transformations when `transformations` is defined set this to an empty array (`[]`)
-     * */
-    transformationsActivity?: SearchAndReplaceRegExp[]
-}
-
 export interface SearchFacetJSONConfig extends TextMatchOptions, TextTransformOptions, ActivityWindow {
     kind: SearchFacetType | SearchFacetType[]
 }

src/Rule/index.ts

@@ -2,7 +2,7 @@ import Snoowrap, {Comment} from "snoowrap";
 import Submission from "snoowrap/dist/objects/Submission";
 import {Logger} from "winston";
 import {findResultByPremise, mergeArr} from "../util";
-import {SubredditResources} from "../Subreddit/SubredditResources";
+import {checkAuthorFilter, SubredditResources} from "../Subreddit/SubredditResources";
 import {ChecksActivityState, TypedActivityStates} from "../Common/interfaces";
 import Author, {AuthorOptions} from "../Author/Author";
 
@@ -65,6 +65,7 @@ export abstract class Rule implements IRule, Triggerable {
             name = this.getKind(),
             logger,
             authorIs: {
+                excludeCondition = 'OR',
                 include = [],
                 exclude = [],
             } = {},
@@ -78,6 +79,7 @@ export abstract class Rule implements IRule, Triggerable {
         this.client = client;
 
         this.authorIs = {
+            excludeCondition,
             exclude: exclude.map(x => new Author(x)),
             include: include.map(x => new Author(x)),
         }
@@ -99,23 +101,10 @@ export abstract class Rule implements IRule, Triggerable {
                 this.logger.verbose(`(Skipped) Item did not pass 'itemIs' test`);
                 return Promise.resolve([null, this.getResult(null, {result: `Item did not pass 'itemIs' test`})]);
             }
-            if (this.authorIs.include !== undefined && this.authorIs.include.length > 0) {
-                for (const auth of this.authorIs.include) {
-                    if (await this.resources.testAuthorCriteria(item, auth)) {
-                        return this.process(item);
-                    }
-                }
-                this.logger.verbose('(Skipped) Inclusive author criteria not matched');
-                return Promise.resolve([null, this.getResult(null, {result: 'Inclusive author criteria not matched'})]);
-            }
-            if (this.authorIs.exclude !== undefined && this.authorIs.exclude.length > 0) {
-                for (const auth of this.authorIs.exclude) {
-                    if (await this.resources.testAuthorCriteria(item, auth, false)) {
-                        return this.process(item);
-                    }
-                }
-                this.logger.verbose('(Skipped) Exclusive author criteria not matched');
-                return Promise.resolve([null, this.getResult(null, {result: 'Exclusive author criteria not matched'})]);
+            const [authFilterResult, authFilterType] = await checkAuthorFilter(item, this.authorIs, this.resources, this.logger);
+            if(!authFilterResult) {
+                this.logger.verbose(`(Skipped) ${authFilterType} Author criteria not matched`);
+                return Promise.resolve([null, this.getResult(null, {result: `${authFilterType} author criteria not matched`})]);
             }
         } catch (err: any) {
             this.logger.error('Error occurred during Rule pre-process checks');

src/Schema/Action.json

@@ -50,24 +50,52 @@
                     ]
                 },
                 "flairCssClass": {
-                    "description": "A list of (user) flair css class values from the subreddit to match against",
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ],
+                    "description": "A (user) flair css class (or list of) from the subreddit to match against",
                     "examples": [
                         "red"
+                    ]
+                },
+                "flairTemplate": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
                     ],
-                    "items": {
-                        "type": "string"
-                    },
-                    "type": "array"
+                    "description": "A (user) flair template id (or list of) from the subreddit to match against"
                 },
                 "flairText": {
-                    "description": "A list of (user) flair text values from the subreddit to match against",
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ],
+                    "description": "A (user) flair text value (or list of) from the subreddit to match against",
                     "examples": [
                         "Approved"
-                    ],
-                    "items": {
-                        "type": "string"
-                    },
-                    "type": "array"
+                    ]
                 },
                 "isMod": {
                     "description": "Is the author a moderator?",
@@ -131,12 +159,21 @@
             ],
             "properties": {
                 "exclude": {
-                    "description": "Only runs if `include` is not present. Will \"pass\" if any of set of the AuthorCriteria **does not** pass",
+                    "description": "Only runs if `include` is not present. Each AuthorCriteria is comprised of conditions that the Author being checked must \"not\" pass. See excludeCondition for set behavior\n\nEX: `isMod: true, name: Automoderator` => Will pass if the Author IS NOT a mod and IS NOT named Automoderator",
                     "items": {
                         "$ref": "#/definitions/AuthorCriteria"
                     },
                     "type": "array"
                 },
+                "excludeCondition": {
+                    "default": "OR",
+                    "description": "* OR => if ANY exclude condition \"does not\" pass then the exclude test passes\n* AND => if ALL exclude conditions \"do not\" pass then the exclude test passes\n\nDefaults to OR",
+                    "enum": [
+                        "AND",
+                        "OR"
+                    ],
+                    "type": "string"
+                },
                 "include": {
                     "description": "Will \"pass\" if any set of AuthorCriteria passes",
                     "items": {
@@ -240,14 +277,51 @@
                 "filtered": {
                     "type": "boolean"
                 },
+                "flairTemplate": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                 "is_self": {
                     "type": "boolean"
                 },
                 "link_flair_css_class": {
-                    "type": "string"
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
                 },
                 "link_flair_text": {
-                    "type": "string"
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
                 },
                 "locked": {
                     "type": "boolean"

src/Schema/App.json

@@ -222,6 +222,17 @@
                     ],
                     "pattern": "^[a-zA-Z]([\\w -]*[\\w])?$",
                     "type": "string"
+                },
+                "targets": {
+                    "description": "Specify which Activities to approve\n\nThis setting is only applicable if the Activity being acted on is a **comment**. On a **submission** the setting does nothing\n\n* self => approve activity being checked (comment)\n* parent => approve parent (submission) of activity being checked (comment)",
+                    "items": {
+                        "enum": [
+                            "parent",
+                            "self"
+                        ],
+                        "type": "string"
+                    },
+                    "type": "array"
                 }
             },
             "required": [
@@ -235,8 +246,7 @@
                     "default": "undefined",
                     "description": "This list determines which categories of domains should be aggregated on. All aggregated domains will be tested against `threshold`\n\n* If `media` is included then aggregate author's submission history which reddit recognizes as media (youtube, vimeo, etc.)\n* If `redditMedia` is included then aggregate on author's submissions history which are media hosted on reddit: galleries, videos, and images (i.redd.it / v.redd.it)\n* If `self` is included then aggregate on author's submission history which are self-post (`self.[subreddit]`) or domain is `reddit.com`\n* If `link` is included then aggregate author's submission history which is external links and not recognized as `media` by reddit\n\nIf nothing is specified or list is empty (default) rule will only aggregate on `link` and `media` (ignores reddit-hosted content and self-posts)",
                     "examples": [
-                        [
-                        ]
+                        []
                     ],
                     "items": {
                         "enum": [
@@ -269,8 +279,7 @@
                 },
                 "domains": {
                     "default": [
-                        [
-                        ]
+                        []
                     ],
                     "description": "A list of domains whose Activities will be tested against `threshold`.\n\nThe values are tested as partial strings so you do not need to include full URLs, just the part that matters.\n\nEX `[\"youtube\"]` will match submissions with the domain `https://youtube.com/c/aChannel`\nEX `[\"youtube.com/c/bChannel\"]` will NOT match submissions with the domain `https://youtube.com/c/aChannel`\n\nIf you wish to aggregate on self-posts for a subreddit use the syntax `self.[subreddit]` EX `self.AskReddit`\n\n**If this Rule is part of a Check for a Submission and you wish to aggregate on the domain of the Submission use the special string `AGG:SELF`**\n\nIf nothing is specified or list is empty (default) aggregate using `aggregateOn`",
                     "items": {
@@ -505,24 +514,52 @@
                     ]
                 },
                 "flairCssClass": {
-                    "description": "A list of (user) flair css class values from the subreddit to match against",
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ],
+                    "description": "A (user) flair css class (or list of) from the subreddit to match against",
                     "examples": [
                         "red"
+                    ]
+                },
+                "flairTemplate": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
                     ],
-                    "items": {
-                        "type": "string"
-                    },
-                    "type": "array"
+                    "description": "A (user) flair template id (or list of) from the subreddit to match against"
                 },
                 "flairText": {
-                    "description": "A list of (user) flair text values from the subreddit to match against",
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ],
+                    "description": "A (user) flair text value (or list of) from the subreddit to match against",
                     "examples": [
                         "Approved"
-                    ],
-                    "items": {
-                        "type": "string"
-                    },
-                    "type": "array"
+                    ]
                 },
                 "isMod": {
                     "description": "Is the author a moderator?",
@@ -586,12 +623,21 @@
             ],
             "properties": {
                 "exclude": {
-                    "description": "Only runs if `include` is not present. Will \"pass\" if any of set of the AuthorCriteria **does not** pass",
+                    "description": "Only runs if `include` is not present. Each AuthorCriteria is comprised of conditions that the Author being checked must \"not\" pass. See excludeCondition for set behavior\n\nEX: `isMod: true, name: Automoderator` => Will pass if the Author IS NOT a mod and IS NOT named Automoderator",
                     "items": {
                         "$ref": "#/definitions/AuthorCriteria"
                     },
                     "type": "array"
                 },
+                "excludeCondition": {
+                    "default": "OR",
+                    "description": "* OR => if ANY exclude condition \"does not\" pass then the exclude test passes\n* AND => if ALL exclude conditions \"do not\" pass then the exclude test passes\n\nDefaults to OR",
+                    "enum": [
+                        "AND",
+                        "OR"
+                    ],
+                    "type": "string"
+                },
                 "include": {
                     "description": "Will \"pass\" if any set of AuthorCriteria passes",
                     "items": {
@@ -915,8 +961,7 @@
             "type": "object"
         },
         "CacheOptions": {
-            "additionalProperties": {
-            },
+            "additionalProperties": {},
             "description": "Configure granular settings for a cache provider with this object",
             "properties": {
                 "auth_pass": {
@@ -981,25 +1026,6 @@
             ],
             "type": "string"
         },
-        "ClearProcessedOptions": {
-            "description": "For very long-running, high-volume subreddits clearing the list of processed activities helps manage memory bloat\n\nAll of these options have default values based on the limit and/or interval set for polling options on each subreddit stream. They only need to modified if the defaults are not sufficient.\n\nIf both `after` and `size` are defined whichever is hit first will trigger the list to clear. `after` will be reset after ever clear.",
-            "properties": {
-                "after": {
-                    "description": "An interval the processed list should be cleared after.\n\n* EX `9 days`\n* EX `3 months`\n* EX `5 minutes`",
-                    "pattern": "^\\s*(?<time>\\d+)\\s*(?<unit>days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)\\s*$",
-                    "type": "string"
-                },
-                "retain": {
-                    "description": "The number of activities to retain in processed list after clearing.\n\nDefaults to `limit` value from `PollingOptions`",
-                    "type": "number"
-                },
-                "size": {
-                    "description": "Number of activities found in processed list after which the list should be cleared.\n\nDefaults to the `limit` value from `PollingOptions`",
-                    "type": "number"
-                }
-            },
-            "type": "object"
-        },
         "CommentActionJson": {
             "description": "Reply to the Activity. For a submission the reply will be a top-level comment.",
             "properties": {
@@ -1436,6 +1462,61 @@
             },
             "type": "object"
         },
+        "FilterCriteriaDefaults": {
+            "properties": {
+                "authorIs": {
+                    "$ref": "#/definitions/AuthorOptions",
+                    "description": "Determine how authorIs defaults behave when authorIs is present on the check\n\n* merge => merges defaults with check's authorIs\n* replace => check authorIs will replace defaults (no defaults used)",
+                    "examples": [
+                        {
+                            "include": [
+                                {
+                                    "flairText": [
+                                        "Contributor",
+                                        "Veteran"
+                                    ]
+                                },
+                                {
+                                    "isMod": true
+                                }
+                            ]
+                        }
+                    ]
+                },
+                "authorIsBehavior": {
+                    "enum": [
+                        "merge",
+                        "replace"
+                    ],
+                    "type": "string"
+                },
+                "itemIs": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "$ref": "#/definitions/SubmissionState"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "items": {
+                                "$ref": "#/definitions/CommentState"
+                            },
+                            "type": "array"
+                        }
+                    ]
+                },
+                "itemIsBehavior": {
+                    "description": "Determine how itemIs defaults behave when itemIs is present on the check\n\n* merge => adds defaults to check's itemIs\n* replace => check itemIs will replace defaults (no defaults used)",
+                    "enum": [
+                        "merge",
+                        "replace"
+                    ],
+                    "type": "string"
+                }
+            },
+            "type": "object"
+        },
         "FlairActionJson": {
             "description": "Flair the Submission",
             "properties": {
@@ -2110,10 +2191,6 @@
                 }
             ],
             "properties": {
-                "clearProcessed": {
-                    "$ref": "#/definitions/ClearProcessedOptions",
-                    "description": "For very long-running, high-volume subreddits clearing the list of processed activities helps manage memory bloat\n\nAll of these options have default values based on the limit and/or interval set for polling options on each subreddit stream. They only need to modified if the defaults are not sufficient.\n\nIf both `after` and `size` are defined whichever is hit first will trigger the list to clear. `after` will be reset after ever clear."
-                },
                 "delayUntil": {
                     "description": "Delay processing Activity until it is `N` seconds old\n\nUseful if there are other bots that may process an Activity and you want this bot to run first/last/etc.\n\nIf the Activity is already `N` seconds old when it is initially retrieved no refresh of the Activity occurs (no API request is made) and it is immediately processed.",
                     "type": "number"
@@ -2336,6 +2413,11 @@
                     "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s+.*)*$",
                     "type": "string"
                 },
+                "mustMatchCurrent": {
+                    "default": false,
+                    "description": "When `true` the Activity being checked MUST pass the `matchThreshold` before the Rule considers any history\n\nFor use with `activityMatchThreshold`/`totalMatchThreshold` -- useful to conserve API calls",
+                    "type": "boolean"
+                },
                 "name": {
                     "description": "A descriptive name that will be used in logging and be available for templating",
                     "examples": [
@@ -2558,6 +2640,9 @@
                     ],
                     "pattern": "^[a-zA-Z]([\\w -]*[\\w])?$",
                     "type": "string"
+                },
+                "spam": {
+                    "type": "boolean"
                 }
             },
             "required": [
@@ -2587,6 +2672,11 @@
                         }
                     ]
                 },
+                "caseSensitive": {
+                    "default": false,
+                    "description": "Should text matching be case sensitive?\n\nDefaults to false",
+                    "type": "boolean"
+                },
                 "exclude": {
                     "description": "If present, activities will be counted only if they are **NOT** found in this list of Subreddits\n\nEach value in the list can be either:\n\n * string (name of subreddit)\n * regular expression to run on the subreddit name\n * `SubredditState`\n\nEX `[\"mealtimevideos\",\"askscience\", \"/onlyfans*\\/i\", {\"over18\": true}]`",
                     "examples": [
@@ -2677,9 +2767,14 @@
                     ],
                     "type": "string"
                 },
+                "matchScore": {
+                    "default": 85,
+                    "description": "The percentage, as a whole number, of a repost title/comment that must match the title/comment being checked in order to consider both a match\n\nNote: Setting to 0 will make every candidate considered a match -- useful if you want to match if the URL has been reposted anywhere\n\nDefaults to `85` (85%)",
+                    "type": "number"
+                },
                 "minWordCount": {
-                    "default": 1,
-                    "description": "For activities that are text-based this is the minimum number of words required for the activity to be considered for a repeat\n\nEX if `minimumWordCount=5` and a comment is `what about you` then it is ignored because `3 is less than 5`\n\n**For self-text submissions** -- title + body text\n\n**For comments* -- body text",
+                    "default": 2,
+                    "description": "The minimum number of words in the activity being checked for which this rule will run on\n\nIf the word count is below the minimum the rule fails\n\nDefaults to 2",
                     "type": "number"
                 },
                 "name": {
@@ -2695,6 +2790,13 @@
                     "description": "The number of repeat submissions that will trigger the rule",
                     "type": "string"
                 },
+                "transformations": {
+                    "description": "A set of search-and-replace operations to perform on text values before performing a match. Transformations are performed in the order they are defined.",
+                    "items": {
+                        "$ref": "#/definitions/SearchAndReplaceRegExp"
+                    },
+                    "type": "array"
+                },
                 "useSubmissionAsReference": {
                     "default": true,
                     "description": "If activity is a Submission and is a link (not self-post) then only look at Submissions that contain this link, otherwise consider all activities.",
@@ -3398,14 +3500,51 @@
                 "filtered": {
                     "type": "boolean"
                 },
+                "flairTemplate": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                 "is_self": {
                     "type": "boolean"
                 },
                 "link_flair_css_class": {
-                    "type": "string"
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
                 },
                 "link_flair_text": {
-                    "type": "string"
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
                 },
                 "locked": {
                     "type": "boolean"
@@ -3488,8 +3627,7 @@
             "type": "object"
         },
         "ThirdPartyCredentialsJsonConfig": {
-            "additionalProperties": {
-            },
+            "additionalProperties": {},
             "properties": {
                 "youtube": {
                     "properties": {
@@ -3798,6 +3936,10 @@
             ],
             "type": "boolean"
         },
+        "filterCriteriaDefaults": {
+            "$ref": "#/definitions/FilterCriteriaDefaults",
+            "description": "Set the default filter criteria for all checks. If this property is specified it will override any defaults passed from the bot's config\n\nDefault behavior is to exclude all mods and automoderator from checks"
+        },
         "footer": {
             "anyOf": [
                 {

src/Schema/OperatorConfig.json

@@ -1,6 +1,189 @@
 {
     "$schema": "http://json-schema.org/draft-07/schema#",
     "definitions": {
+        "AuthorCriteria": {
+            "additionalProperties": false,
+            "description": "Criteria with which to test against the author of an Activity. The outcome of the test is based on:\n\n1. All present properties passing and\n2. If a property is a list then any value from the list matching",
+            "examples": [
+                {
+                    "flairText": [
+                        "Contributor",
+                        "Veteran"
+                    ],
+                    "isMod": true,
+                    "name": [
+                        "FoxxMD",
+                        "AnotherUser"
+                    ]
+                }
+            ],
+            "minProperties": 1,
+            "properties": {
+                "age": {
+                    "description": "Test the age of the Author's account (when it was created) against this comparison\n\nThe syntax is `(< OR > OR <= OR >=) <number> <unit>`\n\n* EX `> 100 days` => Passes if Author's account is older than 100 days\n* EX `<= 2 months` => Passes if Author's account is younger than or equal to 2 months\n\nUnit must be one of [DayJS Duration units](https://day.js.org/docs/en/durations/creating)\n\n[See] https://regexr.com/609n8 for example",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(days?|weeks?|months?|years?|hours?|minutes?|seconds?|milliseconds?)\\s*$",
+                    "type": "string"
+                },
+                "commentKarma": {
+                    "description": "A string containing a comparison operator and a value to compare karma against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign]`\n\n* EX `> 100`  => greater than 100 comment karma\n* EX `<= 75%` => comment karma is less than or equal to 75% of **all karma**",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "type": "string"
+                },
+                "description": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ],
+                    "description": "An (array of) string/regular expression to test contents of an Author's profile description against\n\nIf no flags are specified then the **insensitive** flag is used by default\n\nIf using an array then if **any** value in the array passes the description test passes",
+                    "examples": [
+                        [
+                            "/test$/i",
+                            "look for this string literal"
+                        ]
+                    ]
+                },
+                "flairCssClass": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ],
+                    "description": "A (user) flair css class (or list of) from the subreddit to match against",
+                    "examples": [
+                        "red"
+                    ]
+                },
+                "flairTemplate": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ],
+                    "description": "A (user) flair template id (or list of) from the subreddit to match against"
+                },
+                "flairText": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ],
+                    "description": "A (user) flair text value (or list of) from the subreddit to match against",
+                    "examples": [
+                        "Approved"
+                    ]
+                },
+                "isMod": {
+                    "description": "Is the author a moderator?",
+                    "type": "boolean"
+                },
+                "linkKarma": {
+                    "description": "A string containing a comparison operator and a value to compare link karma against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign]`\n\n* EX `> 100`  => greater than 100 link karma\n* EX `<= 75%` => link karma is less than or equal to 75% of **all karma**",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "type": "string"
+                },
+                "name": {
+                    "description": "A list of reddit usernames (case-insensitive) to match against. Do not include the \"u/\" prefix\n\n EX to match against /u/FoxxMD and /u/AnotherUser use [\"FoxxMD\",\"AnotherUser\"]",
+                    "examples": [
+                        "FoxxMD",
+                        "AnotherUser"
+                    ],
+                    "items": {
+                        "type": "string"
+                    },
+                    "type": "array"
+                },
+                "shadowBanned": {
+                    "description": "Is the author shadowbanned?\n\nThis is determined by trying to retrieve the author's profile. If a 404 is returned it is likely they are shadowbanned",
+                    "type": "boolean"
+                },
+                "totalKarma": {
+                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 100`  => greater than 100",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "type": "string"
+                },
+                "userNotes": {
+                    "description": "A list of UserNote properties to check against the User Notes attached to this Author in this Subreddit (must have Toolbox enabled and used User Notes at least once)",
+                    "items": {
+                        "$ref": "#/definitions/UserNoteCriteria"
+                    },
+                    "type": "array"
+                },
+                "verified": {
+                    "description": "Does Author's account have a verified email?",
+                    "type": "boolean"
+                }
+            },
+            "type": "object"
+        },
+        "AuthorOptions": {
+            "description": "If present then these Author criteria are checked before running the rule. If criteria fails then the rule is skipped.",
+            "examples": [
+                {
+                    "include": [
+                        {
+                            "flairText": [
+                                "Contributor",
+                                "Veteran"
+                            ]
+                        },
+                        {
+                            "isMod": true
+                        }
+                    ]
+                }
+            ],
+            "properties": {
+                "exclude": {
+                    "description": "Only runs if `include` is not present. Each AuthorCriteria is comprised of conditions that the Author being checked must \"not\" pass. See excludeCondition for set behavior\n\nEX: `isMod: true, name: Automoderator` => Will pass if the Author IS NOT a mod and IS NOT named Automoderator",
+                    "items": {
+                        "$ref": "#/definitions/AuthorCriteria"
+                    },
+                    "type": "array"
+                },
+                "excludeCondition": {
+                    "default": "OR",
+                    "description": "* OR => if ANY exclude condition \"does not\" pass then the exclude test passes\n* AND => if ALL exclude conditions \"do not\" pass then the exclude test passes\n\nDefaults to OR",
+                    "enum": [
+                        "AND",
+                        "OR"
+                    ],
+                    "type": "string"
+                },
+                "include": {
+                    "description": "Will \"pass\" if any set of AuthorCriteria passes",
+                    "items": {
+                        "$ref": "#/definitions/AuthorCriteria"
+                    },
+                    "type": "array"
+                }
+            },
+            "type": "object"
+        },
         "BotConnection": {
             "description": "Configuration required to connect to a CM Server",
             "properties": {
@@ -58,6 +241,10 @@
                         }
                     ]
                 },
+                "filterCriteriaDefaults": {
+                    "$ref": "#/definitions/FilterCriteriaDefaults",
+                    "description": "Define the default behavior for all filter criteria on all checks in all subreddits\n\nDefaults to exclude mods and automoderator from checks"
+                },
                 "name": {
                     "type": "string"
                 },
@@ -94,13 +281,36 @@
                         },
                         {
                             "properties": {
+                                "shared": {
+                                    "anyOf": [
+                                        {
+                                            "items": {
+                                                "enum": [
+                                                    "modqueue",
+                                                    "newComm",
+                                                    "newSub",
+                                                    "unmoderated"
+                                                ],
+                                                "type": "string"
+                                            },
+                                            "type": "array"
+                                        },
+                                        {
+                                            "enum": [
+                                                true
+                                            ],
+                                            "type": "boolean"
+                                        }
+                                    ],
+                                    "description": "Set which polling sources should be shared among subreddits using default polling settings for that source\n\n* For `unmoderated and `modqueue` the bot will poll on **r/mod** for new activities\n* For `newSub` and `newComm` all subreddits sharing the source will be combined to poll like **r/subreddit1+subreddit2/new**\n\nIf set to `true` all polling sources will be shared,  otherwise specify which sourcs should be shared as a list"
+                                },
                                 "sharedMod": {
                                     "default": false,
-                                    "description": "If set to `true` all subreddits polling unmoderated/modqueue with default polling settings will share a request to \"r/mod\"\notherwise each subreddit will poll its own mod view\n\n* ENV => `SHARE_MOD`\n* ARG => `--shareMod`",
+                                    "description": "DEPRECATED: See `shared`\n\n Using the ENV or ARG will sett `unmoderated` and `modqueue` on `shared`\n\n* ENV => `SHARE_MOD`\n* ARG => `--shareMod`",
                                     "type": "boolean"
                                 },
                                 "stagger": {
-                                    "description": "If sharing a mod stream stagger pushing relevant Activities to individual subreddits.\n\nUseful when running many subreddits and rules are potentially cpu/memory/traffic heavy -- allows spreading out load",
+                                    "description": "If sharing a stream staggers pushing relevant Activities to individual subreddits.\n\nUseful when running many subreddits and rules are potentially cpu/memory/traffic heavy -- allows spreading out load",
                                     "type": "number"
                                 }
                             },
@@ -187,8 +397,7 @@
             "type": "object"
         },
         "CacheOptions": {
-            "additionalProperties": {
-            },
+            "additionalProperties": {},
             "description": "Configure granular settings for a cache provider with this object",
             "properties": {
                 "auth_pass": {
@@ -253,6 +462,73 @@
             ],
             "type": "string"
         },
+        "CommentState": {
+            "description": "Different attributes a `Comment` can be in. Only include a property if you want to check it.",
+            "examples": [
+                {
+                    "op": true,
+                    "removed": false
+                }
+            ],
+            "properties": {
+                "age": {
+                    "description": "A duration and how to compare it against a value\n\nThe syntax is `(< OR > OR <= OR >=) <number> <unit>` EX `> 100 days`, `<= 2 months`\n\n* EX `> 100 days` => Passes if the date being compared is before 100 days ago\n* EX `<= 2 months` => Passes if the date being compared is after or equal to 2 months\n\nUnit must be one of [DayJS Duration units](https://day.js.org/docs/en/durations/creating)\n\n[See] https://regexr.com/609n8 for example",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(days|weeks|months|years|hours|minutes|seconds|milliseconds)\\s*$",
+                    "type": "string"
+                },
+                "approved": {
+                    "type": "boolean"
+                },
+                "deleted": {
+                    "type": "boolean"
+                },
+                "depth": {
+                    "description": "The (nested) level of a comment.\n\n* 0 mean the comment is at top-level (replying to submission)\n* non-zero, Nth value means the comment has N parent comments",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(days|weeks|months|years|hours|minutes|seconds|milliseconds)\\s*$",
+                    "type": "string"
+                },
+                "distinguished": {
+                    "type": "boolean"
+                },
+                "filtered": {
+                    "type": "boolean"
+                },
+                "locked": {
+                    "type": "boolean"
+                },
+                "op": {
+                    "description": "Is this Comment Author also the Author of the Submission this comment is in?",
+                    "type": "boolean"
+                },
+                "removed": {
+                    "type": "boolean"
+                },
+                "reports": {
+                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 2`  => greater than 2 total reports\n\nDefaults to TOTAL reports on an Activity. Suffix the value with the report type to check that type:\n\n* EX `> 3 mod` => greater than 3 mod reports\n* EX `>= 1 user` => greater than 1 user report",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "type": "string"
+                },
+                "score": {
+                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 100`  => greater than 100",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "type": "string"
+                },
+                "spam": {
+                    "type": "boolean"
+                },
+                "stickied": {
+                    "type": "boolean"
+                },
+                "submissionState": {
+                    "description": "A list of SubmissionState attributes to test the Submission this comment is in",
+                    "items": {
+                        "$ref": "#/definitions/SubmissionState"
+                    },
+                    "type": "array"
+                }
+            },
+            "type": "object"
+        },
         "DiscordProviderConfig": {
             "properties": {
                 "name": {
@@ -275,6 +551,172 @@
             ],
             "type": "object"
         },
+        "FilterCriteriaDefaults": {
+            "properties": {
+                "authorIs": {
+                    "$ref": "#/definitions/AuthorOptions",
+                    "description": "Determine how authorIs defaults behave when authorIs is present on the check\n\n* merge => merges defaults with check's authorIs\n* replace => check authorIs will replace defaults (no defaults used)",
+                    "examples": [
+                        {
+                            "include": [
+                                {
+                                    "flairText": [
+                                        "Contributor",
+                                        "Veteran"
+                                    ]
+                                },
+                                {
+                                    "isMod": true
+                                }
+                            ]
+                        }
+                    ]
+                },
+                "authorIsBehavior": {
+                    "enum": [
+                        "merge",
+                        "replace"
+                    ],
+                    "type": "string"
+                },
+                "itemIs": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "$ref": "#/definitions/SubmissionState"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "items": {
+                                "$ref": "#/definitions/CommentState"
+                            },
+                            "type": "array"
+                        }
+                    ]
+                },
+                "itemIsBehavior": {
+                    "description": "Determine how itemIs defaults behave when itemIs is present on the check\n\n* merge => adds defaults to check's itemIs\n* replace => check itemIs will replace defaults (no defaults used)",
+                    "enum": [
+                        "merge",
+                        "replace"
+                    ],
+                    "type": "string"
+                }
+            },
+            "type": "object"
+        },
+        "LoggingOptions": {
+            "properties": {
+                "console": {
+                    "allOf": [
+                        {
+                            "$ref": "#/definitions/Pick<Transports.ConsoleTransportOptions,\"silent\"|\"eol\"|\"stderrLevels\"|\"consoleWarnLevels\">"
+                        },
+                        {
+                            "properties": {
+                                "level": {
+                                    "enum": [
+                                        "debug",
+                                        "error",
+                                        "info",
+                                        "verbose",
+                                        "warn"
+                                    ],
+                                    "type": "string"
+                                }
+                            },
+                            "type": "object"
+                        }
+                    ],
+                    "description": "Options for logging to console"
+                },
+                "file": {
+                    "allOf": [
+                        {
+                            "$ref": "#/definitions/Omit<DailyRotateFileTransportOptions,\"stream\"|\"dirname\"|\"options\"|\"handleRejections\"|\"format\"|\"handleExceptions\"|\"log\"|\"logv\"|\"close\">"
+                        },
+                        {
+                            "properties": {
+                                "dirname": {
+                                    "description": "The absolute path to a directory where rotating log files should be stored.\n\n* If not present or `null` or `false` no log files will be created\n* If `true` logs will be stored at `[working directory]/logs`\n\n* ENV => `LOG_DIR`\n* ARG => `--logDir [dir]`",
+                                    "examples": [
+                                        "/var/log/contextmod"
+                                    ],
+                                    "type": [
+                                        "null",
+                                        "string",
+                                        "boolean"
+                                    ]
+                                },
+                                "level": {
+                                    "enum": [
+                                        "debug",
+                                        "error",
+                                        "info",
+                                        "verbose",
+                                        "warn"
+                                    ],
+                                    "type": "string"
+                                }
+                            },
+                            "type": "object"
+                        }
+                    ],
+                    "description": "Options for Rotating File logging"
+                },
+                "level": {
+                    "default": "verbose",
+                    "description": "The minimum log level to output. The log level set will output logs at its level **and all levels above it:**\n\n * `error`\n * `warn`\n * `info`\n * `verbose`\n * `debug`\n\n Note: `verbose` will display *a lot* of information on the status/result of run rules/checks/actions etc. which is very useful for testing configurations. Once your bot is stable changing the level to `info` will reduce log noise.\n\n * ENV => `LOG_LEVEL`\n * ARG => `--logLevel <level>`",
+                    "enum": [
+                        "debug",
+                        "error",
+                        "info",
+                        "verbose",
+                        "warn"
+                    ],
+                    "examples": [
+                        "verbose"
+                    ],
+                    "type": "string"
+                },
+                "path": {
+                    "description": "**DEPRECATED** - Use `file.dirname` instead\nThe absolute path to a directory where rotating log files should be stored.\n\n* If not present or `null` or `false` no log files will be created\n* If `true` logs will be stored at `[working directory]/logs`\n\n* ENV => `LOG_DIR`\n* ARG => `--logDir [dir]`",
+                    "examples": [
+                        "/var/log/contextmod"
+                    ],
+                    "type": [
+                        "null",
+                        "string",
+                        "boolean"
+                    ]
+                },
+                "stream": {
+                    "allOf": [
+                        {
+                            "$ref": "#/definitions/Omit<DuplexTransportOptions,\"name\"|\"stream\"|\"handleRejections\"|\"format\"|\"handleExceptions\"|\"log\"|\"logv\"|\"close\">"
+                        },
+                        {
+                            "properties": {
+                                "level": {
+                                    "enum": [
+                                        "debug",
+                                        "error",
+                                        "info",
+                                        "verbose",
+                                        "warn"
+                                    ],
+                                    "type": "string"
+                                }
+                            },
+                            "type": "object"
+                        }
+                    ],
+                    "description": "Options for logging to api/web"
+                }
+            },
+            "type": "object"
+        },
         "NotificationConfig": {
             "properties": {
                 "events": {
@@ -340,6 +782,90 @@
             ],
             "type": "object"
         },
+        "Omit<DailyRotateFileTransportOptions,\"stream\"|\"dirname\"|\"options\"|\"handleRejections\"|\"format\"|\"handleExceptions\"|\"log\"|\"logv\"|\"close\">": {
+            "properties": {
+                "auditFile": {
+                    "description": "A string representing the name of the name of the audit file. (default: './hash-audit.json')",
+                    "type": "string"
+                },
+                "createSymlink": {
+                    "description": "Create a tailable symlink to the current active log file. (default: false)",
+                    "type": "boolean"
+                },
+                "datePattern": {
+                    "description": "A string representing the moment.js date format to be used for rotating. The meta characters used in this string will dictate the frequency of the file rotation. For example, if your datePattern is simply 'HH' you will end up with 24 log files that are picked up and appended to every day. (default 'YYYY-MM-DD')",
+                    "type": "string"
+                },
+                "eol": {
+                    "type": "string"
+                },
+                "extension": {
+                    "description": "A string representing an extension to be added to the filename, if not included in the filename property. (default: '')",
+                    "type": "string"
+                },
+                "filename": {
+                    "description": "Filename to be used to log to. This filename can include the %DATE% placeholder which will include the formatted datePattern at that point in the filename. (default: 'winston.log.%DATE%)",
+                    "type": "string"
+                },
+                "frequency": {
+                    "description": "A string representing the frequency of rotation. (default: 'custom')",
+                    "type": "string"
+                },
+                "json": {
+                    "type": "boolean"
+                },
+                "level": {
+                    "type": "string"
+                },
+                "maxFiles": {
+                    "description": "Maximum number of logs to keep. If not set, no logs will be removed. This can be a number of files or number of days. If using days, add 'd' as the suffix. (default: null)",
+                    "type": [
+                        "string",
+                        "number"
+                    ]
+                },
+                "maxSize": {
+                    "description": "Maximum size of the file after which it will rotate. This can be a number of bytes, or units of kb, mb, and gb. If using the units, add 'k', 'm', or 'g' as the suffix. The units need to directly follow the number. (default: null)",
+                    "type": [
+                        "string",
+                        "number"
+                    ]
+                },
+                "silent": {
+                    "type": "boolean"
+                },
+                "symlinkName": {
+                    "description": "The name of the tailable symlink. (default: 'current.log')",
+                    "type": "string"
+                },
+                "utc": {
+                    "description": "A boolean whether or not to generate file name from \"datePattern\" in UTC format. (default: false)",
+                    "type": "boolean"
+                },
+                "zippedArchive": {
+                    "description": "A boolean to define whether or not to gzip archived log files. (default 'false')",
+                    "type": "boolean"
+                }
+            },
+            "type": "object"
+        },
+        "Omit<DuplexTransportOptions,\"name\"|\"stream\"|\"handleRejections\"|\"format\"|\"handleExceptions\"|\"log\"|\"logv\"|\"close\">": {
+            "properties": {
+                "dump": {
+                    "type": "boolean"
+                },
+                "eol": {
+                    "type": "string"
+                },
+                "level": {
+                    "type": "string"
+                },
+                "silent": {
+                    "type": "boolean"
+                }
+            },
+            "type": "object"
+        },
         "OperatorCacheConfig": {
             "properties": {
                 "actionedEventsDefault": {
@@ -459,6 +985,29 @@
             },
             "type": "object"
         },
+        "Pick<Transports.ConsoleTransportOptions,\"silent\"|\"eol\"|\"stderrLevels\"|\"consoleWarnLevels\">": {
+            "properties": {
+                "consoleWarnLevels": {
+                    "items": {
+                        "type": "string"
+                    },
+                    "type": "array"
+                },
+                "eol": {
+                    "type": "string"
+                },
+                "silent": {
+                    "type": "boolean"
+                },
+                "stderrLevels": {
+                    "items": {
+                        "type": "string"
+                    },
+                    "type": "array"
+                }
+            },
+            "type": "object"
+        },
         "PollingDefaults": {
             "properties": {
                 "delayUntil": {
@@ -543,9 +1092,119 @@
             },
             "type": "object"
         },
-        "ThirdPartyCredentialsJsonConfig": {
-            "additionalProperties": {
+        "SubmissionState": {
+            "description": "Different attributes a `Submission` can be in. Only include a property if you want to check it.",
+            "examples": [
+                {
+                    "over_18": true,
+                    "removed": false
+                }
+            ],
+            "properties": {
+                "age": {
+                    "description": "A duration and how to compare it against a value\n\nThe syntax is `(< OR > OR <= OR >=) <number> <unit>` EX `> 100 days`, `<= 2 months`\n\n* EX `> 100 days` => Passes if the date being compared is before 100 days ago\n* EX `<= 2 months` => Passes if the date being compared is after or equal to 2 months\n\nUnit must be one of [DayJS Duration units](https://day.js.org/docs/en/durations/creating)\n\n[See] https://regexr.com/609n8 for example",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(days|weeks|months|years|hours|minutes|seconds|milliseconds)\\s*$",
+                    "type": "string"
+                },
+                "approved": {
+                    "type": "boolean"
+                },
+                "deleted": {
+                    "type": "boolean"
+                },
+                "distinguished": {
+                    "type": "boolean"
+                },
+                "filtered": {
+                    "type": "boolean"
+                },
+                "flairTemplate": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
+                "is_self": {
+                    "type": "boolean"
+                },
+                "link_flair_css_class": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "link_flair_text": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "locked": {
+                    "type": "boolean"
+                },
+                "over_18": {
+                    "description": "NSFW",
+                    "type": "boolean"
+                },
+                "pinned": {
+                    "type": "boolean"
+                },
+                "removed": {
+                    "type": "boolean"
+                },
+                "reports": {
+                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 2`  => greater than 2 total reports\n\nDefaults to TOTAL reports on an Activity. Suffix the value with the report type to check that type:\n\n* EX `> 3 mod` => greater than 3 mod reports\n* EX `>= 1 user` => greater than 1 user report",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "type": "string"
+                },
+                "score": {
+                    "description": "A string containing a comparison operator and a value to compare against\n\nThe syntax is `(< OR > OR <= OR >=) <number>`\n\n* EX `> 100`  => greater than 100",
+                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)\\s*(%?)(.*)$",
+                    "type": "string"
+                },
+                "spam": {
+                    "type": "boolean"
+                },
+                "spoiler": {
+                    "type": "boolean"
+                },
+                "stickied": {
+                    "type": "boolean"
+                },
+                "title": {
+                    "description": "A valid regular expression to match against the title of the submission",
+                    "type": "string"
+                }
             },
+            "type": "object"
+        },
+        "ThirdPartyCredentialsJsonConfig": {
+            "additionalProperties": {},
             "properties": {
                 "youtube": {
                     "properties": {
@@ -561,6 +1220,43 @@
             },
             "type": "object"
         },
+        "UserNoteCriteria": {
+            "properties": {
+                "count": {
+                    "default": ">= 1",
+                    "description": "Number of occurrences of this type. Ignored if `search` is `current`\n\nA string containing a comparison operator and/or a value to compare number of occurrences against\n\nThe syntax is `(< OR > OR <= OR >=) <number>[percent sign] [ascending|descending]`",
+                    "examples": [
+                        ">= 1"
+                    ],
+                    "pattern": "^\\s*(?<opStr>>|>=|<|<=)\\s*(?<value>\\d+)\\s*(?<percent>%?)\\s*(?<extra>asc.*|desc.*)*$",
+                    "type": "string"
+                },
+                "search": {
+                    "default": "current",
+                    "description": "How to test the notes for this Author:\n\n### current\n\nOnly the most recent note is checked for `type`\n\n### total\n\nThe `count` comparison of `type` must be found within all notes\n\n* EX `count: > 3`   => Must have more than 3 notes of `type`, total\n* EX `count: <= 25%` => Must have 25% or less of notes of `type`, total\n\n### consecutive\n\nThe `count` **number** of `type` notes must be found in a row.\n\nYou may also specify the time-based order in which to search the notes by specifying `ascending (asc)` or `descending (desc)` in the `count` value. Default is `descending`\n\n* EX `count: >= 3` => Must have 3 or more notes of `type` consecutively, in descending order\n* EX `count: < 2`  => Must have less than 2 notes of `type` consecutively, in descending order\n* EX `count: > 4 asc` => Must have greater than 4 notes of `type` consecutively, in ascending order",
+                    "enum": [
+                        "consecutive",
+                        "current",
+                        "total"
+                    ],
+                    "examples": [
+                        "current"
+                    ],
+                    "type": "string"
+                },
+                "type": {
+                    "description": "User Note type key to search for",
+                    "examples": [
+                        "spamwarn"
+                    ],
+                    "type": "string"
+                }
+            },
+            "required": [
+                "type"
+            ],
+            "type": "object"
+        },
         "WebCredentials": {
             "description": "Separate credentials for the web interface can be provided when also running the api.\n\nAll properties not specified will default to values given in ENV/ARG credential properties\n\nRefer to the [required credentials table](https://github.com/FoxxMD/context-mod/blob/master/docs/operatorConfiguration.md#minimum-required-configuration) to see what is necessary for the web interface.",
             "examples": [
@@ -634,32 +1330,8 @@
             "$ref": "#/definitions/ThirdPartyCredentialsJsonConfig"
         },
         "logging": {
-            "description": "Settings to configure global logging defaults",
-            "properties": {
-                "level": {
-                    "default": "verbose",
-                    "description": "The minimum log level to output. The log level set will output logs at its level **and all levels above it:**\n\n * `error`\n * `warn`\n * `info`\n * `verbose`\n * `debug`\n\n Note: `verbose` will display *a lot* of information on the status/result of run rules/checks/actions etc. which is very useful for testing configurations. Once your bot is stable changing the level to `info` will reduce log noise.\n\n * ENV => `LOG_LEVEL`\n * ARG => `--logLevel <level>`",
-                    "enum": [
-                        "debug",
-                        "error",
-                        "info",
-                        "verbose",
-                        "warn"
-                    ],
-                    "examples": [
-                        "verbose"
-                    ],
-                    "type": "string"
-                },
-                "path": {
-                    "description": "The absolute path to a directory where rotating log files should be stored.\n\n* If not present or `null` no log files will be created\n* If `true` logs will be stored at `[working directory]/logs`\n\n* ENV => `LOG_DIR`\n* ARG => `--logDir [dir]`",
-                    "examples": [
-                        "/var/log/contextmod"
-                    ],
-                    "type": "string"
-                }
-            },
-            "type": "object"
+            "$ref": "#/definitions/LoggingOptions",
+            "description": "Settings to configure global logging defaults"
         },
         "mode": {
             "default": "all",
@@ -712,6 +1384,10 @@
             "$ref": "#/definitions/SnoowrapOptions",
             "description": "Set global snoowrap options as well as default snoowrap config for all bots that don't specify their own"
         },
+        "userAgent": {
+            "description": "Added to the User-Agent information sent to reddit\n\nThis string will be added BETWEEN version and your bot name.\n\nEX: `myBranch` => `web:contextMod:v1.0.0-myBranch:BOT-/u/MyBotUser`\n\n* ENV => `USER_AGENT`",
+            "type": "string"
+        },
         "web": {
             "description": "Settings for the web interface",
             "properties": {

src/Schema/Rule.json

@@ -184,8 +184,7 @@
                     "default": "undefined",
                     "description": "This list determines which categories of domains should be aggregated on. All aggregated domains will be tested against `threshold`\n\n* If `media` is included then aggregate author's submission history which reddit recognizes as media (youtube, vimeo, etc.)\n* If `redditMedia` is included then aggregate on author's submissions history which are media hosted on reddit: galleries, videos, and images (i.redd.it / v.redd.it)\n* If `self` is included then aggregate on author's submission history which are self-post (`self.[subreddit]`) or domain is `reddit.com`\n* If `link` is included then aggregate author's submission history which is external links and not recognized as `media` by reddit\n\nIf nothing is specified or list is empty (default) rule will only aggregate on `link` and `media` (ignores reddit-hosted content and self-posts)",
                     "examples": [
-                        [
-                        ]
+                        []
                     ],
                     "items": {
                         "enum": [
@@ -218,8 +217,7 @@
                 },
                 "domains": {
                     "default": [
-                        [
-                        ]
+                        []
                     ],
                     "description": "A list of domains whose Activities will be tested against `threshold`.\n\nThe values are tested as partial strings so you do not need to include full URLs, just the part that matters.\n\nEX `[\"youtube\"]` will match submissions with the domain `https://youtube.com/c/aChannel`\nEX `[\"youtube.com/c/bChannel\"]` will NOT match submissions with the domain `https://youtube.com/c/aChannel`\n\nIf you wish to aggregate on self-posts for a subreddit use the syntax `self.[subreddit]` EX `self.AskReddit`\n\n**If this Rule is part of a Check for a Submission and you wish to aggregate on the domain of the Submission use the special string `AGG:SELF`**\n\nIf nothing is specified or list is empty (default) aggregate using `aggregateOn`",
                     "items": {
@@ -454,24 +452,52 @@
                     ]
                 },
                 "flairCssClass": {
-                    "description": "A list of (user) flair css class values from the subreddit to match against",
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ],
+                    "description": "A (user) flair css class (or list of) from the subreddit to match against",
                     "examples": [
                         "red"
+                    ]
+                },
+                "flairTemplate": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
                     ],
-                    "items": {
-                        "type": "string"
-                    },
-                    "type": "array"
+                    "description": "A (user) flair template id (or list of) from the subreddit to match against"
                 },
                 "flairText": {
-                    "description": "A list of (user) flair text values from the subreddit to match against",
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ],
+                    "description": "A (user) flair text value (or list of) from the subreddit to match against",
                     "examples": [
                         "Approved"
-                    ],
-                    "items": {
-                        "type": "string"
-                    },
-                    "type": "array"
+                    ]
                 },
                 "isMod": {
                     "description": "Is the author a moderator?",
@@ -535,12 +561,21 @@
             ],
             "properties": {
                 "exclude": {
-                    "description": "Only runs if `include` is not present. Will \"pass\" if any of set of the AuthorCriteria **does not** pass",
+                    "description": "Only runs if `include` is not present. Each AuthorCriteria is comprised of conditions that the Author being checked must \"not\" pass. See excludeCondition for set behavior\n\nEX: `isMod: true, name: Automoderator` => Will pass if the Author IS NOT a mod and IS NOT named Automoderator",
                     "items": {
                         "$ref": "#/definitions/AuthorCriteria"
                     },
                     "type": "array"
                 },
+                "excludeCondition": {
+                    "default": "OR",
+                    "description": "* OR => if ANY exclude condition \"does not\" pass then the exclude test passes\n* AND => if ALL exclude conditions \"do not\" pass then the exclude test passes\n\nDefaults to OR",
+                    "enum": [
+                        "AND",
+                        "OR"
+                    ],
+                    "type": "string"
+                },
                 "include": {
                     "description": "Will \"pass\" if any set of AuthorCriteria passes",
                     "items": {
@@ -1251,6 +1286,11 @@
                     "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s+.*)*$",
                     "type": "string"
                 },
+                "mustMatchCurrent": {
+                    "default": false,
+                    "description": "When `true` the Activity being checked MUST pass the `matchThreshold` before the Rule considers any history\n\nFor use with `activityMatchThreshold`/`totalMatchThreshold` -- useful to conserve API calls",
+                    "type": "boolean"
+                },
                 "name": {
                     "description": "A descriptive name that will be used in logging and be available for templating",
                     "examples": [
@@ -1425,6 +1465,11 @@
                         }
                     ]
                 },
+                "caseSensitive": {
+                    "default": false,
+                    "description": "Should text matching be case sensitive?\n\nDefaults to false",
+                    "type": "boolean"
+                },
                 "exclude": {
                     "description": "If present, activities will be counted only if they are **NOT** found in this list of Subreddits\n\nEach value in the list can be either:\n\n * string (name of subreddit)\n * regular expression to run on the subreddit name\n * `SubredditState`\n\nEX `[\"mealtimevideos\",\"askscience\", \"/onlyfans*\\/i\", {\"over18\": true}]`",
                     "examples": [
@@ -1515,9 +1560,14 @@
                     ],
                     "type": "string"
                 },
+                "matchScore": {
+                    "default": 85,
+                    "description": "The percentage, as a whole number, of a repost title/comment that must match the title/comment being checked in order to consider both a match\n\nNote: Setting to 0 will make every candidate considered a match -- useful if you want to match if the URL has been reposted anywhere\n\nDefaults to `85` (85%)",
+                    "type": "number"
+                },
                 "minWordCount": {
-                    "default": 1,
-                    "description": "For activities that are text-based this is the minimum number of words required for the activity to be considered for a repeat\n\nEX if `minimumWordCount=5` and a comment is `what about you` then it is ignored because `3 is less than 5`\n\n**For self-text submissions** -- title + body text\n\n**For comments* -- body text",
+                    "default": 2,
+                    "description": "The minimum number of words in the activity being checked for which this rule will run on\n\nIf the word count is below the minimum the rule fails\n\nDefaults to 2",
                     "type": "number"
                 },
                 "name": {
@@ -1533,6 +1583,13 @@
                     "description": "The number of repeat submissions that will trigger the rule",
                     "type": "string"
                 },
+                "transformations": {
+                    "description": "A set of search-and-replace operations to perform on text values before performing a match. Transformations are performed in the order they are defined.",
+                    "items": {
+                        "$ref": "#/definitions/SearchAndReplaceRegExp"
+                    },
+                    "type": "array"
+                },
                 "useSubmissionAsReference": {
                     "default": true,
                     "description": "If activity is a Submission and is a link (not self-post) then only look at Submissions that contain this link, otherwise consider all activities.",
@@ -1913,14 +1970,51 @@
                 "filtered": {
                     "type": "boolean"
                 },
+                "flairTemplate": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                 "is_self": {
                     "type": "boolean"
                 },
                 "link_flair_css_class": {
-                    "type": "string"
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
                 },
                 "link_flair_text": {
-                    "type": "string"
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
                 },
                 "locked": {
                     "type": "boolean"

src/Schema/RuleSet.json

@@ -158,8 +158,7 @@
                     "default": "undefined",
                     "description": "This list determines which categories of domains should be aggregated on. All aggregated domains will be tested against `threshold`\n\n* If `media` is included then aggregate author's submission history which reddit recognizes as media (youtube, vimeo, etc.)\n* If `redditMedia` is included then aggregate on author's submissions history which are media hosted on reddit: galleries, videos, and images (i.redd.it / v.redd.it)\n* If `self` is included then aggregate on author's submission history which are self-post (`self.[subreddit]`) or domain is `reddit.com`\n* If `link` is included then aggregate author's submission history which is external links and not recognized as `media` by reddit\n\nIf nothing is specified or list is empty (default) rule will only aggregate on `link` and `media` (ignores reddit-hosted content and self-posts)",
                     "examples": [
-                        [
-                        ]
+                        []
                     ],
                     "items": {
                         "enum": [
@@ -192,8 +191,7 @@
                 },
                 "domains": {
                     "default": [
-                        [
-                        ]
+                        []
                     ],
                     "description": "A list of domains whose Activities will be tested against `threshold`.\n\nThe values are tested as partial strings so you do not need to include full URLs, just the part that matters.\n\nEX `[\"youtube\"]` will match submissions with the domain `https://youtube.com/c/aChannel`\nEX `[\"youtube.com/c/bChannel\"]` will NOT match submissions with the domain `https://youtube.com/c/aChannel`\n\nIf you wish to aggregate on self-posts for a subreddit use the syntax `self.[subreddit]` EX `self.AskReddit`\n\n**If this Rule is part of a Check for a Submission and you wish to aggregate on the domain of the Submission use the special string `AGG:SELF`**\n\nIf nothing is specified or list is empty (default) aggregate using `aggregateOn`",
                     "items": {
@@ -428,24 +426,52 @@
                     ]
                 },
                 "flairCssClass": {
-                    "description": "A list of (user) flair css class values from the subreddit to match against",
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ],
+                    "description": "A (user) flair css class (or list of) from the subreddit to match against",
                     "examples": [
                         "red"
+                    ]
+                },
+                "flairTemplate": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
                     ],
-                    "items": {
-                        "type": "string"
-                    },
-                    "type": "array"
+                    "description": "A (user) flair template id (or list of) from the subreddit to match against"
                 },
                 "flairText": {
-                    "description": "A list of (user) flair text values from the subreddit to match against",
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ],
+                    "description": "A (user) flair text value (or list of) from the subreddit to match against",
                     "examples": [
                         "Approved"
-                    ],
-                    "items": {
-                        "type": "string"
-                    },
-                    "type": "array"
+                    ]
                 },
                 "isMod": {
                     "description": "Is the author a moderator?",
@@ -509,12 +535,21 @@
             ],
             "properties": {
                 "exclude": {
-                    "description": "Only runs if `include` is not present. Will \"pass\" if any of set of the AuthorCriteria **does not** pass",
+                    "description": "Only runs if `include` is not present. Each AuthorCriteria is comprised of conditions that the Author being checked must \"not\" pass. See excludeCondition for set behavior\n\nEX: `isMod: true, name: Automoderator` => Will pass if the Author IS NOT a mod and IS NOT named Automoderator",
                     "items": {
                         "$ref": "#/definitions/AuthorCriteria"
                     },
                     "type": "array"
                 },
+                "excludeCondition": {
+                    "default": "OR",
+                    "description": "* OR => if ANY exclude condition \"does not\" pass then the exclude test passes\n* AND => if ALL exclude conditions \"do not\" pass then the exclude test passes\n\nDefaults to OR",
+                    "enum": [
+                        "AND",
+                        "OR"
+                    ],
+                    "type": "string"
+                },
                 "include": {
                     "description": "Will \"pass\" if any set of AuthorCriteria passes",
                     "items": {
@@ -1225,6 +1260,11 @@
                     "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s+.*)*$",
                     "type": "string"
                 },
+                "mustMatchCurrent": {
+                    "default": false,
+                    "description": "When `true` the Activity being checked MUST pass the `matchThreshold` before the Rule considers any history\n\nFor use with `activityMatchThreshold`/`totalMatchThreshold` -- useful to conserve API calls",
+                    "type": "boolean"
+                },
                 "name": {
                     "description": "A descriptive name that will be used in logging and be available for templating",
                     "examples": [
@@ -1399,6 +1439,11 @@
                         }
                     ]
                 },
+                "caseSensitive": {
+                    "default": false,
+                    "description": "Should text matching be case sensitive?\n\nDefaults to false",
+                    "type": "boolean"
+                },
                 "exclude": {
                     "description": "If present, activities will be counted only if they are **NOT** found in this list of Subreddits\n\nEach value in the list can be either:\n\n * string (name of subreddit)\n * regular expression to run on the subreddit name\n * `SubredditState`\n\nEX `[\"mealtimevideos\",\"askscience\", \"/onlyfans*\\/i\", {\"over18\": true}]`",
                     "examples": [
@@ -1489,9 +1534,14 @@
                     ],
                     "type": "string"
                 },
+                "matchScore": {
+                    "default": 85,
+                    "description": "The percentage, as a whole number, of a repost title/comment that must match the title/comment being checked in order to consider both a match\n\nNote: Setting to 0 will make every candidate considered a match -- useful if you want to match if the URL has been reposted anywhere\n\nDefaults to `85` (85%)",
+                    "type": "number"
+                },
                 "minWordCount": {
-                    "default": 1,
-                    "description": "For activities that are text-based this is the minimum number of words required for the activity to be considered for a repeat\n\nEX if `minimumWordCount=5` and a comment is `what about you` then it is ignored because `3 is less than 5`\n\n**For self-text submissions** -- title + body text\n\n**For comments* -- body text",
+                    "default": 2,
+                    "description": "The minimum number of words in the activity being checked for which this rule will run on\n\nIf the word count is below the minimum the rule fails\n\nDefaults to 2",
                     "type": "number"
                 },
                 "name": {
@@ -1507,6 +1557,13 @@
                     "description": "The number of repeat submissions that will trigger the rule",
                     "type": "string"
                 },
+                "transformations": {
+                    "description": "A set of search-and-replace operations to perform on text values before performing a match. Transformations are performed in the order they are defined.",
+                    "items": {
+                        "$ref": "#/definitions/SearchAndReplaceRegExp"
+                    },
+                    "type": "array"
+                },
                 "useSubmissionAsReference": {
                     "default": true,
                     "description": "If activity is a Submission and is a link (not self-post) then only look at Submissions that contain this link, otherwise consider all activities.",
@@ -1887,14 +1944,51 @@
                 "filtered": {
                     "type": "boolean"
                 },
+                "flairTemplate": {
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
+                },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                 "is_self": {
                     "type": "boolean"
                 },
                 "link_flair_css_class": {
-                    "type": "string"
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
                 },
                 "link_flair_text": {
-                    "type": "string"
+                    "anyOf": [
+                        {
+                            "items": {
+                                "type": "string"
+                            },
+                            "type": "array"
+                        },
+                        {
+                            "type": "string"
+                        }
+                    ]
                 },
                 "locked": {
                     "type": "boolean"

src/Subreddit/Manager.ts

@@ -18,17 +18,15 @@ import {
     totalFromMapStats,
     triggeredIndicator,
 } from "../util";
-import {Poll} from "snoostorm";
-import pEvent from "p-event";
 import {RuleResult} from "../Rule";
 import {ConfigBuilder, buildPollingOptions} from "../ConfigBuilder";
 import {
     ActionedEvent,
     ActionResult,
     DEFAULT_POLLING_INTERVAL,
-    DEFAULT_POLLING_LIMIT, Invokee,
+    DEFAULT_POLLING_LIMIT, FilterCriteriaDefaults, Invokee, LogInfo,
     ManagerOptions, ManagerStateChangeOption, ManagerStats, PAUSED,
-    PollingOptionsStrong, ResourceStats, RUNNING, RunState, STOPPED, SYSTEM, USER
+    PollingOptionsStrong, PollOn, RUNNING, RunState, STOPPED, SYSTEM, USER
 } from "../Common/interfaces";
 import Submission from "snoowrap/dist/objects/Submission";
 import {activityIsRemoved, itemContentPeek} from "../Utils/SnoowrapUtils";
@@ -48,10 +46,10 @@ import {queue, QueueObject} from 'async';
 import {JSONConfig} from "../JsonConfig";
 import {CheckStructuredJson} from "../Check";
 import NotificationManager from "../Notification/NotificationManager";
-import action from "../Web/Server/routes/authenticated/user/action";
 import {createHistoricalDefaults, historicalDefaults} from "../Common/defaults";
 import {ExtendedSnoowrap} from "../Utils/SnoowrapClients";
-import {isRateLimitError, isStatusError} from "../Utils/Errors";
+import {CMError, isRateLimitError, isStatusError} from "../Utils/Errors";
+import {ErrorWithCause} from "pony-cause";
 
 export interface RunningState {
     state: RunState,
@@ -73,7 +71,7 @@ export interface CheckTask {
 }
 
 export interface RuntimeManagerOptions extends ManagerOptions {
-    sharedModqueue?: boolean;
+    sharedStreams?: PollOn[];
     wikiLocation?: string;
     botName: string;
     maxWorkers: number;
@@ -89,6 +87,7 @@ export class Manager extends EventEmitter {
     subreddit: Subreddit;
     client: ExtendedSnoowrap;
     logger: Logger;
+    logs: LogInfo[] = [];
     botName: string;
     pollOptions: PollingOptionsStrong[] = [];
     submissionChecks!: SubmissionCheck[];
@@ -98,13 +97,14 @@ export class Manager extends EventEmitter {
     lastWikiRevision?: DayjsObj
     lastWikiCheck: DayjsObj = dayjs();
     wikiFormat: ('yaml' | 'json') = 'yaml';
+    filterCriteriaDefaults?: FilterCriteriaDefaults
     //wikiUpdateRunning: boolean = false;
 
-    streamListedOnce: string[] = [];
-    streams: SPoll<Snoowrap.Submission | Snoowrap.Comment>[] = [];
-    modStreamCallbacks: Map<string, any> = new Map();
+    streams: Map<string, SPoll<Snoowrap.Submission | Snoowrap.Comment>> = new Map();
+    sharedStreamCallbacks: Map<string, any> = new Map();
+    pollingRetryHandler: Function;
     dryRun?: boolean;
-    sharedModqueue: boolean;
+    sharedStreams: PollOn[];
     cacheManager: BotResourcesManager;
     globalDryRun?: boolean;
     queue: QueueObject<CheckTask>;
@@ -198,7 +198,7 @@ export class Manager extends EventEmitter {
     constructor(sub: Subreddit, client: ExtendedSnoowrap, logger: Logger, cacheManager: BotResourcesManager, opts: RuntimeManagerOptions = {botName: 'ContextMod', maxWorkers: 1}) {
         super();
 
-        const {dryRun, sharedModqueue = false, wikiLocation = 'botconfig/contextbot', botName, maxWorkers} = opts;
+        const {dryRun, sharedStreams = [], wikiLocation = 'botconfig/contextbot', botName, maxWorkers, filterCriteriaDefaults} = opts;
         this.displayLabel = opts.nickname || `${sub.display_name_prefixed}`;
         const getLabels = this.getCurrentLabels;
         const getDisplay = this.getDisplay;
@@ -212,9 +212,16 @@ export class Manager extends EventEmitter {
                 return getDisplay()
             }
         }, mergeArr);
+        this.logger.stream().on('log', (log: LogInfo) => {
+            if(log.subreddit !== undefined && log.subreddit === this.getDisplay()) {
+                this.logs = [log, ...this.logs].slice(0, 301);
+            }
+        });
         this.globalDryRun = dryRun;
         this.wikiLocation = wikiLocation;
-        this.sharedModqueue = sharedModqueue;
+        this.filterCriteriaDefaults = filterCriteriaDefaults;
+        this.sharedStreams = sharedStreams;
+        this.pollingRetryHandler = createRetryHandler({maxRequestRetry: 3, maxOtherRetry: 2}, this.logger);
         this.subreddit = sub;
         this.client = client;
         this.botName = botName;
@@ -269,15 +276,21 @@ export class Manager extends EventEmitter {
         })(this), 10000);
     }
 
-    protected async getModPermissions(): Promise<string[]> {
+    public async getModPermissions(): Promise<string[]> {
         if(this.modPermissions !== undefined) {
             return this.modPermissions as string[];
         }
         this.logger.debug('Retrieving mod permissions for bot');
-        const userInfo = parseRedditEntity(this.botName, 'user');
-        const mods = this.subreddit.getModerators({name: userInfo.name});
-        // @ts-ignore
-        this.modPermissions = mods[0].mod_permissions;
+        try {
+            const userInfo = parseRedditEntity(this.botName, 'user');
+            const mods = this.subreddit.getModerators({name: userInfo.name});
+            // @ts-ignore
+            this.modPermissions = mods[0].mod_permissions;
+        } catch (e) {
+            const err = new ErrorWithCause('Unable to retrieve moderator permissions', {cause: e});
+            this.logger.error(err);
+            return [];
+        }
         return this.modPermissions as string[];
     }
 
@@ -359,7 +372,7 @@ export class Manager extends EventEmitter {
         return q;
     }
 
-    protected async parseConfigurationFromObject(configObj: object) {
+    protected async parseConfigurationFromObject(configObj: object, suppressChangeEvent: boolean = false) {
         try {
             const configBuilder = new ConfigBuilder({logger: this.logger});
             const validJson = configBuilder.validateJson(configObj);
@@ -419,7 +432,7 @@ export class Manager extends EventEmitter {
 
             const commentChecks: Array<CommentCheck> = [];
             const subChecks: Array<SubmissionCheck> = [];
-            const structuredChecks = configBuilder.parseToStructured(validJson);
+            const structuredChecks = configBuilder.parseToStructured(validJson, this.filterCriteriaDefaults);
 
             // TODO check that bot has permissions for subreddit for all specified actions
             // can find permissions in this.subreddit.mod_permissions
@@ -449,6 +462,19 @@ export class Manager extends EventEmitter {
                 this.logger.info(checkSummary);
             }
             this.validConfigLoaded = true;
+            if(this.eventsState.state === RUNNING) {
+                // need to update polling, potentially
+                await this.buildPolling();
+                for(const stream of this.streams.values()) {
+                    if(!stream.running) {
+                        this.logger.debug(`Starting Polling for ${stream.name.toUpperCase()} ${stream.frequency / 1000}s interval`);
+                        stream.startInterval();
+                    }
+                }
+            }
+            if(!suppressChangeEvent) {
+                this.emit('configChange');
+            }
         } catch (err: any) {
             this.validConfigLoaded = false;
             throw err;
@@ -456,7 +482,7 @@ export class Manager extends EventEmitter {
     }
 
     async parseConfiguration(causedBy: Invokee = 'system', force: boolean = false, options?: ManagerStateChangeOption) {
-        const {reason, suppressNotification = false} = options || {};
+        const {reason, suppressNotification = false, suppressChangeEvent = false} = options || {};
         //this.wikiUpdateRunning = true;
         this.lastWikiCheck = dayjs();
 
@@ -471,21 +497,21 @@ export class Manager extends EventEmitter {
                     if(isStatusError(err) && err.statusCode === 404) {
                         // see if we can create the page
                         if (!this.client.scope.includes('wikiedit')) {
-                            throw new Error(`Page does not exist and could not be created because Bot does not have oauth permission 'wikiedit'`);
+                            throw new ErrorWithCause(`Page does not exist and could not be created because Bot does not have oauth permission 'wikiedit'`, {cause: err});
                         }
                         const modPermissions = await this.getModPermissions();
                         if (!modPermissions.includes('all') && !modPermissions.includes('wiki')) {
-                            throw new Error(`Page does not exist and could not be created because Bot not have mod permissions for creating wiki pages. Must have 'all' or 'wiki'`);
+                            throw new ErrorWithCause(`Page does not exist and could not be created because Bot not have mod permissions for creating wiki pages. Must have 'all' or 'wiki'`, {cause: err});
                         }
                         if(!this.client.scope.includes('modwiki')) {
-                            throw new Error(`Bot COULD create wiki config page but WILL NOT because it does not have the oauth permissions 'modwiki' which is required to set page visibility and editing permissions. Safety first!`);
+                            throw new ErrorWithCause(`Bot COULD create wiki config page but WILL NOT because it does not have the oauth permissions 'modwiki' which is required to set page visibility and editing permissions. Safety first!`, {cause: err});
                         }
                         // @ts-ignore
                         wiki = await this.subreddit.getWikiPage(this.wikiLocation).edit({
                             text: '',
                             reason: 'Empty configuration created for ContextMod'
                         });
-                        this.logger.info(`Wiki page at ${this.wikiLocation} did not exist, but bot created it!`);
+                        this.logger.info(`Wiki page at ${this.wikiLocation} did not exist so bot created it!`);
 
                         // 0 = use subreddit wiki permissions
                         // 1 = only approved wiki contributors
@@ -529,11 +555,10 @@ export class Manager extends EventEmitter {
             } catch (err: any) {
                 let hint = '';
                 if(isStatusError(err) && err.statusCode === 403) {
-                    hint = `\r\nHINT: Either the page is restricted to mods only and the bot's reddit account does have the mod permission 'all' or 'wiki' OR the bot does not have the 'wikiread' oauth permission`;
+                    hint = ` -- HINT: Either the page is restricted to mods only and the bot's reddit account does have the mod permission 'all' or 'wiki' OR the bot does not have the 'wikiread' oauth permission`;
                 }
-                const msg = `Could not read wiki configuration. Please ensure the page https://reddit.com${this.subreddit.url}wiki/${this.wikiLocation} exists and is readable${hint} -- error: ${err.message}`;
-                this.logger.error(msg);
-                throw new ConfigParseError(msg);
+                const msg = `Could not read wiki configuration. Please ensure the page https://reddit.com${this.subreddit.url}wiki/${this.wikiLocation} exists and is readable${hint}`;
+                throw new ErrorWithCause(msg, {cause: err});
             }
 
             if (sourceData.replace('\r\n', '').trim() === '') {
@@ -541,14 +566,8 @@ export class Manager extends EventEmitter {
                 throw new ConfigParseError('Wiki page contents was empty');
             }
 
-            const [configObj, jsonErr, yamlErr] = parseFromJsonOrYamlToObject(sourceData);
-            if (jsonErr === undefined) {
-                this.wikiFormat = 'json';
-            } else if (yamlErr === undefined) {
-                this.wikiFormat = 'yaml';
-            } else {
-                this.wikiFormat = likelyJson5(sourceData) ? 'json' : 'yaml';
-            }
+            const [format, configObj, jsonErr, yamlErr] = parseFromJsonOrYamlToObject(sourceData);
+            this.wikiFormat = format;
 
             if (configObj === undefined) {
                 this.logger.error(`Could not parse wiki page contents as JSON or YAML. Looks like it should be ${this.wikiFormat}?`);
@@ -564,7 +583,7 @@ export class Manager extends EventEmitter {
                 throw new ConfigParseError('Could not parse wiki page contents as JSON or YAML')
             }
 
-            await this.parseConfigurationFromObject(configObj);
+            await this.parseConfigurationFromObject(configObj.toJS(), suppressChangeEvent);
             this.logger.info('Checks updated');
 
             if(!suppressNotification) {
@@ -573,8 +592,12 @@ export class Manager extends EventEmitter {
 
             return true;
         } catch (err: any) {
+            const error = new ErrorWithCause('Failed to parse subreddit configuration', {cause: err});
+            // @ts-ignore
+           //error.logged = true;
+            this.logger.error(error);
             this.validConfigLoaded = false;
-            throw err;
+            throw error;
         }
     }
 
@@ -724,7 +747,7 @@ export class Manager extends EventEmitter {
                     actionsRun = runActions.length;
 
                     if(check.notifyOnTrigger) {
-                        const ar = runActions.map(x => x.name).join(', ');
+                        const ar = runActions.filter(x => x.success).map(x => x.name).join(', ');
                         this.notificationManager.handle('eventActioned', 'Check Triggered', `Check "${check.name}" was triggered on Event: \n\n ${ePeek} \n\n with the following actions run: ${ar}`);
                     }
                     break;
@@ -768,132 +791,192 @@ export class Manager extends EventEmitter {
         }
     }
 
-    async buildPolling() {
-        // give current handle() time to stop
-        //await sleep(1000);
+    isPollingShared(streamName: string): boolean {
+        const pollOption = this.pollOptions.find(x => x.pollOn === streamName);
+        return pollOption !== undefined && pollOption.limit === DEFAULT_POLLING_LIMIT && pollOption.interval === DEFAULT_POLLING_INTERVAL && this.sharedStreams.includes(streamName as PollOn);
+    }
 
-        const retryHandler = createRetryHandler({maxRequestRetry: 3, maxOtherRetry: 1}, this.logger);
+    async buildPolling() {
+
+        const sources: PollOn[] = ['unmoderated', 'modqueue', 'newComm', 'newSub'];
 
         const subName = this.subreddit.display_name;
 
-        for (const pollOpt of this.pollOptions) {
-            const {
-                pollOn,
-                limit,
-                interval,
-                delayUntil,
-                clearProcessed,
-            } = pollOpt;
-            let stream: SPoll<Snoowrap.Submission | Snoowrap.Comment>;
-            let modStreamType: string | undefined;
+        for (const source of sources) {
 
-            switch (pollOn) {
-                case 'unmoderated':
-                    if (limit === DEFAULT_POLLING_LIMIT && interval === DEFAULT_POLLING_INTERVAL && this.sharedModqueue) {
-                        modStreamType = 'unmoderated';
-                        // use default mod stream from resources
-                        stream = this.cacheManager.modStreams.get('unmoderated') as SPoll<Snoowrap.Submission | Snoowrap.Comment>;
-                    } else {
-                        stream = new UnmoderatedStream(this.client, {
-                            subreddit: this.subreddit.display_name,
-                            limit: limit,
-                            pollTime: interval * 1000,
-                            clearProcessed,
-                        });
-                    }
-                    break;
-                case 'modqueue':
-                    if (limit === DEFAULT_POLLING_LIMIT && interval === DEFAULT_POLLING_INTERVAL) {
-                        modStreamType = 'modqueue';
-                        // use default mod stream from resources
-                        stream = this.cacheManager.modStreams.get('modqueue') as SPoll<Snoowrap.Submission | Snoowrap.Comment>;
-                    } else {
-                        stream = new ModQueueStream(this.client, {
-                            subreddit: this.subreddit.display_name,
-                            limit: limit,
-                            pollTime: interval * 1000,
-                            clearProcessed
-                        });
-                    }
-                    break;
-                case 'newSub':
-                    stream = new SubmissionStream(this.client, {
-                        subreddit: this.subreddit.display_name,
-                        limit: limit,
-                        pollTime: interval * 1000,
-                        clearProcessed
-                    });
-                    break;
-                case 'newComm':
-                    stream = new CommentStream(this.client, {
-                        subreddit: this.subreddit.display_name,
-                        limit: limit,
-                        pollTime: interval * 1000,
-                        clearProcessed
-                    });
-                    break;
+            if (!sources.includes(source)) {
+                this.logger.error(`'${source}' is not a valid polling source. Valid sources: unmoderated | modqueue | newComm | newSub`);
+                continue;
             }
 
-            stream.once('listing', async (listing) => {
-                if (!this.streamListedOnce.includes(pollOn)) {
-                    // warning if poll event could potentially miss activities
-                    if (this.commentChecks.length === 0 && ['unmoderated', 'modqueue', 'newComm'].some(x => x === pollOn)) {
-                        this.logger.warn(`Polling '${pollOn}' may return Comments but no comments checks were configured.`);
-                    }
-                    if (this.submissionChecks.length === 0 && ['unmoderated', 'modqueue', 'newSub'].some(x => x === pollOn)) {
-                        this.logger.warn(`Polling '${pollOn}' may return Submissions but no submission checks were configured.`);
-                    }
-                    this.streamListedOnce.push(pollOn);
+            const pollOpt = this.pollOptions.find(x => x.pollOn.toLowerCase() === source.toLowerCase());
+            if (pollOpt === undefined) {
+                if(this.sharedStreamCallbacks.has(source)) {
+                    this.logger.debug(`Removing listener for shared polling on ${source.toUpperCase()} because it no longer exists in config`);
+                    this.sharedStreamCallbacks.delete(source);
                 }
-            });
-
-            const onItem = async (item: Comment | Submission) => {
-                if (!this.streamListedOnce.includes(pollOn)) {
-                    return;
+                const existingStream = this.streams.get(source);
+                if (existingStream !== undefined) {
+                    this.logger.debug(`Stopping polling on ${source.toUpperCase()} because it no longer exists in config`);
+                    existingStream.end();
+                    this.streams.delete(source);
                 }
-                if (item.subreddit.display_name !== subName || this.eventsState.state !== RUNNING) {
-                    return;
-                }
-                let checkType: 'Submission' | 'Comment' | undefined;
-                if (item instanceof Submission) {
-                    if (this.submissionChecks.length > 0) {
-                        checkType = 'Submission';
-                    }
-                } else if (this.commentChecks.length > 0) {
-                    checkType = 'Comment';
-                }
-                if (checkType !== undefined) {
-                    this.firehose.push({checkType, activity: item, options: {delayUntil}})
-                }
-            };
-
-            if (modStreamType !== undefined) {
-                this.modStreamCallbacks.set(pollOn, onItem);
             } else {
-                stream.on('item', onItem);
-                // @ts-ignore
-                stream.on('error', async (err: any) => {
 
-                    this.emit('error', err);
+                const {
+                    limit,
+                    interval,
+                    delayUntil,
+                } = pollOpt;
+                let stream: SPoll<Snoowrap.Submission | Snoowrap.Comment>;
+                let modStreamType: string | undefined;
 
-                    if(isRateLimitError(err)) {
-                        this.logger.error('Encountered rate limit while polling! Bot is all out of requests :( Stopping subreddit queue and polling.');
-                        await this.stop();
+                switch (source) {
+                    case 'unmoderated':
+                        if (limit === DEFAULT_POLLING_LIMIT && interval === DEFAULT_POLLING_INTERVAL && this.sharedStreams.includes(source)) {
+                            modStreamType = 'unmoderated';
+                            // use default mod stream from resources
+                            stream = this.cacheManager.modStreams.get('unmoderated') as SPoll<Snoowrap.Submission | Snoowrap.Comment>;
+                        } else {
+                            stream = new UnmoderatedStream(this.client, {
+                                subreddit: this.subreddit.display_name,
+                                limit: limit,
+                                pollTime: interval * 1000,
+                                logger: this.logger,
+                            });
+                        }
+                        break;
+                    case 'modqueue':
+                        if (limit === DEFAULT_POLLING_LIMIT && interval === DEFAULT_POLLING_INTERVAL && this.sharedStreams.includes(source)) {
+                            modStreamType = 'modqueue';
+                            // use default mod stream from resources
+                            stream = this.cacheManager.modStreams.get('modqueue') as SPoll<Snoowrap.Submission | Snoowrap.Comment>;
+                        } else {
+                            stream = new ModQueueStream(this.client, {
+                                subreddit: this.subreddit.display_name,
+                                limit: limit,
+                                pollTime: interval * 1000,
+                                logger: this.logger,
+                            });
+                        }
+                        break;
+                    case 'newSub':
+                        if (limit === DEFAULT_POLLING_LIMIT && interval === DEFAULT_POLLING_INTERVAL && this.sharedStreams.includes(source)) {
+                            modStreamType = 'newSub';
+                            // use default mod stream from resources
+                            stream = this.cacheManager.modStreams.get('newSub') as SPoll<Snoowrap.Submission | Snoowrap.Comment>;
+                        } else {
+                            stream = new SubmissionStream(this.client, {
+                                subreddit: this.subreddit.display_name,
+                                limit: limit,
+                                pollTime: interval * 1000,
+                                logger: this.logger,
+                            });
+                        }
+                        break;
+                    case 'newComm':
+                        if (limit === DEFAULT_POLLING_LIMIT && interval === DEFAULT_POLLING_INTERVAL && this.sharedStreams.includes(source)) {
+                            modStreamType = 'newComm';
+                            // use default mod stream from resources
+                            stream = this.cacheManager.modStreams.get('newComm') as SPoll<Snoowrap.Submission | Snoowrap.Comment>;
+                        } else {
+                            stream = new CommentStream(this.client, {
+                                subreddit: this.subreddit.display_name,
+                                limit: limit,
+                                pollTime: interval * 1000,
+                                logger: this.logger,
+                            });
+                        }
+                        break;
+                }
+
+                if (stream === undefined) {
+                    this.logger.error(`Should have found polling source for '${source}' but it did not exist for some reason!`);
+                    continue;
+                }
+
+                const onItem = async (item: Comment | Submission) => {
+                    if (item.subreddit.display_name !== subName || this.eventsState.state !== RUNNING) {
+                        return;
                     }
-                    this.logger.error('Polling error occurred', err);
-                    const shouldRetry = await retryHandler(err);
-                    if (shouldRetry) {
-                        stream.startInterval();
+                    let checkType: 'Submission' | 'Comment' | undefined;
+                    if (item instanceof Submission) {
+                        if (this.submissionChecks.length > 0) {
+                            checkType = 'Submission';
+                        }
+                    } else if (this.commentChecks.length > 0) {
+                        checkType = 'Comment';
+                    }
+                    if (checkType !== undefined) {
+                        this.firehose.push({checkType, activity: item, options: {delayUntil}})
+                    }
+                };
+
+                if (modStreamType !== undefined) {
+                    let removedOwn = false;
+                    const existingStream = this.streams.get(source);
+                    if(existingStream !== undefined) {
+                        existingStream.end();
+                        this.streams.delete(source);
+                        removedOwn = true;
+                    }
+                    if(!this.sharedStreamCallbacks.has(source)) {
+                        stream.once('listing', this.noChecksWarning(source));
+                        this.logger.debug(`${removedOwn ? 'Stopped own polling and replace with ' : 'Set '}listener on shared polling ${source}`);
+                    }
+                    this.sharedStreamCallbacks.set(source, onItem);
+                } else {
+                    let ownPollingMsgParts: string[] = [];
+                    let removedShared = false;
+                    if(this.sharedStreamCallbacks.has(source)) {
+                        removedShared = true;
+                        this.sharedStreamCallbacks.delete(source);
+                        ownPollingMsgParts.push('removed shared polling listener');
+                    }
+
+                    const existingStream = this.streams.get(source);
+                    let processed;
+                    if (existingStream !== undefined) {
+                        ownPollingMsgParts.push('replaced existing');
+                        processed = existingStream.processed;
+                        existingStream.end();
                     } else {
-                        this.logger.warn('Stopping subreddit processing/polling due to too many errors');
-                        await this.stop();
+                        ownPollingMsgParts.push('create new');
+                        stream.once('listing', this.noChecksWarning(source));
                     }
-                });
-                this.streams.push(stream);
+
+                    this.logger.debug(`Polling ${source.toUpperCase()} => ${ownPollingMsgParts.join('and')} dedicated stream`);
+
+                    stream.on('item', onItem);
+                    // @ts-ignore
+                    stream.on('error', async (err: any) => {
+
+                        this.emit('error', err);
+
+                        const shouldRetry = await this.pollingRetryHandler(err);
+                        if (shouldRetry) {
+                            stream.startInterval(false, 'Within retry limits');
+                        } else {
+                            this.logger.warn('Stopping subreddit processing/polling due to too many errors');
+                            await this.stop();
+                        }
+                    });
+
+                    this.streams.set(source, stream);
+                }
             }
         }
     }
 
+    noChecksWarning = (source: PollOn) => (listing: any) => {
+        if (this.commentChecks.length === 0 && ['modqueue', 'newComm'].some(x => x === source)) {
+            this.logger.warn(`Polling '${source.toUpperCase()}' may return Comments but no comments checks were configured.`);
+        }
+        if (this.submissionChecks.length === 0 && ['unmoderated', 'modqueue', 'newSub'].some(x => x === source)) {
+            this.logger.warn(`Polling '${source.toUpperCase()}' may return Submissions but no submission checks were configured.`);
+        }
+    }
+
     startQueue(causedBy: Invokee = 'system', options?: ManagerStateChangeOption) {
         const {reason, suppressNotification = false} = options || {};
         if(this.queueState.state === RUNNING) {
@@ -1019,7 +1102,10 @@ export class Manager extends EventEmitter {
                 this.logger.warn('No submission or comment checks found!');
             }
 
-            for (const s of this.streams) {
+            if (this.streams.size > 0) {
+                this.logger.debug(`Starting own streams => ${[...this.streams.values()].map(x => `${x.name.toUpperCase()} ${x.frequency / 1000}s interval`).join(' | ')}`)
+            }
+            for (const s of this.streams.values()) {
                 s.startInterval();
             }
             this.startedAt = dayjs();
@@ -1044,7 +1130,7 @@ export class Manager extends EventEmitter {
                 state: PAUSED,
                 causedBy
             };
-            for(const s of this.streams) {
+            for(const s of this.streams.values()) {
                 s.end();
             }
             if(causedBy === USER) {
@@ -1061,15 +1147,10 @@ export class Manager extends EventEmitter {
     stopEvents(causedBy: Invokee = 'system', options?: ManagerStateChangeOption) {
         const {reason, suppressNotification = false} = options || {};
         if(this.eventsState.state !== STOPPED) {
-            for (const s of this.streams) {
+            for (const s of this.streams.values()) {
                 s.end();
             }
-            this.streams = [];
-            // for (const [k, v] of this.modStreamCallbacks) {
-            //     const stream = this.cacheManager.modStreams.get(k) as Poll<Snoowrap.Submission | Snoowrap.Comment>;
-            //     stream.removeListener('item', v);
-            // }
-            this.modStreamCallbacks = new Map();
+            this.streams = new Map();
             this.startedAt = undefined;
             this.logger.info(`Events STOPPED by ${causedBy}`);
             this.eventsState = {

src/Subreddit/Streams.ts

@@ -1,52 +1,69 @@
 import {Poll, SnooStormOptions} from "snoostorm"
-import Snoowrap from "snoowrap";
+import Snoowrap, {Listing} from "snoowrap";
 import {EventEmitter} from "events";
 import {PollConfiguration} from "snoostorm/out/util/Poll";
-import {ClearProcessedOptions, DEFAULT_POLLING_INTERVAL} from "../Common/interfaces";
-import dayjs, {Dayjs} from "dayjs";
-import { Duration } from "dayjs/plugin/duration";
-import {parseDuration, random} from "../util";
+import {DEFAULT_POLLING_INTERVAL} from "../Common/interfaces";
+import {mergeArr, parseDuration, random} from "../util";
+import { Logger } from "winston";
+import {ErrorWithCause} from "pony-cause";
 
 type Awaitable<T> = Promise<T> | T;
 
-interface RCBPollingOptions extends SnooStormOptions {
+interface RCBPollingOptions<T> extends SnooStormOptions {
     subreddit: string,
-    clearProcessed?: ClearProcessedOptions
+    enforceContinuity?: boolean
+    logger: Logger
+    name?: string,
+    processed?: Set<T[keyof T]>
+    label?: string
 }
 
-interface RCBPollConfiguration<T> extends PollConfiguration<T> {
-    clearProcessed?: ClearProcessedOptions
+interface RCBPollConfiguration<T> extends PollConfiguration<T>,RCBPollingOptions<T> {
+    get: () => Promise<Listing<T>>
 }
 
 export class SPoll<T extends object> extends Poll<T> {
     identifier: keyof T;
-    getter: () => Awaitable<T[]>;
+    getter: () => Promise<Listing<T>>;
     frequency;
     running: boolean = false;
-    clearProcessedDuration?: Duration;
-    clearProcessedSize?: number;
-    clearProcessedAfter?: Dayjs;
-    retainProcessed: number = 0;
+    // intention of newStart is to make polling behavior such that only "new" items AFTER polling has started get emitted
+    // -- that is, we don't want to emit the items we immediately fetch on a fresh poll start since they existed "before" polling started
+    newStart: boolean = true;
+    enforceContinuity: boolean;
     randInterval?: { clear: () => void };
+    name: string = 'Reddit Stream';
+    logger: Logger;
+    subreddit: string;
 
     constructor(options: RCBPollConfiguration<T>) {
         super(options);
-        this.identifier = options.identifier;
-        this.getter = options.get;
-        this.frequency = options.frequency;
         const {
-            after,
-            size,
-            retain = 0,
-        } = options.clearProcessed || {};
-        if(after !== undefined) {
-            this.clearProcessedDuration = parseDuration(after);
-        }
-        this.clearProcessedSize = size;
-        this.retainProcessed = retain;
-        if (this.clearProcessedDuration !== undefined) {
-            this.clearProcessedAfter = dayjs().add(this.clearProcessedDuration.asSeconds(), 's');
+            identifier,
+            get,
+            frequency,
+            enforceContinuity = false,
+            logger,
+            name,
+            subreddit,
+            label = 'Polling',
+            processed
+        } = options;
+        this.subreddit = subreddit;
+        this.name = name !== undefined ? name : this.name;
+        this.logger = logger.child({labels: [label, this.name]}, mergeArr)
+        this.identifier = identifier;
+        this.getter = get;
+        this.frequency = frequency;
+        this.enforceContinuity = enforceContinuity;
+
+        // if we pass in processed on init the intention is to "continue" from where the previous stream left off
+        // WITHOUT new start behavior
+        if (processed !== undefined) {
+            this.processed = processed;
+            this.newStart = false;
         }
+
         clearInterval(this.interval);
     }
 
@@ -54,37 +71,81 @@ export class SPoll<T extends object> extends Poll<T> {
         this.interval = setTimeout((function (self) {
             return async () => {
                 try {
-                    const batch = await self.getter();
+                    self.logger.debug('Polling...');
+                    let batch = await self.getter();
                     const newItems: T[] = [];
-                    for (const item of batch) {
-                        const id = item[self.identifier];
-                        if (self.processed.has(id)) continue;
+                    let anyAlreadySeen = false;
+                    let page = 1;
+                    // initial iteration should always run
+                    // but only continue iterating if stream enforces continuity and we've only seen new items so far
+                    while(page === 1 || (self.enforceContinuity && !self.newStart && !anyAlreadySeen)) {
+                        if(page !== 1) {
+                            self.logger.debug(`Did not find any already seen activities and continuity is enforced. This probably means there were more new items than 1 api call can return. Fetching next page (${page})...`);
+                            // @ts-ignore
+                            batch = await batch.fetchMore({amount: 100});
+                        }
+                        if(batch.length === 0 || batch.isFinished) {
+                            // if nothing is returned we don't want to end up in an endless loop!
+                            anyAlreadySeen = true;
+                        }
+                        for (const item of batch) {
+                            const id = item[self.identifier];
+                            if (self.processed.has(id)) {
+                                anyAlreadySeen = true;
+                                continue;
+                            }
 
-                        // Emit for new items and add it to the list
-                        newItems.push(item);
-                        self.processed.add(id);
-                        self.emit("item", item);
+                            // Emit for new items and add it to the list
+                            newItems.push(item);
+                            self.processed.add(id);
+                            // but don't emit on new start since we are "buffering" already existing activities
+                            if(!self.newStart) {
+                                self.emit("item", item);
+                            }
+                        }
+                        page++;
                     }
-
-                    // Emit the new listing of all new items
-                    self.emit("listing", newItems);
-
+                    const newItemMsg = `Found ${newItems.length} new items out of ${batch.length} returned`;
+                    if(self.newStart) {
+                        self.logger.debug(`${newItemMsg} but will ignore all on first start.`);
+                        self.emit("listing", []);
+                    } else {
+                        self.logger.debug(newItemMsg);
+                        // Emit the new listing of all new items
+                        self.emit("listing", newItems);
+                    }
+                    // no longer new start on n+1 interval
+                    self.newStart = false;
                     // if everything succeeded then create a new timeout
                     self.createInterval();
                 } catch (err: any) {
+                    self.running = false;
+                    self.logger.error(new ErrorWithCause('Polling Interval stopped due to error encountered', {cause: err}));
                     self.emit('error', err);
                 }
             }
         })(this), random(this.frequency - 1, this.frequency + 1));
     }
 
-    startInterval = () => {
+    // allow controlling newStart state
+    startInterval = (newStartState?: boolean, msg?: string) => {
         this.running = true;
+        if(newStartState !== undefined) {
+            this.newStart = newStartState;
+        }
+        const startMsg = `Polling Interval Started${msg !== undefined ? `: ${msg}` : ''}`;
+        this.logger.debug(startMsg)
         this.createInterval();
     }
 
-    end = () => {
+    end = (reason?: string) => {
+        let msg ='Stopping Polling Interval';
+        if(reason !== undefined) {
+            msg += `: ${reason}`;
+        }
+        this.logger.debug(msg);
         this.running = false;
+        this.newStart = true;
         super.end();
     }
 }
@@ -92,12 +153,13 @@ export class SPoll<T extends object> extends Poll<T> {
 export class UnmoderatedStream extends SPoll<Snoowrap.Submission | Snoowrap.Comment> {
     constructor(
         client: Snoowrap,
-        options: RCBPollingOptions) {
+        options: RCBPollingOptions<Snoowrap.Submission | Snoowrap.Comment>) {
         super({
             frequency: options.pollTime || DEFAULT_POLLING_INTERVAL * 1000,
             get: async () => client.getSubreddit(options.subreddit).getUnmoderated(options),
             identifier: "id",
-            clearProcessed: options.clearProcessed
+            name: 'Unmoderated',
+            ...options,
         });
     }
 }
@@ -105,12 +167,13 @@ export class UnmoderatedStream extends SPoll<Snoowrap.Submission | Snoowrap.Comm
 export class ModQueueStream extends SPoll<Snoowrap.Submission | Snoowrap.Comment> {
     constructor(
         client: Snoowrap,
-        options: RCBPollingOptions) {
+        options: RCBPollingOptions<Snoowrap.Submission | Snoowrap.Comment>) {
         super({
             frequency: options.pollTime || DEFAULT_POLLING_INTERVAL * 1000,
             get: async () => client.getSubreddit(options.subreddit).getModqueue(options),
             identifier: "id",
-            clearProcessed: options.clearProcessed
+            name: 'Modqueue',
+            ...options,
         });
     }
 }
@@ -118,12 +181,13 @@ export class ModQueueStream extends SPoll<Snoowrap.Submission | Snoowrap.Comment
 export class SubmissionStream extends SPoll<Snoowrap.Submission | Snoowrap.Comment> {
     constructor(
         client: Snoowrap,
-        options: RCBPollingOptions) {
+        options: RCBPollingOptions<Snoowrap.Submission | Snoowrap.Comment>) {
         super({
             frequency: options.pollTime || DEFAULT_POLLING_INTERVAL * 1000,
             get: async () => client.getNew(options.subreddit, options),
             identifier: "id",
-            clearProcessed: options.clearProcessed
+            name: 'Submission',
+            ...options,
         });
     }
 }
@@ -131,12 +195,13 @@ export class SubmissionStream extends SPoll<Snoowrap.Submission | Snoowrap.Comme
 export class CommentStream extends SPoll<Snoowrap.Submission | Snoowrap.Comment> {
     constructor(
         client: Snoowrap,
-        options: RCBPollingOptions) {
+        options: RCBPollingOptions<Snoowrap.Submission | Snoowrap.Comment>) {
         super({
             frequency: options.pollTime || DEFAULT_POLLING_INTERVAL * 1000,
             get: async () => client.getNewComments(options.subreddit, options),
             identifier: "id",
-            clearProcessed: options.clearProcessed
+            name: 'Comment',
+            ...options,
         });
     }
 }

src/Subreddit/SubredditResources.ts

@@ -12,13 +12,29 @@ import winston, {Logger} from "winston";
 import as from 'async';
 import fetch from 'node-fetch';
 import {
+    asActivity,
     asSubmission,
-    buildCacheOptionsFromProvider, buildCachePrefix,
-    cacheStats, compareDurationValue, comparisonTextOp, createCacheManager, createHistoricalStatsDisplay,
-    formatNumber, getActivityAuthorName, getActivitySubredditName, isStrongSubredditState,
-    mergeArr, parseDurationComparison,
-    parseExternalUrl, parseGenericValueComparison, parseRedditEntity,
-    parseWikiContext, shouldCacheSubredditStateCriteriaResult, subredditStateIsNameOnly, toStrongSubredditState
+    buildCacheOptionsFromProvider,
+    buildCachePrefix,
+    cacheStats,
+    compareDurationValue,
+    comparisonTextOp,
+    createCacheManager,
+    createHistoricalStatsDisplay, escapeRegex, FAIL,
+    fetchExternalUrl, filterCriteriaSummary,
+    formatNumber,
+    getActivityAuthorName,
+    getActivitySubredditName,
+    isStrongSubredditState, isSubmission, isUser,
+    mergeArr,
+    parseDurationComparison,
+    parseExternalUrl,
+    parseGenericValueComparison,
+    parseRedditEntity, parseStringToRegex,
+    parseWikiContext, PASS, redisScanIterator,
+    shouldCacheSubredditStateCriteriaResult,
+    subredditStateIsNameOnly,
+    toStrongSubredditState
 } from "../util";
 import LoggedError from "../Utils/LoggedError";
 import {
@@ -40,12 +56,12 @@ import {
     HistoricalStats,
     HistoricalStatUpdateData,
     SubredditHistoricalStats,
-    SubredditHistoricalStatsDisplay, ThirdPartyCredentialsJsonConfig,
+    SubredditHistoricalStatsDisplay, ThirdPartyCredentialsJsonConfig, FilterCriteriaResult,
 } from "../Common/interfaces";
 import UserNotes from "./UserNotes";
 import Mustache from "mustache";
 import he from "he";
-import {AuthorCriteria} from "../Author/Author";
+import {AuthorCriteria, AuthorOptions} from "../Author/Author";
 import {SPoll} from "./Streams";
 import {Cache} from 'cache-manager';
 import {Submission, Comment, Subreddit} from "snoowrap/dist/objects";
@@ -54,6 +70,7 @@ import {check} from "tcp-port-used";
 import {ExtendedSnoowrap} from "../Utils/SnoowrapClients";
 import dayjs from "dayjs";
 import ImageData from "../Common/ImageData";
+import globrex from 'globrex';
 
 export const DEFAULT_FOOTER = '\r\n*****\r\nThis action was performed by [a bot.]({{botLink}}) Mention a moderator or [send a modmail]({{modmailLink}}) if you any ideas, questions, or concerns about this action.';
 
@@ -170,7 +187,7 @@ export class SubredditResources {
             this.stats.cache.userNotes.requests++;
             this.stats.cache.userNotes.miss += miss ? 1 : 0;
         }
-        this.userNotes = new UserNotes(userNotesTTL, this.subreddit, this.logger, this.cache, cacheUseCB)
+        this.userNotes = new UserNotes(userNotesTTL, this.subreddit, this.client, this.logger, this.cache, cacheUseCB)
 
         if(this.cacheType === 'memory' && this.cacheSettingsHash !== 'default') {
             const min = Math.min(...([this.wikiTTL, this.authorTTL, this.submissionTTL, this.commentTTL, this.filterCriteriaTTL].filter(x => typeof x === 'number' && x !== 0) as number[]));
@@ -190,24 +207,23 @@ export class SubredditResources {
          const at = await this.cache.wrap(`${this.name}-historical-allTime`, () => createHistoricalDefaults(), {ttl: 0}) as object;
          const rehydratedAt: any = {};
          for(const [k, v] of Object.entries(at)) {
-            if(Array.isArray(v)) {
+             const t = typeof v;
+             if(t === 'number') {
+                 // simple number stat like eventsCheckedTotal
+                 rehydratedAt[k] = v;
+             } else if(Array.isArray(v)) {
+                 // a map stat that we have data for is serialized as an array of KV pairs
                 rehydratedAt[k] = new Map(v);
-            } else {
-                rehydratedAt[k] = v;
-            }
+             } else if(v === null || v === undefined || (t === 'object' && Object.keys(v).length === 0)) {
+                 // a map stat that was not serialized (for some reason) or serialized without any data
+                 rehydratedAt[k] = new Map();
+             } else {
+                 // ???? shouldn't get here
+                 this.logger.warn(`Did not recognize rehydrated historical stat "${k}" of type ${t}`);
+                 rehydratedAt[k] = v;
+             }
          }
          this.stats.historical.allTime = rehydratedAt as HistoricalStats;
-
-        // const lr = await this.cache.wrap(`${this.name}-historical-lastReload`, () => createHistoricalDefaults(), {ttl: 0}) as object;
-        // const rehydratedLr: any = {};
-        // for(const [k, v] of Object.entries(lr)) {
-        //     if(Array.isArray(v)) {
-        //         rehydratedLr[k] = new Map(v);
-        //     } else {
-        //         rehydratedLr[k] = v;
-        //     }
-        // }
-        // this.stats.historical.lastReload = rehydratedLr;
     }
 
     updateHistoricalStats(data: HistoricalStatUpdateData) {
@@ -283,6 +299,88 @@ export class SubredditResources {
         return 0;
     }
 
+    async interactWithCacheByKeyPattern(pattern: string | RegExp, action: 'get' | 'delete') {
+        let patternIsReg = pattern instanceof RegExp;
+        let regPattern: RegExp;
+        let globPattern = pattern;
+
+        const cacheDict: Record<string, any> = {};
+
+        if (typeof pattern === 'string') {
+            const possibleRegPattern = parseStringToRegex(pattern, 'ig');
+            if (possibleRegPattern !== undefined) {
+                regPattern = possibleRegPattern;
+                patternIsReg = true;
+            } else {
+                if (this.prefix !== undefined && !pattern.includes(this.prefix)) {
+                    // need to add wildcard to beginning of pattern so that the regex will still match a key with a prefix
+                    globPattern = `${this.prefix}${pattern}`;
+                }
+                // @ts-ignore
+                const result = globrex(globPattern, {flags: 'i'});
+                regPattern = result.regex;
+            }
+        } else {
+            regPattern = pattern;
+        }
+
+        if (this.cacheType === 'redis') {
+            // @ts-ignore
+            const redisClient = this.cache.store.getClient();
+            if (patternIsReg) {
+                // scan all and test key by regex
+                for await (const key of redisClient.scanIterator()) {
+                    if (regPattern.test(key) && (this.prefix === undefined || key.includes(this.prefix))) {
+                        if (action === 'delete') {
+                            await redisClient.del(key)
+                        } else {
+                            cacheDict[key] = await redisClient.get(key);
+                        }
+                    }
+                }
+            } else {
+                // not a regex means we can use glob pattern (more efficient!)
+                for await (const key of redisScanIterator(redisClient, { MATCH: globPattern })) {
+                    if (action === 'delete') {
+                        await redisClient.del(key)
+                    } else {
+                        cacheDict[key] = await redisClient.get(key);
+                    }
+                }
+            }
+        } else if (this.cache.store.keys !== undefined) {
+            for (const key of await this.cache.store.keys()) {
+                if (regPattern.test(key) && (this.prefix === undefined || key.includes(this.prefix))) {
+                    if (action === 'delete') {
+                        await this.cache.del(key)
+                    } else {
+                        cacheDict[key] = await this.cache.get(key);
+                    }
+                }
+            }
+        }
+        return cacheDict;
+    }
+
+    async deleteCacheByKeyPattern(pattern: string | RegExp) {
+        return await this.interactWithCacheByKeyPattern(pattern, 'delete');
+    }
+
+    async getCacheByKeyPattern(pattern: string | RegExp) {
+        return await this.interactWithCacheByKeyPattern(pattern, 'get');
+    }
+
+    async resetCacheForItem(item: Comment | Submission | RedditUser) {
+        if (asActivity(item)) {
+            if (this.filterCriteriaTTL !== false) {
+                await this.deleteCacheByKeyPattern(`itemCrit-${item.name}*`);
+            }
+            await this.setActivity(item, false);
+        } else if (isUser(item) && this.filterCriteriaTTL !== false) {
+            await this.deleteCacheByKeyPattern(`authorCrit-*-${getActivityAuthorName(item)}*`);
+        }
+    }
+
     async getStats() {
         const totals = Object.values(this.stats.cache).reduce((acc, curr) => ({
             miss: acc.miss + curr.miss,
@@ -364,11 +462,8 @@ export class SubredditResources {
                     this.logger.debug(`Cache Hit: Submission ${item.name}`);
                     return cachedSubmission;
                 }
-                // @ts-ignore
-                const submission = await item.fetch();
                 this.stats.cache.submission.miss++;
-                await this.cache.set(hash, submission, {ttl: this.submissionTTL});
-                return submission;
+                return await this.setActivity(item);
             } else if (this.commentTTL !== false) {
                 hash = `comm-${item.name}`;
                 await this.stats.cache.comment.identifierRequestCount.set(hash, (await this.stats.cache.comment.identifierRequestCount.wrap(hash, () => 0) as number) + 1);
@@ -379,11 +474,8 @@ export class SubredditResources {
                     this.logger.debug(`Cache Hit: Comment ${item.name}`);
                     return cachedComment;
                 }
-                // @ts-ignore
-                const comment = await item.fetch();
                 this.stats.cache.comment.miss++;
-                await this.cache.set(hash, comment, {ttl: this.commentTTL});
-                return comment;
+                return this.setActivity(item);
             } else {
                 // @ts-ignore
                 return await item.fetch();
@@ -394,6 +486,37 @@ export class SubredditResources {
         }
     }
 
+    // @ts-ignore
+    public async setActivity(item: Submission | Comment, tryToFetch = true)
+    {
+        let hash = '';
+        if(this.submissionTTL !== false && isSubmission(item)) {
+            hash = `sub-${item.name}`;
+            if(tryToFetch && item instanceof Submission) {
+                // @ts-ignore
+                const itemToCache = await item.fetch();
+                await this.cache.set(hash, itemToCache, {ttl: this.submissionTTL});
+                return itemToCache;
+            } else {
+                // @ts-ignore
+                await this.cache.set(hash, item, {ttl: this.submissionTTL});
+                return item;
+            }
+        } else if(this.commentTTL !== false){
+            hash = `comm-${item.name}`;
+            if(tryToFetch && item instanceof Comment) {
+                // @ts-ignore
+                const itemToCache = await item.fetch();
+                await this.cache.set(hash, itemToCache, {ttl: this.commentTTL});
+                return itemToCache;
+            } else {
+                // @ts-ignore
+                await this.cache.set(hash, item, {ttl: this.commentTTL});
+                return item;
+            }
+        }
+    }
+
     async hasActivity(item: Submission | Comment) {
         const hash = asSubmission(item) ? `sub-${item.name}` : `comm-${item.name}`;
         const res = await this.cache.get(hash);
@@ -593,8 +716,7 @@ export class SubredditResources {
             }
         } else {
             try {
-                const response = await fetch(extUrl as string);
-                wikiContent = await response.text();
+                wikiContent = await fetchExternalUrl(extUrl as string, this.logger);
             } catch (err: any) {
                 const msg = `Error occurred while trying to fetch the url ${extUrl}`;
                 this.logger.error(msg, err);
@@ -711,7 +833,7 @@ export class SubredditResources {
         return await this.isSubreddit(await this.getSubreddit(item), state, this.logger);
     }
 
-    async testAuthorCriteria(item: (Comment | Submission), authorOpts: AuthorCriteria, include = true) {
+    async testAuthorCriteria(item: (Comment | Submission), authorOpts: AuthorCriteria, include = true): Promise<FilterCriteriaResult<AuthorCriteria>> {
         if (this.filterCriteriaTTL !== false) {
             // in the criteria check we only actually use the `item` to get the author flair
             // which will be the same for the entire subreddit
@@ -724,17 +846,18 @@ export class SubredditResources {
             await this.stats.cache.authorCrit.identifierRequestCount.set(hash, (await this.stats.cache.authorCrit.identifierRequestCount.wrap(hash, () => 0) as number) + 1);
             this.stats.cache.authorCrit.requestTimestamps.push(Date.now());
             this.stats.cache.authorCrit.requests++;
-            let miss = false;
-            const cachedAuthorTest = await this.cache.wrap(hash, async () => {
-                miss = true;
-                return await testAuthorCriteria(item, authorOpts, include, this.userNotes);
-            }, {ttl: this.filterCriteriaTTL});
-            if (!miss) {
+
+            // need to check shape of result to invalidate old result type
+            let cachedAuthorTest: FilterCriteriaResult<AuthorCriteria> = await this.cache.get(hash) as FilterCriteriaResult<AuthorCriteria>;
+            if(cachedAuthorTest !== null && cachedAuthorTest !== undefined && typeof cachedAuthorTest === 'object') {
                 this.logger.debug(`Cache Hit: Author Check on ${userName} (Hash ${hash})`);
+                return cachedAuthorTest;
             } else {
                 this.stats.cache.authorCrit.miss++;
+                cachedAuthorTest = await testAuthorCriteria(item, authorOpts, include, this.userNotes);
+                await this.cache.set(hash, cachedAuthorTest, {ttl: this.filterCriteriaTTL});
+                return cachedAuthorTest;
             }
-            return cachedAuthorTest;
         }
 
         return await testAuthorCriteria(item, authorOpts, include, this.userNotes);
@@ -764,7 +887,7 @@ export class SubredditResources {
                 const cachedItem = await this.cache.get(hash);
                 if (cachedItem !== undefined && cachedItem !== null) {
                     this.logger.debug(`Cache Hit: Item Check on ${item.name} (Hash ${hash})`);
-                    return cachedItem as boolean;
+                    //return cachedItem as boolean;
                 }
                 const itemResult = await this.isItem(item, states, this.logger);
                 this.stats.cache.itemCrit.miss++;
@@ -781,8 +904,8 @@ export class SubredditResources {
         return await this.isItem(i, activityStates, this.logger);
     }
 
-    async isSubreddit (subreddit: Subreddit, stateCriteria: SubredditState | StrongSubredditState, logger: Logger) {
-        delete stateCriteria.stateDescription;
+    async isSubreddit (subreddit: Subreddit, stateCriteriaRaw: SubredditState | StrongSubredditState, logger: Logger) {
+        const {stateDescription, ...stateCriteria} = stateCriteriaRaw;
 
         if (Object.keys(stateCriteria).length === 0) {
             return true;
@@ -858,7 +981,7 @@ export class SubredditResources {
                     if (crit[k] !== undefined) {
                         switch (k) {
                             case 'submissionState':
-                                if(!(item instanceof Comment)) {
+                                if(isSubmission(item)) {
                                     log.warn('`submissionState` is not allowed in `itemIs` criteria when the main Activity is a Submission');
                                     continue;
                                 }
@@ -963,6 +1086,20 @@ export class SubredditResources {
                                     return false
                                 }
                                 break;
+                            case 'isRedditMediaDomain':
+                                if((item instanceof Comment)) {
+                                    log.warn('`isRedditMediaDomain` is not allowed in `itemIs` criteria when the main Activity is a Comment');
+                                    continue;
+                                }
+                                // @ts-ignore
+                                const isRedditDomain = crit[k] as boolean;
+                                // @ts-ignore
+                                if (item.is_reddit_media_domain !== isRedditDomain) {
+                                    // @ts-ignore
+                                    log.debug(`Failed: Expected => ${k}:${crit[k]} | Found => ${k}:${item.is_reddit_media_domain}`)
+                                    return false
+                                }
+                                break;
                             case 'approved':
                             case 'spam':
                                 if(!item.can_mod_post) {
@@ -977,7 +1114,7 @@ export class SubredditResources {
                                 }
                                 break;
                             case 'op':
-                                if(item instanceof Submission) {
+                                if(isSubmission(item)) {
                                     log.warn(`On a Submission the 'op' property will always be true. Did you mean to use this on a comment instead?`);
                                     break;
                                 }
@@ -989,7 +1126,7 @@ export class SubredditResources {
                                 }
                                 break;
                             case 'depth':
-                                if(item instanceof Submission) {
+                                if(isSubmission(item)) {
                                     log.warn(`Cannot test for 'depth' on a Submission`);
                                     break;
                                 }
@@ -1001,6 +1138,36 @@ export class SubredditResources {
                                     return false
                                 }
                                 break;
+                            case 'flairTemplate':
+                            case 'link_flair_text':
+                            case 'link_flair_css_class':
+                                if(asSubmission(item)) {
+                                    const subCrit = crit as SubmissionState;
+                                    let propertyValue: string | null;
+                                    if(k === 'flairTemplate') {
+                                        propertyValue = await item.link_flair_template_id;
+                                    } else {
+                                        propertyValue = await item[k];
+                                    }
+                                    const expectedValues = typeof subCrit[k] === 'string' ? [subCrit[k]] : (subCrit[k] as string[]);
+                                    const VALUEPass = () => {
+                                        for (const c of expectedValues) {
+                                            if (c === propertyValue) {
+                                                return true;
+                                            }
+                                        }
+                                        return false;
+                                    };
+                                    const valueResult = VALUEPass();
+                                    if(!valueResult) {
+                                        log.debug(`Failed: Expected => ${k} ${expectedValues.join(' OR ')} | Found => ${k}:${propertyValue}`);
+                                        return false;
+                                    }
+                                    break;
+                                } else {
+                                    log.warn(`Cannot test for ${k} on Comment`);
+                                    break;
+                                }
                             default:
                                 // @ts-ignore
                                 if (item[k] !== undefined) {
@@ -1277,3 +1444,74 @@ export class BotResourcesManager {
         return;
     }
 }
+
+export const checkAuthorFilter = async (item: (Submission | Comment), filter: AuthorOptions, resources: SubredditResources, logger: Logger): Promise<[boolean, ('inclusive' | 'exclusive' | undefined)]> => {
+    const authLogger = logger.child({labels: ['Author Filter']}, mergeArr);
+    const {
+        include = [],
+        excludeCondition = 'AND',
+        exclude = [],
+    } = filter;
+    let authorPass = null;
+    if (include.length > 0) {
+        let index = 1;
+        for (const auth of include) {
+            const critResult = await resources.testAuthorCriteria(item, auth);
+            const [summary, details] = filterCriteriaSummary(critResult);
+            if (critResult.passed) {
+                authLogger.verbose(`${PASS} => Inclusive Author Criteria ${index} => ${summary}`);
+                authLogger.debug(`Criteria Details: \n${details.join('\n')}`);
+                return [true, 'inclusive'];
+            } else {
+                authLogger.debug(`${FAIL} => Inclusive Author Criteria ${index} => ${summary}`);
+                authLogger.debug(`Criteria Details: \n${details.join('\n')}`);
+            }
+            index++;
+        }
+        authLogger.verbose(`${FAIL} => No Inclusive Author Criteria matched`);
+        return [false, 'inclusive'];
+    }
+    if (exclude.length > 0) {
+        let index = 1;
+        const summaries: string[] = [];
+        for (const auth of exclude) {
+            const critResult = await resources.testAuthorCriteria(item, auth, false);
+            const [summary, details] = filterCriteriaSummary(critResult);
+            if (critResult.passed) {
+                if(excludeCondition === 'OR') {
+                    authLogger.verbose(`${PASS} (OR) => Exclusive Author Criteria ${index} => ${summary}`);
+                    authLogger.debug(`Criteria Details: \n${details.join('\n')}`);
+                    authorPass = true;
+                    break;
+                }
+                summaries.push(summary);
+                authLogger.debug(`${PASS} (AND) => Exclusive Author Criteria ${index} => ${summary}`);
+                authLogger.debug(`Criteria Details: \n${details.join('\n')}`);
+            } else if (!critResult.passed) {
+                if(excludeCondition === 'AND') {
+                    authLogger.verbose(`${FAIL} (AND) => Exclusive Author Criteria ${index} => ${summary}`);
+                    authLogger.debug(`Criteria Details: \n${details.join('\n')}`);
+                    authorPass = false;
+                    break;
+                }
+                summaries.push(summary);
+                authLogger.debug(`${FAIL} (OR) => Exclusive Author Criteria ${index} => ${summary}`);
+                authLogger.debug(`Criteria Details: \n${details.join('\n')}`);
+            }
+            index++;
+        }
+        if(excludeCondition === 'AND' && authorPass === null) {
+            authorPass = true;
+        }
+        if (authorPass !== true) {
+            if(excludeCondition === 'OR') {
+                authLogger.verbose(`${FAIL} => Exclusive author criteria not matched => ${summaries.length === 1 ? `${summaries[0]}` : '(many, see debug)'}`);
+            }
+            return [false, 'exclusive']
+        } else if(excludeCondition === 'AND') {
+            authLogger.verbose(`${PASS} => Exclusive author criteria matched => ${summaries.length === 1 ? `${summaries[0]}` : '(many, see debug)'}`);
+        }
+        return [true, 'exclusive'];
+    }
+    return [true, undefined];
+}

src/Subreddit/UserNotes.ts

@@ -1,5 +1,5 @@
 import dayjs, {Dayjs} from "dayjs";
-import {Comment, RedditUser, WikiPage} from "snoowrap";
+import Snoowrap, {Comment, RedditUser, WikiPage} from "snoowrap";
 import {
     COMMENT_URL_ID,
     deflateUserNotes, getActivityAuthorName,
@@ -14,6 +14,7 @@ import Submission from "snoowrap/dist/objects/Submission";
 import {RichContent} from "../Common/interfaces";
 import {Cache} from 'cache-manager';
 import {isScopeError} from "../Utils/Errors";
+import {ErrorWithCause} from "pony-cause";
 
 interface RawUserNotesPayload {
     ver: number,
@@ -57,12 +58,13 @@ export type UserNotesConstants = Pick<any, "users" | "warnings">;
 export class UserNotes {
     notesTTL: number | false;
     subreddit: Subreddit;
-    wiki: WikiPage;
+    client: Snoowrap;
     moderators?: RedditUser[];
     logger: Logger;
     identifier: string;
     cache: Cache
     cacheCB: Function;
+    mod?: RedditUser;
 
     users: Map<string, UserNote[]> = new Map();
 
@@ -70,14 +72,14 @@ export class UserNotes {
     debounceCB: any;
     batchCount: number = 0;
 
-    constructor(ttl: number | boolean, subreddit: Subreddit, logger: Logger, cache: Cache, cacheCB: Function) {
+    constructor(ttl: number | boolean, subreddit: Subreddit, client: Snoowrap, logger: Logger, cache: Cache, cacheCB: Function) {
         this.notesTTL = ttl === true ? 0 : ttl;
         this.subreddit = subreddit;
         this.logger = logger;
-        this.wiki = subreddit.getWikiPage('usernotes');
         this.identifier = `${this.subreddit.display_name}-usernotes`;
         this.cache = cache;
         this.cacheCB = cacheCB;
+        this.client = client;
     }
 
     async getUserNotes(user: RedditUser): Promise<UserNote[]> {
@@ -110,14 +112,22 @@ export class UserNotes {
         }
     }
 
+    // @ts-ignore
+    async getMod() {
+        if(this.mod === undefined) {
+            // idgaf
+            // @ts-ignore
+            this.mod = await this.subreddit._r.getMe();
+        }
+        return this.mod as RedditUser;
+    }
+
     async addUserNote(item: (Submission|Comment), type: string | number, text: string = ''): Promise<UserNote>
     {
         const payload = await this.retrieveData();
         const userName = getActivityAuthorName(item.author);
 
-        // idgaf
-        // @ts-ignore
-        const mod = await this.subreddit._r.getMe();
+        const mod = await this.getMod();
         if(!payload.constants.users.includes(mod.name)) {
             this.logger.info(`Mod ${mod.name} does not exist in UserNote constants, adding them`);
             payload.constants.users.push(mod.name);
@@ -134,11 +144,11 @@ export class UserNotes {
         }
         payload.blob[userName].ns.push(newNote.toRaw(payload.constants));
 
+        const existingNotes = await this.getUserNotes(item.author);
         await this.saveData(payload);
         if(this.notesTTL > 0) {
-            const currNotes = this.users.get(userName) || [];
-            currNotes.push(newNote);
-            this.users.set(userName, currNotes);
+            existingNotes.push(newNote);
+            this.users.set(userName, existingNotes);
         }
         return newNote;
     }
@@ -150,7 +160,6 @@ export class UserNotes {
     }
 
     async retrieveData(): Promise<RawUserNotesPayload> {
-        let cacheMiss;
         if (this.notesTTL > 0) {
             const cachedPayload = await this.cache.get(this.identifier);
             if (cachedPayload !== undefined && cachedPayload !== null) {
@@ -158,22 +167,12 @@ export class UserNotes {
                 return cachedPayload as unknown as RawUserNotesPayload;
             }
             this.cacheCB(true);
-            cacheMiss = true;
         }
 
         try {
-            // DISABLED for now because I think its causing issues
-            // if(cacheMiss && this.debounceCB !== undefined) {
-            //     // timeout is still delayed. its our wiki data and we want it now! cm cacheworth 877 cache now
-            //     this.logger.debug(`Detected missed cache on usernotes retrieval while batch (${this.batchCount}) save is in progress, executing save immediately before retrieving new notes...`);
-            //     clearTimeout(this.saveDebounce);
-            //     await this.debounceCB();
-            //     this.debounceCB = undefined;
-            //     this.saveDebounce = undefined;
-            // }
             // @ts-ignore
-            this.wiki = await this.subreddit.getWikiPage('usernotes').fetch();
-            const wikiContent = this.wiki.content_md;
+            const wiki = this.client.getSubreddit(this.subreddit.display_name).getWikiPage('usernotes');
+            const wikiContent = await wiki.content_md;
             // TODO don't handle for versions lower than 6
             const userNotes = JSON.parse(wikiContent);
 
@@ -197,54 +196,27 @@ export class UserNotes {
         const blob = deflateUserNotes(payload.blob);
         const wikiPayload = {text: JSON.stringify({...payload, blob}), reason: 'ContextBot edited usernotes'};
         try {
+            const wiki = this.client.getSubreddit(this.subreddit.display_name).getWikiPage('usernotes');
             if (this.notesTTL !== false) {
-                // DISABLED for now because if it fails throws an uncaught rejection
-                // and need to figured out how to handle this other than just logging (want to interrupt action flow too?)
-                //
-                // debounce usernote save by 5 seconds -- effectively batch usernote saves
-                //
-                // so that if we are processing a ton of checks that write user notes we aren't calling to save the wiki page on every call
-                // since we also have everything in cache (most likely...)
-                //
-                // TODO might want to increase timeout to 10 seconds
-                // if(this.saveDebounce !== undefined) {
-                //     clearTimeout(this.saveDebounce);
-                // }
-                // this.debounceCB = (async function () {
-                //     const p = wikiPayload;
-                //     // @ts-ignore
-                //     const self = this as UserNotes;
-                //     // @ts-ignore
-                //     self.wiki = await self.subreddit.getWikiPage('usernotes').edit(p);
-                //     self.logger.debug(`Batch saved ${self.batchCount} usernotes`);
-                //     self.debounceCB = undefined;
-                //     self.saveDebounce = undefined;
-                //     self.batchCount = 0;
-                // }).bind(this);
-                // this.saveDebounce = setTimeout(this.debounceCB,5000);
-                // this.batchCount++;
-                // this.logger.debug(`Saving Usernotes has been debounced for 5 seconds (${this.batchCount} batched)`)
-
                 // @ts-ignore
-                await this.subreddit.getWikiPage('usernotes').edit(wikiPayload);
+                await wiki.edit(wikiPayload);
                 await this.cache.set(this.identifier, payload, {ttl: this.notesTTL});
                 this.users = new Map();
             } else {
                 // @ts-ignore
-                this.wiki = await this.subreddit.getWikiPage('usernotes').edit(wikiPayload);
+                await wiki.edit(wikiPayload);
             }
 
             return payload as RawUserNotesPayload;
         } catch (err: any) {
-            let msg = 'Could not edit usernotes.';
+            let msg = 'Could not edit usernotes!';
             // Make sure at least one moderator has used toolbox and usernotes before and that this account has editing permissions`;
             if(isScopeError(err)) {
                 msg = `${msg} The bot account did not have sufficient OAUTH scope to perform this action. You must re-authenticate the bot and ensure it has has 'wikiedit' permissions.`
             } else {
                 msg = `${msg} Make sure at least one moderator has used toolbox, created a usernote, and that this account has editing permissions for the wiki page.`;
             }
-            this.logger.error(msg, err);
-            throw new LoggedError(msg);
+            throw new ErrorWithCause(msg, {cause: err});
         }
     }
 }

src/Utils/CommandConfig.ts

@@ -56,7 +56,7 @@ export const port = new commander.Option('-p, --port <port>', 'Port for web serv
 export const sharedMod = new commander.Option('-q, --shareMod', `If enabled then all subreddits using the default settings to poll "unmoderated" or "modqueue" will retrieve results from a shared request to /r/mod (default: process.env.SHARE_MOD || false)`)
     .argParser(parseBool);
 
-export const operatorConfig = new commander.Option('-c, --operatorConfig <path>', 'An absolute path to a JSON file to load all parameters from (default: process.env.OPERATOR_CONFIG)');
+export const operatorConfig = new commander.Option('-c, --operatorConfig <path>', 'An absolute path to a YAML/JSON file to load all parameters from (default: process.env.OPERATOR_CONFIG | CWD/config.yaml)');
 
 export const getUniversalWebOptions = (): commander.Option[] => {
     return [

src/Utils/Errors.ts

@@ -1,22 +1,39 @@
-import {StatusCodeError, RequestError} from "../Common/interfaces";
+import {RateLimitError, RequestError, StatusCodeError} from 'snoowrap/dist/errors';
+import ExtendableError from "es6-error";
+import {ErrorWithCause} from "pony-cause";
 
 
-export const isRateLimitError = (err: any) => {
-    return typeof err === 'object' && err.name === 'RateLimitError';
+export const isRateLimitError = (err: any): err is RateLimitError => {
+    return isRequestError(err) && err.name === 'RateLimitError';
 }
 
 export const isScopeError = (err: any): boolean => {
-    if(typeof err === 'object' && err.name === 'StatusCodeError' && err.response !== undefined) {
+    if(isStatusError(err)) {
         const authHeader = err.response.headers['www-authenticate'];
         return authHeader !== undefined && authHeader.includes('insufficient_scope');
     }
     return false;
 }
 
+export const getScopeError = (err: any): string | undefined => {
+    if(isScopeError(err)) {
+        return err.response.headers['www-authenticate'];
+    }
+    return undefined;
+}
+
 export const isStatusError = (err: any): err is StatusCodeError => {
-    return typeof err === 'object' && err.name === 'StatusCodeError' && err.response !== undefined;
+    return isRequestError(err) && err.name === 'StatusCodeError';
 }
 
 export const isRequestError = (err: any): err is RequestError => {
-    return typeof err === 'object' && err.name === 'RequestError' && err.response !== undefined;
+    return typeof err === 'object' && err.response !== undefined;
+}
+
+export class SimpleError extends ExtendableError {
+
+}
+
+export class CMError extends ErrorWithCause {
+    logged: boolean = false;
 }

src/Utils/SimpleError.ts

@@ -1,7 +0,0 @@
-import ExtendableError from "es6-error";
-
-class SimpleError extends ExtendableError {
-
-}
-
-export default SimpleError;

src/Utils/SnoowrapUtils.ts

@@ -8,30 +8,31 @@ import he from "he";
 import {RuleResult, UserNoteCriteria} from "../Rule";
 import {
     ActivityWindowType, CommentState, DomainInfo,
-    DurationVal,
+    DurationVal, FilterCriteriaPropertyResult, FilterCriteriaResult,
     SubmissionState,
     TypedActivityStates
 } from "../Common/interfaces";
 import {
+    asUserNoteCriteria,
     compareDurationValue,
-    comparisonTextOp, escapeRegex, getActivityAuthorName,
-    isActivityWindowCriteria,
+    comparisonTextOp, escapeRegex, formatNumber, getActivityAuthorName,
+    isActivityWindowCriteria, isUserNoteCriteria,
     normalizeName,
     parseDuration,
     parseDurationComparison,
     parseGenericValueComparison,
     parseGenericValueOrPercentComparison,
     parseRuleResultsToMarkdownSummary, parseStringToRegex,
-    parseSubredditName,
-    truncateStringToLength, windowToActivityWindowCriteria
+    parseSubredditName, removeUndefinedKeys,
+    truncateStringToLength, userNoteCriteriaSummary, windowToActivityWindowCriteria
 } from "../util";
 import UserNotes from "../Subreddit/UserNotes";
 import {Logger} from "winston";
 import InvalidRegexError from "./InvalidRegexError";
-import SimpleError from "./SimpleError";
 import {AuthorCriteria} from "../Author/Author";
 import {URL} from "url";
-import {isStatusError} from "./Errors";
+import {SimpleError, isStatusError} from "./Errors";
+import {Dictionary, ElementOf, SafeDictionary} from "ts-essentials";
 
 export const BOT_LINK = 'https://www.reddit.com/r/ContextModBot/comments/otz396/introduction_to_contextmodbot';
 
@@ -354,24 +355,54 @@ export const renderContent = async (template: string, data: (Submission | Commen
     return he.decode(rendered);
 }
 
-export const testAuthorCriteria = async (item: (Comment | Submission), authorOpts: AuthorCriteria, include = true, userNotes: UserNotes) => {
-    const {shadowBanned, ...rest} = authorOpts;
+type AuthorCritPropHelper = SafeDictionary<FilterCriteriaPropertyResult<AuthorCriteria>, keyof AuthorCriteria>;
+type RequiredAuthorCrit = Required<AuthorCriteria>;
 
-    if(shadowBanned !== undefined) {
+export const testAuthorCriteria = async (item: (Comment | Submission), authorOpts: AuthorCriteria, include = true, userNotes: UserNotes): Promise<FilterCriteriaResult<AuthorCriteria>> => {
+
+
+    const definedAuthorOpts = (removeUndefinedKeys(authorOpts) as RequiredAuthorCrit);
+
+    const propResultsMap = Object.entries(definedAuthorOpts).reduce((acc: AuthorCritPropHelper, [k, v]) => {
+        const key = (k as keyof AuthorCriteria);
+        let ex;
+        if (Array.isArray(v)) {
+            ex = v.map(x => {
+                if (asUserNoteCriteria(x)) {
+                    return userNoteCriteriaSummary(x);
+                }
+                return x;
+            });
+        } else {
+            ex = [v];
+        }
+        acc[key] = {
+            property: key,
+            expected: ex,
+            behavior: include ? 'include' : 'exclude',
+        };
+        return acc;
+    }, {});
+
+    const {shadowBanned} = authorOpts;
+
+    if (shadowBanned !== undefined) {
         try {
             // @ts-ignore
             await item.author.fetch();
             // user is not shadowbanned
             // if criteria specifies they SHOULD be shadowbanned then return false now
-            if(shadowBanned) {
-                return false;
+            if (shadowBanned) {
+                propResultsMap.shadowBanned!.found = false;
+                propResultsMap.shadowBanned!.passed = false;
             }
         } catch (err: any) {
-            if(isStatusError(err) && err.statusCode === 404) {
+            if (isStatusError(err) && err.statusCode === 404) {
                 // user is shadowbanned
                 // if criteria specifies they should not be shadowbanned then return false now
-                if(!shadowBanned) {
-                    return false;
+                if (!shadowBanned) {
+                    propResultsMap.shadowBanned!.found = true;
+                    propResultsMap.shadowBanned!.passed = false;
                 }
             } else {
                 throw err;
@@ -379,17 +410,30 @@ export const testAuthorCriteria = async (item: (Comment | Submission), authorOpt
         }
     }
 
-    try {
-        const authorName = getActivityAuthorName(item.author);
 
-        for (const k of Object.keys(rest)) {
-            // @ts-ignore
-            if (authorOpts[k] !== undefined) {
+
+    if (propResultsMap.shadowBanned === undefined || propResultsMap.shadowBanned.passed === undefined) {
+        try {
+            const authorName = getActivityAuthorName(item.author);
+
+            const keys = Object.keys(propResultsMap) as (keyof AuthorCriteria)[]
+
+            let shouldContinue = true;
+            for (const k of keys) {
+                if (k === 'shadowBanned') {
+                    // we have already taken care of this with shadowban check above
+                    continue;
+                }
+
+                const authorOptVal = definedAuthorOpts[k];
+
+                //if (authorOpts[k] !== undefined) {
                 switch (k) {
                     case 'name':
+                        const nameVal = authorOptVal as RequiredAuthorCrit['name'];
                         const authPass = () => {
-                            // @ts-ignore
-                            for (const n of authorOpts[k]) {
+
+                            for (const n of nameVal) {
                                 if (n.toLowerCase() === authorName.toLowerCase()) {
                                     return true;
                                 }
@@ -397,8 +441,10 @@ export const testAuthorCriteria = async (item: (Comment | Submission), authorOpt
                             return false;
                         }
                         const authResult = authPass();
-                        if ((include && !authResult) || (!include && authResult)) {
-                            return false;
+                        propResultsMap.name!.found = authorName;
+                        propResultsMap.name!.passed = !((include && !authResult) || (!include && authResult));
+                        if (!propResultsMap.name!.passed) {
+                            shouldContinue = false;
                         }
                         break;
                     case 'flairCssClass':
@@ -413,8 +459,10 @@ export const testAuthorCriteria = async (item: (Comment | Submission), authorOpt
                             return false;
                         }
                         const cssResult = cssPass();
-                        if ((include && !cssResult) || (!include && cssResult)) {
-                            return false;
+                        propResultsMap.flairCssClass!.found = css;
+                        propResultsMap.flairCssClass!.passed = !((include && !cssResult) || (!include && cssResult));
+                        if (!propResultsMap.flairCssClass!.passed) {
+                            shouldContinue = false;
                         }
                         break;
                     case 'flairText':
@@ -429,68 +477,103 @@ export const testAuthorCriteria = async (item: (Comment | Submission), authorOpt
                             return false;
                         };
                         const textResult = textPass();
-                        if ((include && !textResult) || (!include && textResult)) {
+                        propResultsMap.flairText!.found = text;
+                        propResultsMap.flairText!.passed = !((include && !textResult) || (!include && textResult));
+                        if (!propResultsMap.flairText!.passed) {
+                            shouldContinue = false;
+                        }
+                        break;
+                    case 'flairTemplate':
+                        const templateId = await item.author_flair_template_id;
+                        const templatePass = () => {
+                            // @ts-ignore
+                            for (const c of authorOpts[k]) {
+                                if (c === templateId) {
+                                    return true;
+                                }
+                            }
                             return false;
+                        };
+                        const templateResult = templatePass();
+                        propResultsMap.flairTemplate!.found = templateId;
+                        propResultsMap.flairTemplate!.passed = !((include && !templateResult) || (!include && templateResult));
+                        if (!propResultsMap.flairTemplate!.passed) {
+                            shouldContinue = false;
                         }
                         break;
                     case 'isMod':
                         const mods: RedditUser[] = await item.subreddit.getModerators();
-                        const isModerator = mods.some(x => x.name === authorName);
+                        const isModerator = mods.some(x => x.name === authorName) || authorName.toLowerCase() === 'automoderator';
                         const modMatch = authorOpts.isMod === isModerator;
-                        if ((include && !modMatch) || (!include && modMatch)) {
-                            return false;
+                        propResultsMap.isMod!.found = isModerator;
+                        propResultsMap.isMod!.passed = !((include && !modMatch) || (!include && modMatch));
+                        if (!propResultsMap.isMod!.passed) {
+                            shouldContinue = false;
                         }
                         break;
                     case 'age':
-                        const ageTest = compareDurationValue(parseDurationComparison(await authorOpts.age as string), dayjs.unix(await item.author.created));
-                        if ((include && !ageTest) || (!include && ageTest)) {
-                            return false;
+                        const authorAge = dayjs.unix(await item.author.created);
+                        const ageTest = compareDurationValue(parseDurationComparison(await authorOpts.age as string), authorAge);
+                        propResultsMap.age!.found = authorAge.fromNow(true);
+                        propResultsMap.age!.passed = !((include && !ageTest) || (!include && ageTest));
+                        if (!propResultsMap.age!.passed) {
+                            shouldContinue = false;
                         }
                         break;
                     case 'linkKarma':
+                        // @ts-ignore
+                        const tk = await item.author.total_karma as number;
                         const lkCompare = parseGenericValueOrPercentComparison(await authorOpts.linkKarma as string);
                         let lkMatch;
                         if (lkCompare.isPercent) {
-                            // @ts-ignore
-                            const tk = await item.author.total_karma as number;
+
                             lkMatch = comparisonTextOp(item.author.link_karma / tk, lkCompare.operator, lkCompare.value / 100);
                         } else {
                             lkMatch = comparisonTextOp(item.author.link_karma, lkCompare.operator, lkCompare.value);
                         }
-                        if ((include && !lkMatch) || (!include && lkMatch)) {
-                            return false;
+                        propResultsMap.linkKarma!.found = tk;
+                        propResultsMap.linkKarma!.passed = !((include && !lkMatch) || (!include && lkMatch));
+                        if (!propResultsMap.linkKarma!.passed) {
+                            shouldContinue = false;
                         }
                         break;
                     case 'commentKarma':
+                        // @ts-ignore
+                        const ck = await item.author.total_karma as number;
                         const ckCompare = parseGenericValueOrPercentComparison(await authorOpts.commentKarma as string);
                         let ckMatch;
                         if (ckCompare.isPercent) {
-                            // @ts-ignore
-                            const ck = await item.author.total_karma as number;
                             ckMatch = comparisonTextOp(item.author.comment_karma / ck, ckCompare.operator, ckCompare.value / 100);
                         } else {
                             ckMatch = comparisonTextOp(item.author.comment_karma, ckCompare.operator, ckCompare.value);
                         }
-                        if ((include && !ckMatch) || (!include && ckMatch)) {
-                            return false;
+                        propResultsMap.commentKarma!.found = ck;
+                        propResultsMap.commentKarma!.passed = !((include && !ckMatch) || (!include && ckMatch));
+                        if (!propResultsMap.commentKarma!.passed) {
+                            shouldContinue = false;
                         }
                         break;
                     case 'totalKarma':
+                        // @ts-ignore
+                        const totalKarma = await item.author.total_karma as number;
                         const tkCompare = parseGenericValueComparison(await authorOpts.totalKarma as string);
                         if (tkCompare.isPercent) {
                             throw new SimpleError(`'totalKarma' value on AuthorCriteria cannot be a percentage`);
                         }
-                        // @ts-ignore
-                        const totalKarma = await item.author.total_karma as number;
                         const tkMatch = comparisonTextOp(totalKarma, tkCompare.operator, tkCompare.value);
-                        if ((include && !tkMatch) || (!include && tkMatch)) {
-                            return false;
+                        propResultsMap.totalKarma!.found = totalKarma;
+                        propResultsMap.totalKarma!.passed = !((include && !tkMatch) || (!include && tkMatch));
+                        if (!propResultsMap.totalKarma!.passed) {
+                            shouldContinue = false;
                         }
                         break;
                     case 'verified':
-                        const vMatch = await item.author.has_verified_mail === authorOpts.verified as boolean;
-                        if ((include && !vMatch) || (!include && vMatch)) {
-                            return false;
+                        const verified = await item.author.has_verified_mail;
+                        const vMatch = verified === authorOpts.verified as boolean;
+                        propResultsMap.verified!.found = verified;
+                        propResultsMap.verified!.passed = !((include && !vMatch) || (!include && vMatch));
+                        if (!propResultsMap.verified!.passed) {
+                            shouldContinue = false;
                         }
                         break;
                     case 'description':
@@ -498,25 +581,32 @@ export const testAuthorCriteria = async (item: (Comment | Submission), authorOpt
                         const desc = await item.author.subreddit?.display_name.public_description;
                         const dVals = authorOpts[k] as string[];
                         let passed = false;
-                        for(const val of dVals) {
+                        let passReg;
+                        for (const val of dVals) {
                             let reg = parseStringToRegex(val, 'i');
-                            if(reg === undefined) {
+                            if (reg === undefined) {
                                 reg = parseStringToRegex(`/.*${escapeRegex(val.trim())}.*/`, 'i');
-                                if(reg === undefined) {
+                                if (reg === undefined) {
                                     throw new SimpleError(`Could not convert 'description' value to a valid regex: ${authorOpts[k] as string}`);
                                 }
                             }
-                            if(reg.test(desc)) {
+                            if (reg.test(desc)) {
                                 passed = true;
+                                passReg = reg.toString();
                                 break;
                             }
                         }
-                        if(!passed) {
-                            return false;
+                        propResultsMap.description!.found = typeof desc === 'string' ? truncateStringToLength(50)(desc) : desc;
+                        propResultsMap.description!.passed = !((include && !passed) || (!include && passed));
+                        if (!propResultsMap.description!.passed) {
+                            shouldContinue = false;
+                        } else {
+                            propResultsMap.description!.reason = `Matched with: ${passReg as string}`;
                         }
                         break;
                     case 'userNotes':
                         const notes = await userNotes.getUserNotes(item.author);
+                        let foundNoteResult: string[] = [];
                         const notePass = () => {
                             for (const noteCriteria of authorOpts[k] as UserNoteCriteria[]) {
                                 const {count = '>= 1', search = 'current', type} = noteCriteria;
@@ -529,8 +619,14 @@ export const testAuthorCriteria = async (item: (Comment | Submission), authorOpt
                                 const order = extra.includes('asc') ? 'ascending' : 'descending';
                                 switch (search) {
                                     case 'current':
-                                        if (notes.length > 0 && notes[notes.length - 1].noteType === type) {
-                                            return true;
+                                        if (notes.length > 0) {
+                                            const currentNoteType = notes[notes.length - 1].noteType;
+                                            foundNoteResult.push(`Current => ${currentNoteType}`);
+                                            if (currentNoteType === type) {
+                                                return true;
+                                            }
+                                        } else {
+                                            foundNoteResult.push('No notes present');
                                         }
                                         break;
                                     case 'consecutive':
@@ -549,39 +645,64 @@ export const testAuthorCriteria = async (item: (Comment | Submission), authorOpt
                                             if (isPercent) {
                                                 throw new SimpleError(`When comparing UserNotes with 'consecutive' search 'count' cannot be a percentage. Given: ${count}`);
                                             }
+                                            foundNoteResult.push(`Found ${currCount} ${type} consecutively`);
                                             if (comparisonTextOp(currCount, operator, value)) {
                                                 return true;
                                             }
                                         }
                                         break;
                                     case 'total':
+                                        const filteredNotes = notes.filter(x => x.noteType === type);
                                         if (isPercent) {
-                                            if (comparisonTextOp(notes.filter(x => x.noteType === type).length / notes.length, operator, value / 100)) {
+                                            // avoid divide by zero
+                                            const percent = notes.length === 0 ? 0 : filteredNotes.length / notes.length;
+                                            foundNoteResult.push(`${formatNumber(percent)}% are ${type}`);
+                                            if (comparisonTextOp(percent, operator, value / 100)) {
+                                                return true;
+                                            }
+                                        } else {
+                                            foundNoteResult.push(`${filteredNotes.length} are ${type}`);
+                                            if (comparisonTextOp(notes.filter(x => x.noteType === type).length, operator, value)) {
                                                 return true;
                                             }
-                                        } else if (comparisonTextOp(notes.filter(x => x.noteType === type).length, operator, value)) {
-                                            return true;
                                         }
+                                        break;
                                 }
                             }
                             return false;
                         }
                         const noteResult = notePass();
-                        if ((include && !noteResult) || (!include && noteResult)) {
-                            return false;
+                        propResultsMap.userNotes!.found = foundNoteResult.join(' | ');
+                        propResultsMap.userNotes!.passed = !((include && !noteResult) || (!include && noteResult));
+                        if (!propResultsMap.userNotes!.passed) {
+                            shouldContinue = false;
                         }
                         break;
                 }
+                //}
+                if (!shouldContinue) {
+                    break;
+                }
+            }
+        } catch (err: any) {
+            if (isStatusError(err) && err.statusCode === 404) {
+                throw new SimpleError('Reddit returned a 404 while trying to retrieve User profile. It is likely this user is shadowbanned.');
+            } else {
+                throw err;
             }
         }
-        return true;
-    } catch (err: any) {
-        if(isStatusError(err) && err.statusCode === 404) {
-            throw new SimpleError('Reddit returned a 404 while trying to retrieve User profile. It is likely this user is shadowbanned.');
-        } else {
-            throw err;
-        }
     }
+
+    // gather values and determine overall passed
+    const propResults = Object.values(propResultsMap);
+    const passed = propResults.filter(x => typeof x.passed === 'boolean').every(x => x.passed === true);
+
+    return {
+        behavior: include ? 'include' : 'exclude',
+        criteria: authorOpts,
+        propertyResults: propResults,
+        passed,
+    };
 }
 
 export interface ItemContent {

src/Utils/loggerFactory.ts

@@ -1,19 +1,28 @@
 import {labelledFormat, logLevels} from "../util";
 import winston, {Logger} from "winston";
 import {DuplexTransport} from "winston-duplex";
+import {LoggerFactoryOptions} from "../Common/interfaces";
+import process from "process";
+import path from "path";
 
 const {transports} = winston;
 
-export const getLogger = (options: any, name = 'app'): Logger => {
+export const getLogger = (options: LoggerFactoryOptions, name = 'app'): Logger => {
     if(!winston.loggers.has(name)) {
         const {
-            path,
             level,
             additionalTransports = [],
             defaultLabel = 'App',
+            file: {
+                dirname,
+                ...fileRest
+            },
+            console,
+            stream
         } = options || {};
 
         const consoleTransport = new transports.Console({
+            ...console,
             handleExceptions: true,
             handleRejections: true,
         });
@@ -28,21 +37,39 @@ export const getLogger = (options: any, name = 'app'): Logger => {
                     objectMode: true,
                 },
                 name: 'duplex',
-                dump: false,
                 handleExceptions: true,
                 handleRejections: true,
+                ...stream,
+                dump: false,
             }),
             ...additionalTransports,
         ];
 
-        if (path !== undefined && path !== '') {
+        if (dirname !== undefined && dirname !== '' && dirname !== null) {
+
+            let realDir: string | undefined;
+            if(typeof dirname === 'boolean') {
+                if(!dirname) {
+                    realDir = undefined;
+                } else {
+                    realDir = path.resolve(__dirname, '../../logs')
+                }
+            } else if(dirname === 'true') {
+                realDir = path.resolve(__dirname, '../../logs')
+            } else if(dirname === 'false') {
+                realDir = undefined;
+            } else {
+                realDir = dirname;
+            }
+
             const rotateTransport = new winston.transports.DailyRotateFile({
-                dirname: path,
                 createSymlink: true,
                 symlinkName: 'contextBot-current.log',
                 filename: 'contextBot-%DATE%.log',
                 datePattern: 'YYYY-MM-DD',
                 maxSize: '5m',
+                dirname: realDir,
+                ...fileRest,
                 handleExceptions: true,
                 handleRejections: true,
             });

src/Web/Client/CMInstance.ts

@@ -0,0 +1,148 @@
+import {URL} from "url";
+import {Logger} from "winston";
+import {BotInstance, CMInstanceInterface, CMInstanceInterface as CMInterface} from "../interfaces";
+import dayjs from 'dayjs';
+import {BotConnection, LogInfo} from "../../Common/interfaces";
+import normalizeUrl from "normalize-url";
+import {HeartbeatResponse} from "../Common/interfaces";
+import jwt from "jsonwebtoken";
+import got from "got";
+import {ErrorWithCause} from "pony-cause";
+
+export class CMInstance implements CMInterface {
+    friendly?: string;
+    operators: string[] = [];
+    operatorDisplay: string = '';
+    url: URL;
+    normalUrl: string;
+    lastCheck?: number;
+    online: boolean = false;
+    subreddits: string[] = [];
+    bots: BotInstance[] = [];
+    error?: string | undefined;
+    host: string;
+    secret: string;
+
+    logger: Logger;
+    logs: LogInfo[] = [];
+
+    constructor(options: BotConnection, logger: Logger) {
+        const {
+            host,
+            secret
+        } = options;
+
+        this.host = host;
+        this.secret = secret;
+
+        const normalized = normalizeUrl(options.host);
+        this.normalUrl = normalized;
+        this.url = new URL(normalized);
+
+        const name = this.getName;
+
+        this.logger = logger.child({
+            get instance() {
+                return name();
+            }
+        });
+
+        this.logger.stream().on('log', (log: LogInfo) => {
+            if(log.instance !== undefined && log.instance === this.getName()) {
+                this.logs = [log, ...this.logs].slice(0, 301);
+            }
+        });
+    }
+
+    getData(): CMInterface {
+        return {
+            friendly: this.getName(),
+            operators: this.operators,
+            operatorDisplay: this.operatorDisplay,
+            url: this.url,
+            normalUrl: this.normalUrl,
+            lastCheck: this.lastCheck,
+            online: this.online,
+            subreddits: this.subreddits,
+            bots: this.bots,
+            error: this.error,
+            host: this.host,
+            secret: this.secret
+        }
+    }
+
+    getName = () => {
+        if (this.friendly !== undefined) {
+            return this.friendly
+        }
+        return this.url.host;
+    }
+
+    matchesHost = (val: string) => {
+        return normalizeUrl(val) == this.normalUrl;
+    }
+
+    updateFromHeartbeat = (resp: HeartbeatResponse, otherFriendlies: string[] = []) => {
+        this.operators = resp.operators ?? [];
+        this.operatorDisplay = resp.operatorDisplay ?? '';
+
+        const fr = resp.friendly;
+        if (fr !== undefined) {
+            if (otherFriendlies.includes(fr)) {
+                this.logger.warn(`Client returned a friendly name that is not unique (${fr}), will fallback to host as friendly (${this.url.host})`);
+            } else {
+                this.friendly = fr;
+            }
+        }
+
+        this.subreddits = resp.subreddits;
+        this.bots = resp.bots.map(x => ({...x, instance: this}));
+    }
+
+    checkHeartbeat = async (force = false, otherFriendlies: string[] = []) => {
+        let shouldCheck = force;
+        if (!shouldCheck) {
+            if (this.lastCheck === undefined) {
+                shouldCheck = true;
+            } else {
+                const lastCheck = dayjs().diff(dayjs.unix(this.lastCheck), 's');
+                if (!this.online) {
+                    if (lastCheck > 15) {
+                        shouldCheck = true;
+                    }
+                } else if (lastCheck > 60) {
+                    shouldCheck = true;
+                }
+            }
+        }
+        if (shouldCheck) {
+            this.logger.debug('Starting Heartbeat check');
+            this.lastCheck = dayjs().unix();
+            const machineToken = jwt.sign({
+                data: {
+                    machine: true,
+                },
+            }, this.secret, {
+                expiresIn: '1m'
+            });
+
+            try {
+                const resp = await got.get(`${this.normalUrl}/heartbeat`, {
+                    headers: {
+                        'Authorization': `Bearer ${machineToken}`,
+                    }
+                }).json() as CMInstanceInterface;
+
+                this.online = true;
+                this.updateFromHeartbeat(resp as HeartbeatResponse, otherFriendlies);
+                this.logger.verbose(`Heartbeat detected`);
+            } catch (err: any) {
+                this.online = false;
+                this.error = err.message;
+                const badHeartbeat = new ErrorWithCause('Heartbeat response was not ok', {cause: err});
+                this.logger.error(badHeartbeat);
+            }
+        }
+    }
+
+}

src/Web/Client/index.ts

@@ -9,12 +9,12 @@ import {Strategy as CustomStrategy} from 'passport-custom';
 import {OperatorConfig, BotConnection, LogInfo} from "../../Common/interfaces";
 import {
     buildCachePrefix,
-    createCacheManager, filterLogBySubreddit,
-    formatLogLineToHtml,
+    createCacheManager, defaultFormat, filterLogBySubreddit, filterLogs,
+    formatLogLineToHtml, getUserAgent,
     intersect, isLogLineMinLevel,
-    LogEntry, parseFromJsonOrYamlToObject, parseInstanceLogInfoName, parseInstanceLogName,
+    LogEntry, parseInstanceLogInfoName, parseInstanceLogName, parseRedditEntity,
     parseSubredditLogName, permissions,
-    randomId, sleep, triggeredIndicator
+    randomId, replaceApplicationIdentifier, sleep, triggeredIndicator
 } from "../../util";
 import {Cache} from "cache-manager";
 import session, {Session, SessionData} from "express-session";
@@ -24,7 +24,6 @@ import EventEmitter from "events";
 import stream, {Readable, Writable, Transform} from "stream";
 import winston from "winston";
 import tcpUsed from "tcp-port-used";
-import SimpleError from "../../Utils/SimpleError";
 import http from "http";
 import jwt from 'jsonwebtoken';
 import {Server as SocketServer} from "socket.io";
@@ -40,12 +39,18 @@ import DelimiterStream from 'delimiter-stream';
 import {pipeline} from 'stream/promises';
 import {defaultBotStatus} from "../Common/defaults";
 import {arrayMiddle, booleanMiddle} from "../Common/middleware";
-import {BotInstance, CMInstance} from "../interfaces";
+import {BotInstance, CMInstanceInterface} from "../interfaces";
 import { URL } from "url";
 import {MESSAGE} from "triple-beam";
 import Autolinker from "autolinker";
 import path from "path";
 import {ExtendedSnoowrap} from "../../Utils/SnoowrapClients";
+import ClientUser from "../Common/User/ClientUser";
+import {BotStatusResponse} from "../Common/interfaces";
+import {TransformableInfo} from "logform";
+import {SimpleError} from "../../Utils/Errors";
+import {ErrorWithCause} from "pony-cause";
+import {CMInstance} from "./CMInstance";
 
 const emitter = new EventEmitter();
 
@@ -89,19 +94,6 @@ declare module 'express-session' {
     }
 }
 
-// declare global {
-//     namespace Express {
-//         interface User {
-//             name: string
-//             subreddits: string[]
-//             machine?: boolean
-//             isOperator?: boolean
-//             realManagers?: string[]
-//             moderatedManagers?: string[]
-//         }
-//     }
-// }
-
 interface ConnectedUserInfo {
     level?: string,
     user?: string,
@@ -115,7 +107,7 @@ interface ConnectUserObj {
     [key: string]: ConnectedUserInfo
 }
 
-const createToken = (bot: CMInstance, user?: Express.User | any, ) => {
+const createToken = (bot: CMInstanceInterface, user?: Express.User | any, ) => {
     const payload = user !== undefined ? {...user, machine: false} : {machine: true};
     return jwt.sign({
         data: payload,
@@ -126,14 +118,13 @@ const createToken = (bot: CMInstance, user?: Express.User | any, ) => {
 
 const availableLevels = ['error', 'warn', 'info', 'verbose', 'debug'];
 
-const instanceLogMap: Map<string, LogEntry[]> = new Map();
-
 const webClient = async (options: OperatorConfig) => {
     const {
         operator: {
             name,
             display,
         },
+        userAgent: uaFragment,
         web: {
             port,
             caching,
@@ -158,22 +149,18 @@ const webClient = async (options: OperatorConfig) => {
         },
     } = options;
 
+    const userAgent = getUserAgent(`web:contextBot:{VERSION}{FRAG}:dashboard`, uaFragment);
+
+    app.use((req, res, next) => {
+        res.locals.applicationIdentifier = replaceApplicationIdentifier('{VERSION}{FRAG}', uaFragment);
+        next();
+    });
+
     const webOps = operators.map(x => x.toLowerCase());
 
     const logger = getLogger({defaultLabel: 'Web', ...options.logging}, 'Web');
 
     logger.stream().on('log', (log: LogInfo) => {
-        const logEntry: LogEntry = [dayjs(log.timestamp).unix(), log];
-        const {instance: instanceLogName} = log;
-        if (instanceLogName !== undefined) {
-            const subLogs = instanceLogMap.get(instanceLogName) || [];
-            subLogs.unshift(logEntry);
-            instanceLogMap.set(instanceLogName, subLogs.slice(0, 200 + 1));
-        } else {
-            const appLogs = instanceLogMap.get('web') || [];
-            appLogs.unshift(logEntry);
-            instanceLogMap.set('web', appLogs.slice(0, 200 + 1));
-        }
         emitter.emit('log', log[MESSAGE]);
     });
 
@@ -202,8 +189,9 @@ const webClient = async (options: OperatorConfig) => {
         done(null, { subreddits: subreddits.map((x: Subreddit) => x.display_name), isOperator: webOps.includes(user.toLowerCase()), name: user, scope, token, tokenExpiresAt: dayjs().unix() + (60 * 60) });
     });
 
-    passport.deserializeUser(async function (obj, done) {
-        done(null, obj as Express.User);
+    passport.deserializeUser(async function (obj: any, done) {
+        const user = new ClientUser(obj.name, obj.subreddits, {token: obj.token, scope: obj.scope, webOperator: obj.isOperator, tokenExpiresAt: obj.tokenExpiresAt});
+        done(null, user);
         // const data = await webCache.get(`userSession-${obj}`) as object;
         // if (data === undefined) {
         //     done('Not Found');
@@ -236,7 +224,10 @@ const webClient = async (options: OperatorConfig) => {
                 code: code as string,
             });
             const user = await client.getMe().name as string;
-            const subs = await client.getModeratedSubreddits();
+            let subs = await client.getModeratedSubreddits({count: 100});
+            while(!subs.isFinished) {
+                subs = await subs.fetchMore({amount: 100});
+            }
             io.to(req.session.id).emit('authStatus', {canSaveWiki: req.session.scope?.includes('wikiedit')});
             return done(null, {user, subreddits: subs, scope: req.session.scope, token: client.accessToken});
         }
@@ -326,32 +317,36 @@ const webClient = async (options: OperatorConfig) => {
             });
             // @ts-ignore
             const user = await client.getMe();
+            const userName = `u/${user.name}`;
             // @ts-ignore
             await webCache.del(`invite:${req.session.inviteId}`);
             let data: any = {
                 accessToken: client.accessToken,
                 refreshToken: client.refreshToken,
+                userName,
             };
             if(invite.instance !== undefined) {
-                const bot = cmInstances.find(x => x.friendly === invite.instance);
+                const bot = cmInstances.find(x => x.getName() === invite.instance);
                 if(bot !== undefined) {
                     const botPayload: any = {
+                        overwrite: invite.overwrite === true,
+                        name: userName,
                         credentials: {
-                            accessToken: client.accessToken,
-                            refreshToken: client.refreshToken,
-                            clientId: invite.clientId,
-                            clientSecret: invite.clientSecret,
+                            reddit: {
+                                accessToken: client.accessToken,
+                                refreshToken: client.refreshToken,
+                                clientId: invite.clientId,
+                                clientSecret: invite.clientSecret,
+                            }
                         }
                     };
-                    if(invite.subreddit !== undefined) {
-                        botPayload.subreddits =  {names: [invite.subreddit]};
+                    if(invite.subreddits !== undefined && invite.subreddits.length > 0) {
+                        botPayload.subreddits =  {names: invite.subreddits};
                     }
                     const botAddResult: any = await addBot(bot, {name: invite.creator}, botPayload);
-                    let msg = botAddResult.success ? 'Bot successfully added to running instance' : 'An error occurred while adding the bot to the instance';
-                    if(botAddResult.success) {
-                        msg = `${msg}. ${botAddResult.stored === false ? 'Additionally, the bot was not stored in config so the operator will need to add it manually to persist after a restart.' : ''}`;
-                    }
-                    data.addResult = msg;
+                    // stored
+                    // success
+                    data = {...data, ...botAddResult};
                     // @ts-ignore
                     req.session.destroy();
                     req.logout();
@@ -396,12 +391,13 @@ const webClient = async (options: OperatorConfig) => {
     let token = randomId();
     interface InviteData {
         permissions: string[],
-        subreddit?: string,
+        subreddits?: string,
         instance?: string,
         clientId: string
         clientSecret: string
         redirectUri: string
         creator: string
+        overwrite?: boolean
     }
 
     const helperAuthed = async (req: express.Request, res: express.Response, next: Function) => {
@@ -417,7 +413,7 @@ const webClient = async (options: OperatorConfig) => {
                     '<div>or as an argument: <span class="font-mono">--operator YourRedditUsername</span></div>'});
         }
         // or if there is an operator and current user is operator
-        if(req.user.isOperator) {
+        if(req.user?.clientData?.webOperator) {
             return next();
         } else {
             return res.render('error', {error: 'You must be an <b>Operator</b> to access this route.'});
@@ -429,7 +425,8 @@ const webClient = async (options: OperatorConfig) => {
             redirectUri,
             clientId,
             clientSecret,
-            token: req.isAuthenticated() && req.user.isOperator ? token : undefined
+            token: req.isAuthenticated() && req.user?.clientData?.webOperator ? token : undefined,
+            instances: cmInstances.filter(x => req.user?.isInstanceOperator(x)).map(x => x.getName()),
         });
     });
 
@@ -457,7 +454,7 @@ const webClient = async (options: OperatorConfig) => {
             clientSecret: ce,
             redirect: redir,
             instance,
-            subreddit,
+            subreddits,
             code,
         } = req.body as any;
 
@@ -482,7 +479,7 @@ const webClient = async (options: OperatorConfig) => {
             clientSecret: (ce || clientSecret).trim(),
             redirectUri: redir.trim(),
             instance,
-            subreddit,
+            subreddits: subreddits.trim() === '' ? [] : subreddits.split(',').map((x: string) => parseRedditEntity(x).name),
             creator: (req.user as Express.User).name,
         }, {ttl: invitesMaxAge * 1000});
         return res.send(inviteId);
@@ -527,6 +524,8 @@ const webClient = async (options: OperatorConfig) => {
 
     const cmInstances: CMInstance[] = [];
     let init = false;
+    const formatter = defaultFormat();
+    const formatTransform = formatter.transform as (info: TransformableInfo, opts?: any) => TransformableInfo;
 
     let server: http.Server,
         io: SocketServer;
@@ -541,7 +540,7 @@ const webClient = async (options: OperatorConfig) => {
                 delimiter: '\r\n',
             });
 
-            const currInstance = cmInstances.find(x => x.friendly === sessionData.botId);
+            const currInstance = cmInstances.find(x => x.getName() === sessionData.botId);
             if(currInstance !== undefined) {
                 const ac = new AbortController();
                 const options = {
@@ -554,7 +553,8 @@ const webClient = async (options: OperatorConfig) => {
                     });
 
                     if(err !== undefined) {
-                        logger.warn(`Log streaming encountered an error, trying to reconnect (retries: ${retryCount}) -- ${err.code !== undefined ? `(${err.code}) ` : ''}${err.message}`, {instance: currInstance.friendly});
+                        // @ts-ignore
+                        currInstance.logger.warn(new ErrorWithCause(`Log streaming encountered an error, trying to reconnect (retries: ${retryCount})`, {cause: err}), {user: user.name});
                     }
                     const gotStream = got.stream.get(`${currInstance.normalUrl}/logs`, {
                         retry: {
@@ -567,13 +567,15 @@ const webClient = async (options: OperatorConfig) => {
                             limit: sessionData.limit,
                             sort: sessionData.sort,
                             level: sessionData.level,
-                            stream: true
+                            stream: true,
+                            streamObjects: true,
+                            formatted: false,
                         }
                     });
 
                     if(err !== undefined) {
                         gotStream.once('data', () => {
-                            logger.info('Streaming resumed', {subreddit: currInstance.friendly});
+                            currInstance.logger.info('Streaming resumed', {instance: currInstance.getName(), user: user.name});
                         });
                     }
 
@@ -587,12 +589,29 @@ const webClient = async (options: OperatorConfig) => {
                     // ECONNRESET
                     s.catch((err) => {
                         if(err.code !== 'ABORT_ERR' && err.code !== 'ERR_STREAM_PREMATURE_CLOSE') {
-                            logger.error(`Unexpected error, or too many retries, occurred while streaming logs -- ${err.code !== undefined ? `(${err.code}) ` : ''}${err.message}`, {instance: currInstance.friendly});
+                            // @ts-ignore
+                            currInstance.logger.error(new ErrorWithCause('Unexpected error, or too many retries, occurred while streaming logs', {cause: err}), {user: user.name});
                         }
                     });
 
+
                     delim.on('data', (c: any) => {
-                        io.to(sessionId).emit('log', formatLogLineToHtml(c.toString()));
+                        const logObj = JSON.parse(c) as LogInfo;
+                        let subredditMessage;
+                        let allMessage;
+                        if(logObj.subreddit !== undefined) {
+                            const {subreddit, bot, ...rest} = logObj
+                            // @ts-ignore
+                            subredditMessage = formatLogLineToHtml(formatter.transform(rest)[MESSAGE], rest.timestamp);
+                        }
+                        if(logObj.bot !== undefined) {
+                            const {bot, ...rest} = logObj
+                            // @ts-ignore
+                            allMessage = formatLogLineToHtml(formatter.transform(rest)[MESSAGE], rest.timestamp);
+                        }
+                        // @ts-ignore
+                        let formattedMessage = formatLogLineToHtml(formatter.transform(logObj)[MESSAGE], logObj.timestamp);
+                        io.to(sessionId).emit('log', {...logObj, subredditMessage, allMessage, formattedMessage});
                     });
 
                     gotStream.once('retry', retryFn);
@@ -610,9 +629,7 @@ const webClient = async (options: OperatorConfig) => {
         server = await app.listen(port);
         io = new SocketServer(server);
     } catch (err: any) {
-        logger.error('Error occurred while initializing web or socket.io server', err);
-        err.logged = true;
-        throw err;
+        throw new ErrorWithCause('[Web] Error occurred while initializing web or socket.io server', {cause: err});
     }
     logger.info(`Web UI started: http://localhost:${port}`, {label: ['Web']});
 
@@ -621,32 +638,28 @@ const webClient = async (options: OperatorConfig) => {
         delete req.session.authBotId;
 
         const msg = 'Bot does not exist or you do not have permission to access it';
-        const instance = cmInstances.find(x => x.friendly === req.query.instance);
+        const instance = cmInstances.find(x => x.getName() === req.query.instance);
         if (instance === undefined) {
             return res.status(404).render('error', {error: msg});
         }
 
-        const user = req.user as Express.User;
-
-        const isOperator = instance.operators.includes(user.name);
-        const canAccessBot = isOperator || intersect(user.subreddits, instance.subreddits).length > 0;
-        if (!user.isOperator && !canAccessBot) {
+        if (!req.user?.clientData?.webOperator && !req.user?.canAccessInstance(instance)) {
             return res.status(404).render('error', {error: msg});
         }
 
-        if (req.params.subreddit !== undefined && !isOperator && !user.subreddits.includes(req.params.subreddit)) {
+        if (req.params.subreddit !== undefined && !req.user?.isInstanceOperator(instance) && !req.user?.subreddits.includes(req.params.subreddit)) {
             return res.status(404).render('error', {error: msg});
         }
         req.instance = instance;
-        req.session.botId = instance.friendly;
-        if(canAccessBot) {
-            req.session.authBotId = instance.friendly;
+        req.session.botId = instance.getName();
+        if(req.user?.canAccessInstance(instance)) {
+            req.session.authBotId = instance.getName();
         }
         return next();
     }
 
 
-    const botWithPermissions = async (req: express.Request, res: express.Response, next: Function) => {
+    const botWithPermissions = (required: boolean = false, setDefault: boolean = false) => async (req: express.Request, res: express.Response, next: Function) => {
 
         const instance = req.instance;
         if(instance === undefined) {
@@ -655,32 +668,42 @@ const webClient = async (options: OperatorConfig) => {
 
         const msg = 'Bot does not exist or you do not have permission to access it';
         const botVal = req.query.bot as string;
-        if(botVal === undefined) {
+        if(botVal === undefined && required) {
             return res.status(400).render('error', {error: `"bot" param must be defined`});
         }
 
-        const botInstance = instance.bots.find(x => x.botName === botVal);
-        if(botInstance === undefined) {
-            return res.status(404).render('error', {error: msg});
+        if(botVal !== undefined || setDefault) {
+
+            let botInstance;
+            if(botVal === undefined) {
+                // find a bot they can access
+                botInstance = instance.bots.find(x => req.user?.canAccessBot(x));
+                if(botInstance !== undefined) {
+                    req.query.bot = botInstance.botName;
+                }
+            } else {
+                botInstance = instance.bots.find(x => x.botName === botVal);
+            }
+
+            if(botInstance === undefined) {
+                return res.status(404).render('error', {error: msg});
+            }
+
+            if (!req.user?.clientData?.webOperator && !req.user?.canAccessBot(botInstance)) {
+                return res.status(404).render('error', {error: msg});
+            }
+
+            if (req.params.subreddit !== undefined && !req.user?.isInstanceOperator(instance) && !req.user?.subreddits.includes(req.params.subreddit)) {
+                return res.status(404).render('error', {error: msg});
+            }
+            req.bot = botInstance;
         }
 
-        const user = req.user as Express.User;
-
-        const isOperator = instance.operators.includes(user.name);
-        const canAccessBot = isOperator || intersect(user.subreddits, botInstance.subreddits.map(x => x.replace(/\\*r\/*/,''))).length > 0;
-        if (!user.isOperator && !canAccessBot) {
-            return res.status(404).render('error', {error: msg});
-        }
-
-        if (req.params.subreddit !== undefined && !isOperator && !user.subreddits.includes(req.params.subreddit)) {
-            return res.status(404).render('error', {error: msg});
-        }
-        req.bot = botInstance;
         next();
     }
 
     const createUserToken = async (req: express.Request, res: express.Response, next: Function) => {
-        req.token = createToken(req.instance as CMInstance, req.user);
+        req.token = createToken(req.instance as CMInstanceInterface, req.user);
         next();
     }
 
@@ -704,10 +727,10 @@ const webClient = async (options: OperatorConfig) => {
     // botUserRouter.use([ensureAuthenticated, defaultSession, botWithPermissions, createUserToken]);
     // app.use(botUserRouter);
 
-    app.useAsync('/api/', [ensureAuthenticated, defaultSession, instanceWithPermissions, botWithPermissions, createUserToken], (req: express.Request, res: express.Response) => {
+    app.useAsync('/api/', [ensureAuthenticated, defaultSession, instanceWithPermissions, botWithPermissions(true), createUserToken], (req: express.Request, res: express.Response) => {
         req.headers.Authorization = `Bearer ${req.token}`
 
-        const instance = req.instance as CMInstance;
+        const instance = req.instance as CMInstanceInterface;
         return proxy.web(req, res, {
             target: {
                 protocol: instance.url.protocol,
@@ -733,16 +756,25 @@ const webClient = async (options: OperatorConfig) => {
             });
 
             if(accessibleInstance === undefined) {
+                logger.warn(`User ${user.name} is not an operator and has no subreddits in common with any *running* bot instances. If you are sure they should have common subreddits then this client may not be able to access all defined CM servers or the bot may be offline.`, {user: user.name});
                 return res.render('noAccess');
             }
 
             return res.redirect(`/?instance=${accessibleInstance.friendly}`);
         }
-        const instance = cmInstances.find(x => x.friendly === req.query.instance);
+        const instance = cmInstances.find(x => x.getName() === req.query.instance);
         req.instance = instance;
         next();
     }
 
+    const defaultSubreddit = async (req: express.Request, res: express.Response, next: Function) => {
+        if(req.bot !== undefined && req.query.subreddit === undefined) {
+            const firstAccessibleSub = req.bot.subreddits.find(x => req.user?.isInstanceOperator(req.instance) || req.user?.subreddits.includes(x));
+            req.query.subreddit = firstAccessibleSub;
+        }
+        next();
+    }
+
     const initHeartbeat = async (req: express.Request, res: express.Response, next: Function) => {
         if(!init) {
             for(const c of clients) {
@@ -762,7 +794,7 @@ const webClient = async (options: OperatorConfig) => {
         next();
     }
 
-    app.getAsync('/', [initHeartbeat, redirectBotsNotAuthed, ensureAuthenticated, defaultSession, defaultInstance, instanceWithPermissions, createUserToken], async (req: express.Request, res: express.Response) => {
+    app.getAsync('/', [initHeartbeat, redirectBotsNotAuthed, ensureAuthenticated, defaultSession, defaultInstance, instanceWithPermissions, botWithPermissions(false, true), createUserToken], async (req: express.Request, res: express.Response) => {
 
         const user = req.user as Express.User;
         const instance = req.instance as CMInstance;
@@ -772,16 +804,16 @@ const webClient = async (options: OperatorConfig) => {
         const level = req.session.level;
 
         const shownInstances = cmInstances.reduce((acc: CMInstance[], curr) => {
-            const isBotOperator = curr.operators.map(x => x.toLowerCase()).includes(user.name.toLowerCase());
-            if(user.isOperator) {
+            const isBotOperator = req.user?.isInstanceOperator(curr);
+            if(user?.clientData?.webOperator) {
                 // @ts-ignore
-                return acc.concat({...curr, canAccessLocation: true, isOperator: isBotOperator});
+                return acc.concat({...curr.getData(), canAccessLocation: true, isOperator: isBotOperator});
             }
-            if(!isBotOperator && intersect(user.subreddits, curr.subreddits).length === 0) {
+            if(!isBotOperator && !req.user?.canAccessInstance(curr)) {
                 return acc;
             }
             // @ts-ignore
-            return acc.concat({...curr, canAccessLocation: isBotOperator, isOperator: isBotOperator, botId: curr.friendly});
+            return acc.concat({...curr.getData(), canAccessLocation: isBotOperator, isOperator: isBotOperator, botId: curr.getName()});
         },[]);
 
         let resp;
@@ -791,6 +823,8 @@ const webClient = async (options: OperatorConfig) => {
                     'Authorization': `Bearer ${req.token}`,
                 },
                 searchParams: {
+                    bot: req.query.bot as (string | undefined),
+                    subreddit: req.query.sub as (string | undefined) ?? 'all',
                     limit,
                     sort,
                     level,
@@ -799,14 +833,15 @@ const webClient = async (options: OperatorConfig) => {
             }).json() as any;
 
         } catch(err: any) {
-            logger.error(`Error occurred while retrieving bot information. Will update heartbeat -- ${err.message}`, {instance: instance.friendly});
-            refreshClient(clients.find(x => normalizeUrl(x.host) === instance.normalUrl) as BotConnection);
+            instance.logger.error(new ErrorWithCause(`Could not retrieve instance information. Will attempted to update heartbeat.`, {cause: err}));
+            refreshClient({host: instance.host, secret: instance.secret});
+            const isOp = req.user?.isInstanceOperator(instance);
             return res.render('offline', {
                 instances: shownInstances,
-                instanceId: (req.instance as CMInstance).friendly,
-                isOperator: instance.operators.includes((req.user as Express.User).name),
+                instanceId: (req.instance as CMInstance).getName(),
+                isOperator: isOp,
                 // @ts-ignore
-                logs: filterLogBySubreddit(instanceLogMap, [instance.friendly], {limit, sort, level, allLogName: 'web', allLogsParser: parseInstanceLogInfoName }).get(instance.friendly),
+                logs: filterLogs((isOp ? instance.logs : instance.logs.filter(x => x.user === undefined || x.user.includes(req.user.name))), {limit, sort, level}),
                 logSettings: {
                     limitSelect: [10, 20, 50, 100, 200].map(x => `<option ${limit === x ? 'selected' : ''} class="capitalize ${limit === x ? 'font-bold' : ''}" data-value="${x}">${x}</option>`).join(' | '),
                     sortSelect: ['ascending', 'descending'].map(x => `<option ${sort === x ? 'selected' : ''} class="capitalize ${sort === x ? 'font-bold' : ''}" data-value="${x}">${x}</option>`).join(' '),
@@ -832,12 +867,39 @@ const webClient = async (options: OperatorConfig) => {
         //     return acc.concat({...curr, isOperator: instanceOperator});
         // },[]);
 
+        const isOp = req.user?.isInstanceOperator(instance);
+
         res.render('status', {
             instances: shownInstances,
-            bots: resp.bots,
-            botId: (req.instance as CMInstance).friendly,
-            instanceId: (req.instance as CMInstance).friendly,
-            isOperator: instance.operators.includes((req.user as Express.User).name),
+            bots: resp.bots.map((x: BotStatusResponse) => {
+                const {subreddits = []} = x;
+                const subredditsWithSimpleLogs = subreddits.map(y => {
+                    let transformedLogs: string[];
+                    if(y.name === 'All') {
+                        // only need to remove bot name here
+                        transformedLogs = (y.logs as LogInfo[]).map((z: LogInfo) => {
+                           const {bot, ...rest} = z;
+                           // @ts-ignore
+                           return formatLogLineToHtml(formatter.transform(rest)[MESSAGE] as string, rest.timestamp);
+                        });
+                    } else {
+                        transformedLogs = (y.logs as LogInfo[]).map((z: LogInfo) => {
+                            const {bot, subreddit, ...rest} = z;
+                            // @ts-ignore
+                            return formatLogLineToHtml(formatter.transform(rest)[MESSAGE] as string, rest.timestamp);
+                        });
+                    }
+                    y.logs = transformedLogs;
+                    return y;
+                });
+                return {...x, subreddits: subredditsWithSimpleLogs};
+            }),
+            botId: (req.instance as CMInstanceInterface).friendly,
+            instanceId: (req.instance as CMInstanceInterface).friendly,
+            isOperator: isOp,
+            system: isOp ? {
+                logs: resp.system.logs,
+                } : undefined,
             operators: instance.operators.join(', '),
             operatorDisplay: instance.operatorDisplay,
             logSettings: {
@@ -859,11 +921,11 @@ const webClient = async (options: OperatorConfig) => {
         res.render('config', {
             title: `Configuration Editor`,
             format,
-            canSave: req.user?.scope?.includes('wikiedit') && req.user?.tokenExpiresAt !== undefined && dayjs.unix(req.user?.tokenExpiresAt).isAfter(dayjs())
+            canSave: req.user?.clientData?.scope?.includes('wikiedit') && req.user?.clientData?.tokenExpiresAt !== undefined && dayjs.unix(req.user?.clientData.tokenExpiresAt).isAfter(dayjs())
         });
     });
 
-    app.postAsync('/config', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions], async (req: express.Request, res: express.Response) => {
+    app.postAsync('/config', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions(true)], async (req: express.Request, res: express.Response) => {
         const {subreddit} = req.query as any;
         const {location, data, create = false} = req.body as any;
 
@@ -871,7 +933,7 @@ const webClient = async (options: OperatorConfig) => {
             userAgent,
             clientId,
             clientSecret,
-            accessToken: req.user?.token
+            accessToken: req.user?.clientData?.token
         });
 
         try {
@@ -904,9 +966,9 @@ const webClient = async (options: OperatorConfig) => {
         return res.send();
     });
 
-    app.getAsync('/events', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions, createUserToken], async (req: express.Request, res: express.Response) => {
+    app.getAsync('/events', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions(true), createUserToken], async (req: express.Request, res: express.Response) => {
         const {subreddit} = req.query as any;
-        const resp = await got.get(`${(req.instance as CMInstance).normalUrl}/events`, {
+        const resp = await got.get(`${(req.instance as CMInstanceInterface).normalUrl}/events`, {
             headers: {
                 'Authorization': `Bearer ${req.token}`,
             },
@@ -1017,6 +1079,9 @@ const webClient = async (options: OperatorConfig) => {
         const session = socket.handshake.session as (Session & Partial<SessionData> | undefined);
         // @ts-ignore
         const user = session !== undefined ? session?.passport?.user as Express.User : undefined;
+
+        let liveInterval: any = undefined;
+
         if (session !== undefined && user !== undefined) {
             clearSockStreams(socket.id);
             socket.join(session.id);
@@ -1024,15 +1089,51 @@ const webClient = async (options: OperatorConfig) => {
             // setup general web log event
             const webLogListener = (log: string) => {
                 const subName = parseSubredditLogName(log);
-                if((subName === undefined || user.isOperator) && isLogLineMinLevel(log, session.level as string)) {
+                if((subName === undefined || user.clientData?.webOperator === true) && isLogLineMinLevel(log, session.level as string)) {
                     io.to(session.id).emit('webLog', formatLogLineToHtml(log));
                 }
             }
             emitter.on('log', webLogListener);
             socketListeners.set(socket.id, [...(socketListeners.get(socket.id) || []), webLogListener]);
 
+            socket.on('viewing', (data) => {
+                if(user !== undefined) {
+                    const {subreddit, bot: botVal} = data;
+                    const currBot = cmInstances.find(x => x.getName() === session.botId);
+                    if(currBot !== undefined) {
+
+                        if(liveInterval !== undefined) {
+                            clearInterval(liveInterval)
+                        }
+
+                        const liveEmit = async () => {
+                            try {
+                                const resp = await got.get(`${currBot.normalUrl}/liveStats`, {
+                                    headers: {
+                                        'Authorization': `Bearer ${createToken(currBot, user)}`,
+                                    },
+                                    searchParams: {
+                                        bot: botVal,
+                                        subreddit
+                                    }
+                                });
+                                const stats = JSON.parse(resp.body);
+                                io.to(session.id).emit('liveStats', stats);
+                            } catch (err: any) {
+                                currBot.logger.error(new ErrorWithCause('Could not retrieve live stats', {cause: err}));
+                            }
+                        }
+
+                        // do an initial get
+                        liveEmit();
+                        // and then every 5 seconds after that
+                        liveInterval = setInterval(async () => await liveEmit(), 5000);
+                    }
+                }
+            });
+
             if(session.botId !== undefined) {
-                const bot = cmInstances.find(x => x.friendly === session.botId);
+                const bot = cmInstances.find(x => x.getName() === session.botId);
                 if(bot !== undefined) {
                     // web log listener for bot specifically
                     const botWebLogListener = (log: string) => {
@@ -1062,7 +1163,7 @@ const webClient = async (options: OperatorConfig) => {
                                 }).json() as object;
                                 io.to(session.id).emit('opStats', resp);
                             } catch (err: any) {
-                                logger.error(`Could not retrieve stats ${err.message}`, {instance: bot.friendly});
+                                bot.logger.error(new ErrorWithCause('Could not retrieve stats', {cause: err}));
                                 clearInterval(interval);
                             }
                         }, 5000);
@@ -1075,12 +1176,12 @@ const webClient = async (options: OperatorConfig) => {
         socket.on('disconnect', (reason) => {
             clearSockStreams(socket.id);
             clearSockListeners(socket.id);
+            clearInterval(liveInterval);
         });
     });
 
     const loopHeartbeat = async () => {
         while(true) {
-            logger.debug('Starting heartbeat check');
             for(const c of clients) {
                 await refreshClient(c);
             }
@@ -1089,12 +1190,13 @@ const webClient = async (options: OperatorConfig) => {
         }
     }
 
-    const addBot = async (bot: CMInstance, userPayload: any, botPayload: any) => {
+    const addBot = async (bot: CMInstanceInterface, userPayload: any, botPayload: any) => {
         try {
             const token = createToken(bot, userPayload);
             const resp = await got.post(`${bot.normalUrl}/bot`, {
-                body: botPayload,
+                body: JSON.stringify(botPayload),
                 headers: {
+                    'Content-Type': 'application/json',
                     'Authorization': `Bearer ${token}`,
                 }
             }).json() as object;
@@ -1105,79 +1207,13 @@ const webClient = async (options: OperatorConfig) => {
     }
 
     const refreshClient = async (client: BotConnection, force = false) => {
-        const normalized = normalizeUrl(client.host);
-        const existingClientIndex = cmInstances.findIndex(x => x.normalUrl === normalized);
-        const existingClient = existingClientIndex === -1 ? undefined : cmInstances[existingClientIndex];
+        const existingClientIndex = cmInstances.findIndex(x => x.matchesHost(client.host));
+        const instance = existingClientIndex === -1 ? new CMInstance(client, logger) : cmInstances[existingClientIndex];
 
-        let shouldCheck = false;
-        if(!existingClient) {
-            shouldCheck = true;
-        } else if(force) {
-            shouldCheck = true;
-        } else  {
-            const lastCheck = dayjs().diff(dayjs.unix(existingClient.lastCheck), 's');
-            if(!existingClient.online) {
-                if(lastCheck > 15) {
-                    shouldCheck = true;
-                }
-            } else if(lastCheck > 300) {
-                shouldCheck = true;
-            }
-        }
-        if(shouldCheck)
-        {
-            const machineToken = jwt.sign({
-                data: {
-                    machine: true,
-                },
-            }, client.secret, {
-                expiresIn: '1m'
-            });
-            //let base = `${c.host}${c.port !== undefined ? `:${c.port}` : ''}`;
-            const normalized = normalizeUrl(client.host);
-            const url = new URL(normalized);
-            let botStat: CMInstance = {
-                ...client,
-                subreddits: [] as string[],
-                operators: [] as string[],
-                operatorDisplay: '',
-                online: false,
-                friendly: url.host,
-                lastCheck: dayjs().unix(),
-                normalUrl: normalized,
-                url,
-                bots: [],
-            };
-            try {
-                const resp = await got.get(`${normalized}/heartbeat`, {
-                    headers: {
-                        'Authorization': `Bearer ${machineToken}`,
-                    }
-                }).json() as CMInstance;
+        await instance.checkHeartbeat(force);
 
-                botStat = {...botStat, ...resp, online: true};
-                const sameNameIndex = cmInstances.findIndex(x => x.friendly === botStat.friendly);
-                if(sameNameIndex > -1 && sameNameIndex !== existingClientIndex) {
-                    logger.warn(`Client returned a friendly name that is not unique (${botStat.friendly}), will fallback to host as friendly (${botStat.normalUrl})`);
-                    botStat.friendly = botStat.normalUrl;
-                }
-                botStat.online = true;
-                // if(botStat.online) {
-                //     botStat.indicator = botStat.running ? 'green' : 'yellow';
-                // } else {
-                //     botStat.indicator = 'red';
-                // }
-                logger.verbose(`Heartbeat detected`, {instance: botStat.friendly});
-            } catch (err: any) {
-                botStat.error = err.message;
-                logger.error(`Heartbeat response from ${botStat.friendly} was not ok: ${err.message}`, {instance: botStat.friendly});
-            } finally {
-                if(existingClientIndex !== -1) {
-                    cmInstances.splice(existingClientIndex, 1, botStat);
-                } else {
-                    cmInstances.push(botStat);
-                }
-            }
+        if(existingClientIndex === -1) {
+            cmInstances.push(instance);
         }
     }
 }

src/Web/Common/User/CMUser.ts

@@ -0,0 +1,23 @@
+import {IUser} from "../interfaces";
+
+export interface ClientUserData {
+    token?: string
+    tokenExpiresAt?: number
+    scope?: string[]
+    webOperator?: boolean
+}
+
+abstract class CMUser<Instance, Bot, SubredditEntity> implements IUser {
+    constructor(public name: string, public subreddits: string[], public clientData: ClientUserData = {}) {
+
+    }
+
+    public abstract isInstanceOperator(val: Instance): boolean;
+    public abstract canAccessInstance(val: Instance): boolean;
+    public abstract canAccessBot(val: Bot): boolean;
+    public abstract accessibleBots(bots: Bot[]): Bot[]
+    public abstract canAccessSubreddit(val: Bot, name: string): boolean;
+    public abstract accessibleSubreddits(bot: Bot): SubredditEntity[]
+}
+
+export default CMUser;

src/Web/Common/User/ClientUser.ts

@@ -0,0 +1,41 @@
+import {BotInstance, CMInstanceInterface} from "../../interfaces";
+import CMUser from "./CMUser";
+import {intersect, parseRedditEntity} from "../../../util";
+
+class ClientUser extends CMUser<CMInstanceInterface, BotInstance, string> {
+
+    isInstanceOperator(val: CMInstanceInterface): boolean {
+        return val.operators.map(x=> x.toLowerCase()).includes(this.name.toLowerCase());
+    }
+
+    canAccessInstance(val: CMInstanceInterface): boolean {
+        return this.isInstanceOperator(val) || intersect(this.subreddits, val.subreddits.map(x => parseRedditEntity(x).name)).length > 0;
+    }
+
+    canAccessBot(val: BotInstance): boolean {
+        return this.isInstanceOperator(val.instance) || intersect(this.subreddits, val.subreddits.map(x => parseRedditEntity(x).name)).length > 0;
+    }
+
+    canAccessSubreddit(val: BotInstance, name: string): boolean {
+        return this.isInstanceOperator(val.instance) || this.subreddits.map(x => x.toLowerCase()).includes(parseRedditEntity(name).name.toLowerCase());
+    }
+
+    accessibleBots(bots: BotInstance[]): BotInstance[] {
+        if (bots.length === 0) {
+            return bots;
+        }
+        return bots.filter(x => {
+            if (this.isInstanceOperator(x.instance)) {
+                return true;
+            }
+            return intersect(this.subreddits, x.subreddits.map(y => parseRedditEntity(y).name)).length > 0
+        });
+    }
+
+    accessibleSubreddits(bot: BotInstance): string[] {
+        return this.isInstanceOperator(bot.instance) ? bot.subreddits.map(x => parseRedditEntity(x).name) : intersect(this.subreddits, bot.subreddits.map(x => parseRedditEntity(x).name));
+    }
+
+}
+
+export default ClientUser;

src/Web/Common/User/ServerUser.ts

@@ -0,0 +1,39 @@
+import {BotInstance, CMInstanceInterface} from "../../interfaces";
+import CMUser from "./CMUser";
+import {intersect, parseRedditEntity} from "../../../util";
+import {App} from "../../../App";
+import Bot from "../../../Bot";
+import {Manager} from "../../../Subreddit/Manager";
+
+class ServerUser extends CMUser<App, Bot, Manager> {
+
+    constructor(public name: string, public subreddits: string[], public machine: boolean, public isOperator: boolean) {
+        super(name, subreddits);
+    }
+
+    isInstanceOperator(): boolean {
+        return this.isOperator;
+    }
+
+    canAccessInstance(val: App): boolean {
+        return this.isOperator || val.bots.filter(x => intersect(this.subreddits, x.subManagers.map(y => y.subreddit.display_name))).length > 0;
+    }
+
+    canAccessBot(val: Bot): boolean {
+        return this.isOperator || intersect(this.subreddits, val.subManagers.map(y => y.subreddit.display_name)).length > 0;
+    }
+
+    accessibleBots(bots: Bot[]): Bot[] {
+        return this.isOperator ? bots : bots.filter(x => intersect(this.subreddits, x.subManagers.map(y => y.subreddit.display_name)).length > 0);
+    }
+
+    canAccessSubreddit(val: Bot, name: string): boolean {
+        return this.isOperator || this.subreddits.includes(parseRedditEntity(name).name) && val.subManagers.some(y => y.subreddit.display_name.toLowerCase() === parseRedditEntity(name).name.toLowerCase());
+    }
+
+    accessibleSubreddits(bot: Bot): Manager[] {
+        return this.isOperator ? bot.subManagers : bot.subManagers.filter(x => intersect(this.subreddits, [x.subreddit.display_name]).length > 0);
+    }
+}
+
+export default ServerUser;

src/Web/Common/interfaces.ts

@@ -1,5 +1,6 @@
 import {RunningState} from "../../Subreddit/Manager";
-import {ManagerStats} from "../../Common/interfaces";
+import {LogInfo, ManagerStats} from "../../Common/interfaces";
+import {BotInstance} from "../interfaces";
 
 export interface BotStats {
     startedAtHuman: string,
@@ -15,7 +16,7 @@ export interface BotStats {
 
 export interface SubredditDataResponse {
     name: string
-    logs: string[]
+    logs: (string|LogInfo)[]
     botState: RunningState
     eventsState: RunningState
     queueState: RunningState
@@ -59,3 +60,25 @@ export interface BotStatusResponse {
     }
     subreddits: SubredditDataResponse[]
 }
+
+export interface IUser {
+    name: string
+    subreddits: string[]
+    machine?: boolean
+    isOperator?: boolean
+    realManagers?: string[]
+    moderatedManagers?: string[]
+    realBots?: string[]
+    moderatedBots?: string[]
+    scope?: string[]
+    token?: string
+    tokenExpiresAt?: number
+}
+
+export interface HeartbeatResponse {
+    subreddits: string[]
+    operators: string[]
+    operatorDisplay?: string
+    friendly?: string
+    bots: BotInstance[]
+}

src/Web/Common/util.ts

@@ -5,7 +5,7 @@ import {formatNumber} from "../../util";
 import Bot from "../../Bot";
 
 export const opStats = (bot: Bot): BotStats => {
-    const limitReset = dayjs(bot.client.ratelimitExpiration);
+    const limitReset = bot.client === undefined ? dayjs() : dayjs(bot.client.ratelimitExpiration);
     const nextHeartbeat = bot.nextHeartbeat !== undefined ? bot.nextHeartbeat.local().format('MMMM D, YYYY h:mm A Z') : 'N/A';
     const nextHeartbeatHuman = bot.nextHeartbeat !== undefined ? `in ${dayjs.duration(bot.nextHeartbeat.diff(dayjs())).humanize()}` : 'N/A'
     return {

src/Web/Server/interfaces.ts

@@ -1,27 +0,0 @@
-import { Request } from "express";
-import {App} from "../../App";
-import Bot from "../../Bot";
-
-// export interface ServerRequest extends Request {
-//     botApp: App
-//     bot?: Bot
-//     //user?: AuthenticatedUser
-// }
-//
-// export interface ServerRequestRedditor extends ServerRequest {
-//     user?: AuthenticatedRedditUser
-// }
-//
-// export interface AuthenticatedUser extends Express.User {
-//     machine: boolean
-// }
-//
-// export interface AuthenticatedRedditUser extends AuthenticatedUser {
-//     name: string
-//     subreddits: string[]
-//     isOperator: boolean
-//     realManagers: string[]
-//     moderatedManagers: string[]
-//     realBots: string[]
-//     moderatedBots: string[]
-// }

src/Web/Server/middleware.ts

@@ -1,9 +1,10 @@
-import {Request, Response} from "express";
+import {Request, Response, NextFunction} from "express";
 import Bot from "../../Bot";
+import ServerUser from "../Common/User/ServerUser";
 
 export const authUserCheck = (userRequired: boolean = true) => async (req: Request, res: Response, next: Function) => {
     if (req.isAuthenticated()) {
-        if (userRequired && req.user.machine) {
+        if (userRequired && (req.user as ServerUser).machine) {
             return res.status(403).send('Must be authenticated as a user to access this route');
         }
         return next();
@@ -23,10 +24,15 @@ export const botRoute = (required = true) => async (req: Request, res: Response,
     const botStr = botVal as string;
 
     if(req.user !== undefined) {
-        if (req.user.realBots === undefined || !req.user.realBots.map(x => x.toLowerCase()).includes(botStr.toLowerCase())) {
+        const serverBot = req.botApp.bots.find(x => x.botName === botStr) as Bot;
+
+        if(serverBot === undefined) {
             return res.status(404).send(`Bot named ${botStr} does not exist or you do not have permission to access it.`);
         }
-        req.serverBot = req.botApp.bots.find(x => x.botName === botStr) as Bot;
+        if (!req.user?.canAccessBot(serverBot)) {
+            return res.status(404).send(`Bot named ${botStr} does not exist or you do not have permission to access it.`);
+        }
+        req.serverBot = serverBot;
         return next();
     }
     return next();
@@ -37,20 +43,27 @@ export const subredditRoute = (required = true) => async (req: Request, res: Res
     const bot = req.serverBot;
 
     const {subreddit} = req.query as any;
-    if(subreddit === undefined && required === false) {
+    if(subreddit === undefined && !required) {
         next();
     } else {
-        const {name: userName, realManagers = [], isOperator} = req.user as Express.User;
-        if (!isOperator && !realManagers.includes(subreddit)) {
-            return res.status(400).send('Cannot access route for subreddit you do not manage or is not run by the bot')
-        }
-        const manager = bot.subManagers.find(x => x.displayLabel === subreddit);
-        if (manager === undefined) {
-            return res.status(400).send('Cannot access route for subreddit you do not manage or is not run by the bot')
-        }
 
-        req.manager = manager;
+        if(subreddit.toLowerCase() === 'all') {
+            next();
+        } else {
+            //const {name: userName} = req.user as Express.User;
 
-        next();
+            const manager = bot.subManagers.find(x => x.displayLabel === subreddit);
+            if (manager === undefined) {
+                return res.status(400).send('Cannot access route for subreddit you do not manage or is not run by the bot')
+            }
+
+            if (!req.user?.canAccessSubreddit(bot, subreddit)) {
+                return res.status(400).send('Cannot access route for subreddit you do not manage or is not run by the bot')
+            }
+
+            req.manager = manager;
+
+            next();
+        }
     }
 }

src/Web/Server/routes/authenticated/user/action.ts

@@ -1,31 +1,25 @@
-import express, {Request, Response} from 'express';
+import {Request, Response} from 'express';
 import {RUNNING, USER} from "../../../../../Common/interfaces";
 import Submission from "snoowrap/dist/objects/Submission";
 import LoggedError from "../../../../../Utils/LoggedError";
-import winston from "winston";
 import {authUserCheck, botRoute} from "../../../middleware";
 import {booleanMiddle} from "../../../../Common/middleware";
+import {Manager} from "../../../../../Subreddit/Manager";
+import {parseRedditEntity} from "../../../../../util";
 
-const action = async (req: express.Request, res: express.Response) => {
+const action = async (req: Request, res: Response) => {
     const bot = req.serverBot;
 
     const {type, action, subreddit, force = false} = req.query as any;
-    const {name: userName, realManagers = [], isOperator} = req.user as Express.User;
-    let subreddits: string[] = [];
-    if (subreddit === 'All') {
-        subreddits = realManagers;
-    } else if (realManagers.includes(subreddit)) {
-        subreddits = [subreddit];
+    const userName = req.user?.name;
+    let subreddits: Manager[] = req.user?.accessibleSubreddits(bot) as Manager[];
+    if (subreddit !== 'All') {
+        subreddits = subreddits.filter(x => x.subreddit.display_name === parseRedditEntity(subreddit).name);
     }
 
-    for (const s of subreddits) {
-        const manager = bot.subManagers.find(x => x.displayLabel === s);
-        if (manager === undefined) {
-            winston.loggers.get('app').warn(`Manager for ${s} does not exist`, {subreddit: `/u/${userName}`});
-            continue;
-        }
+    for (const manager of subreddits) {
         const mLogger = manager.logger;
-        mLogger.info(`/u/${userName} invoked '${action}' action for ${type} on ${manager.displayLabel}`);
+        mLogger.info(`/u/${userName} invoked '${action}' action for ${type} on ${manager.displayLabel}`, {user: userName});
         try {
             switch (action) {
                 case 'start':

src/Web/Server/routes/authenticated/user/addBot.ts

@@ -3,6 +3,8 @@ import {BotInstanceConfig} from "../../../../../Common/interfaces";
 import {authUserCheck} from "../../../middleware";
 import Bot from "../../../../../Bot";
 import LoggedError from "../../../../../Utils/LoggedError";
+import {open} from 'fs/promises';
+import {buildBotConfig} from "../../../../../ConfigBuilder";
 
 const addBot = () => {
 
@@ -12,13 +14,41 @@ const addBot = () => {
 
     const response = async (req: Request, res: Response) => {
 
-        if (!(req.user as Express.User).isOperator) {
+        if (!req.user?.isInstanceOperator(req.app)) {
             return res.status(401).send("Must be an Operator to use this route");
         }
 
-        const newBot = new Bot(req.body as BotInstanceConfig, req.botApp.logger);
+        if (!req.botApp.fileConfig.isWriteable) {
+            return res.status(409).send('Operator config is not writeable');
+        }
+
+        const {overwrite = false, ...botData} = req.body;
+
+        // check if bot is new or overwriting
+        let existingBot = req.botApp.bots.find(x => x.botAccount === botData.name);
+        // spin down existing
+        if (existingBot !== undefined) {
+            const {
+                bots: botsFromConfig = []
+            } = req.botApp.fileConfig.document.toJS();
+            if(botsFromConfig.length === 0 || botsFromConfig.some(x => x.name !== botData.name)) {
+                req.botApp.logger.warn('Overwriting existing bot with the same name BUT this bot does not exist in the operator CONFIG FILE. You should check how you have provided config before next start or else this bot may be started twice (once from file, once from arg/env)');
+
+            }
+
+            await existingBot.destroy('system');
+            req.botApp.bots.filter(x => x.botAccount !== botData.name);
+        }
+
+        req.botApp.fileConfig.document.addBot(botData);
+
+        const handle = await open(req.botApp.fileConfig.document.location as string, 'w');
+        await handle.writeFile(req.botApp.fileConfig.document.toString());
+        await handle.close();
+
+        const newBot = new Bot(buildBotConfig(botData, req.botApp.config), req.botApp.logger);
         req.botApp.bots.push(newBot);
-        let result: any = {stored: true};
+        let result: any = {stored: true, success: true};
         try {
             if (newBot.error !== undefined) {
                 result.error = newBot.error;
@@ -26,13 +56,14 @@ const addBot = () => {
             }
             await newBot.testClient();
             await newBot.buildManagers();
-            newBot.runManagers('user').catch((err) => {
+            newBot.runManagers('system').catch((err) => {
                 req.botApp.logger.error(`Unexpected error occurred while running Bot ${newBot.botName}. Bot must be re-built to restart`);
                 if (!err.logged || !(err instanceof LoggedError)) {
                     req.botApp.logger.error(err);
                 }
             });
         } catch (err: any) {
+            result.success = false;
             if (newBot.error === undefined) {
                 newBot.error = err.message;
                 result.error = err.message;

src/Web/Server/routes/authenticated/user/index.ts

@@ -66,7 +66,7 @@ const actionedEvents = async (req: Request, res: Response) => {
         managers.push(manager);
     } else {
         for(const manager of req.serverBot.subManagers) {
-            if((req.user?.realManagers as string[]).includes(manager.displayLabel)) {
+            if(req.user?.canAccessSubreddit(req.serverBot, manager.subreddit.display_name)) {
                 managers.push(manager);
             }
         }
@@ -89,7 +89,7 @@ const action = async (req: Request, res: Response) => {
     const bot = req.serverBot;
 
     const {url, dryRun = false, subreddit} = req.query as any;
-    const {name: userName, realManagers = [], isOperator} = req.user as Express.User;
+    const {name: userName} = req.user as Express.User;
 
     let a;
     const commentId = commentReg(url);
@@ -106,7 +106,7 @@ const action = async (req: Request, res: Response) => {
     }
 
     if (a === undefined) {
-        winston.loggers.get('app').error('Could not parse Comment or Submission ID from given URL', {subreddit: `/u/${userName}`});
+        winston.loggers.get('app').error('Could not parse Comment or Submission ID from given URL', {user: userName});
         return res.send('OK');
     } else {
         // @ts-ignore
@@ -115,18 +115,18 @@ const action = async (req: Request, res: Response) => {
 
         let manager = subreddit === 'All' ? bot.subManagers.find(x => x.subreddit.display_name === sub) : bot.subManagers.find(x => x.displayLabel === subreddit);
 
-        if (manager === undefined || (!realManagers.includes(manager.displayLabel))) {
+        if (manager === undefined || !req.user?.canAccessSubreddit(req.serverBot, manager.subreddit.display_name)) {
             let msg = 'Activity does not belong to a subreddit you moderate or the bot runs on.';
             if (subreddit === 'All') {
-                msg = `${msg} If you want to test an Activity against a Subreddit\'s config it does not belong to then switch to that Subreddit's tab first.`
+                msg = `${msg} If you want to test an Activity against a Subreddit's config it does not belong to then switch to that Subreddit's tab first.`
             }
-            winston.loggers.get('app').error(msg, {subreddit: `/u/${userName}`});
+            winston.loggers.get('app').error(msg, {user: userName});
             return res.send('OK');
         }
 
         // will run dryrun if specified or if running activity on subreddit it does not belong to
         const dr: boolean | undefined = (dryRun || manager.subreddit.display_name !== sub) ? true : undefined;
-        manager.logger.info(`/u/${userName} running${dr === true ? ' DRY RUN ' : ' '}check on${manager.subreddit.display_name !== sub ? ' FOREIGN ACTIVITY ' : ' '}${url}`);
+        manager.logger.info(`/u/${userName} running${dr === true ? ' DRY RUN ' : ' '}check on${manager.subreddit.display_name !== sub ? ' FOREIGN ACTIVITY ' : ' '}${url}`, {user: userName, subreddit});
         await manager.runChecks(activity instanceof Submission ? 'Submission' : 'Comment', activity, {dryRun: dr, force: true})
     }
     res.send('OK');

src/Web/Server/routes/authenticated/user/liveStats.ts

@@ -0,0 +1,261 @@
+import {authUserCheck, botRoute, subredditRoute} from "../../../middleware";
+import {Request, Response} from "express";
+import Bot from "../../../../../Bot";
+import {boolToString, cacheStats, filterLogs, formatNumber, logSortFunc, pollingInfo} from "../../../../../util";
+import dayjs from "dayjs";
+import {LogInfo, ResourceStats, RUNNING, STOPPED, SYSTEM} from "../../../../../Common/interfaces";
+import {Manager} from "../../../../../Subreddit/Manager";
+import winston from "winston";
+import {opStats} from "../../../../Common/util";
+import {BotStatusResponse} from "../../../../Common/interfaces";
+
+const liveStats = () => {
+    const middleware = [
+        authUserCheck(),
+        botRoute(),
+        subredditRoute(false),
+    ]
+
+    const response = async (req: Request, res: Response) =>
+    {
+        const bot = req.serverBot as Bot;
+        const manager = req.manager;
+        
+        if(manager === undefined) {
+            // getting all
+            const subManagerData: any[] = [];
+            for (const m of req.user?.accessibleSubreddits(bot) as Manager[]) {
+                const sd = {
+                    queuedActivities: m.queue.length(),
+                    runningActivities: m.queue.running(),
+                    maxWorkers: m.queue.concurrency,
+                    subMaxWorkers: m.subMaxWorkers || bot.maxWorkers,
+                    globalMaxWorkers: bot.maxWorkers,
+                    checks: {
+                        submissions: m.submissionChecks === undefined ? 0 : m.submissionChecks.length,
+                        comments: m.commentChecks === undefined ? 0 : m.commentChecks.length,
+                    },
+                    stats: await m.getStats(),
+                }
+            }
+
+            const totalStats = subManagerData.reduce((acc, curr) => {
+                return {
+                    checks: {
+                        submissions: acc.checks.submissions + curr.checks.submissions,
+                        comments: acc.checks.comments + curr.checks.comments,
+                    },
+                    historical: {
+                        allTime: {
+                            eventsCheckedTotal: acc.historical.allTime.eventsCheckedTotal + curr.stats.historical.allTime.eventsCheckedTotal,
+                            eventsActionedTotal: acc.historical.allTime.eventsActionedTotal + curr.stats.historical.allTime.eventsActionedTotal,
+                            checksRunTotal: acc.historical.allTime.checksRunTotal + curr.stats.historical.allTime.checksRunTotal,
+                            checksFromCacheTotal: acc.historical.allTime.checksFromCacheTotal + curr.stats.historical.allTime.checksFromCacheTotal,
+                            checksTriggeredTotal: acc.historical.allTime.checksTriggeredTotal + curr.stats.historical.allTime.checksTriggeredTotal,
+                            rulesRunTotal: acc.historical.allTime.rulesRunTotal + curr.stats.historical.allTime.rulesRunTotal,
+                            rulesCachedTotal: acc.historical.allTime.rulesCachedTotal + curr.stats.historical.allTime.rulesCachedTotal,
+                            rulesTriggeredTotal: acc.historical.allTime.rulesTriggeredTotal + curr.stats.historical.allTime.rulesTriggeredTotal,
+                            actionsRunTotal: acc.historical.allTime.actionsRunTotal + curr.stats.historical.allTime.actionsRunTotal,
+                        }
+                    },
+                    maxWorkers: acc.maxWorkers + curr.maxWorkers,
+                    subMaxWorkers: acc.subMaxWorkers + curr.subMaxWorkers,
+                    globalMaxWorkers: acc.globalMaxWorkers + curr.globalMaxWorkers,
+                    runningActivities: acc.runningActivities + curr.runningActivities,
+                    queuedActivities: acc.queuedActivities + curr.queuedActivities,
+                };
+            }, {
+                checks: {
+                    submissions: 0,
+                    comments: 0,
+                },
+                historical: {
+                    allTime: {
+                        eventsCheckedTotal: 0,
+                        eventsActionedTotal: 0,
+                        checksRunTotal: 0,
+                        checksFromCacheTotal: 0,
+                        checksTriggeredTotal: 0,
+                        rulesRunTotal: 0,
+                        rulesCachedTotal: 0,
+                        rulesTriggeredTotal: 0,
+                        actionsRunTotal: 0,
+                    }
+                },
+                maxWorkers: 0,
+                subMaxWorkers: 0,
+                globalMaxWorkers: 0,
+                runningActivities: 0,
+                queuedActivities: 0,
+            });
+            const {
+                checks,
+                maxWorkers,
+                globalMaxWorkers,
+                subMaxWorkers,
+                runningActivities,
+                queuedActivities,
+                ...rest
+            } = totalStats;
+
+            let cumRaw = subManagerData.reduce((acc, curr) => {
+                Object.keys(curr.stats.cache.types as ResourceStats).forEach((k) => {
+                    acc[k].requests += curr.stats.cache.types[k].requests;
+                    acc[k].miss += curr.stats.cache.types[k].miss;
+                    // @ts-ignore
+                    acc[k].identifierAverageHit += (typeof curr.stats.cache.types[k].identifierAverageHit === 'string' ? Number.parseFloat(curr.stats.cache.types[k].identifierAverageHit) : curr.stats.cache.types[k].identifierAverageHit);
+                    acc[k].averageTimeBetweenHits += curr.stats.cache.types[k].averageTimeBetweenHits === 'N/A' ? 0 : Number.parseFloat(curr.stats.cache.types[k].averageTimeBetweenHits)
+                });
+                return acc;
+            }, cacheStats());
+            cumRaw = Object.keys(cumRaw).reduce((acc, curr) => {
+                const per = acc[curr].miss === 0 ? 0 : formatNumber(acc[curr].miss / acc[curr].requests) * 100;
+                // @ts-ignore
+                acc[curr].missPercent = `${formatNumber(per, {toFixed: 0})}%`;
+                acc[curr].identifierAverageHit = formatNumber(acc[curr].identifierAverageHit);
+                acc[curr].averageTimeBetweenHits = formatNumber(acc[curr].averageTimeBetweenHits)
+                return acc;
+            }, cumRaw);
+            const cacheReq = subManagerData.reduce((acc, curr) => acc + curr.stats.cache.totalRequests, 0);
+            const cacheMiss = subManagerData.reduce((acc, curr) => acc + curr.stats.cache.totalMiss, 0);
+            const sharedSub = subManagerData.find(x => x.stats.cache.isShared);
+            const sharedCount = sharedSub !== undefined ? sharedSub.stats.cache.currentKeyCount : 0;
+            const scopes = req.user?.isInstanceOperator(bot) ? bot.client.scope : [];
+            let allManagerData: any = {
+                name: 'All',
+                status: bot.running ? 'RUNNING' : 'NOT RUNNING',
+                indicator: bot.running ? 'green' : 'grey',
+                maxWorkers,
+                globalMaxWorkers,
+                scopes: scopes === null || !Array.isArray(scopes) ? [] : scopes,
+                subMaxWorkers,
+                runningActivities,
+                queuedActivities,
+                botState: {
+                    state: RUNNING,
+                    causedBy: SYSTEM
+                },
+                dryRun: boolToString(bot.dryRun === true),
+                checks: checks,
+                softLimit: bot.softLimit,
+                hardLimit: bot.hardLimit,
+                stats: {
+                    ...rest,
+                    cache: {
+                        currentKeyCount: sharedCount + subManagerData.reduce((acc, curr) => curr.stats.cache.isShared ? acc : acc + curr.stats.cache.currentKeyCount,0),
+                        isShared: false,
+                        totalRequests: cacheReq,
+                        totalMiss: cacheMiss,
+                        missPercent: `${formatNumber(cacheMiss === 0 || cacheReq === 0 ? 0 : (cacheMiss / cacheReq) * 100, {toFixed: 0})}%`,
+                        types: {
+                            ...cumRaw,
+                        }
+                    }
+                },
+            };
+            // if(isOperator) {
+            allManagerData.startedAt = bot.startedAt.local().format('MMMM D, YYYY h:mm A Z');
+            allManagerData.heartbeatHuman = dayjs.duration({seconds: bot.heartbeatInterval}).humanize();
+            allManagerData.heartbeat = bot.heartbeatInterval;
+            allManagerData = {...allManagerData, ...opStats(bot)};
+            //}
+
+            const botDur = dayjs.duration(dayjs().diff(bot.startedAt))
+            if (allManagerData.stats.cache.totalRequests > 0) {
+                const minutes = botDur.asMinutes();
+                if (minutes < 10) {
+                    allManagerData.stats.cache.requestRate = formatNumber((10 / minutes) * allManagerData.stats.cache.totalRequests, {
+                        toFixed: 0,
+                        round: {enable: true, indicate: true}
+                    });
+                } else {
+                    allManagerData.stats.cache.requestRate = formatNumber(allManagerData.stats.cache.totalRequests / (minutes / 10), {
+                        toFixed: 0,
+                        round: {enable: true, indicate: true}
+                    });
+                }
+            } else {
+                allManagerData.stats.cache.requestRate = 0;
+            }
+
+            const data = {
+                bot: bot.getBotName(),
+                system: {
+                    startedAt: bot.startedAt.local().format('MMMM D, YYYY h:mm A Z'),
+                    running: bot.running,
+                    error: bot.error,
+                    ...opStats(bot),
+                },
+            };
+            return res.json(data);
+        } else {
+            // getting specific subreddit stats
+            const sd = {
+                name: manager.displayLabel,
+                botState: manager.botState,
+                eventsState: manager.eventsState,
+                queueState: manager.queueState,
+                indicator: 'gray',
+                permissions: await manager.getModPermissions(),
+                queuedActivities: manager.queue.length(),
+                runningActivities: manager.queue.running(),
+                maxWorkers: manager.queue.concurrency,
+                subMaxWorkers: manager.subMaxWorkers || bot.maxWorkers,
+                globalMaxWorkers: bot.maxWorkers,
+                validConfig: boolToString(manager.validConfigLoaded),
+                configFormat: manager.wikiFormat,
+                dryRun: boolToString(manager.dryRun === true),
+                pollingInfo: manager.pollOptions.length === 0 ? ['nothing :('] : manager.pollOptions.map(pollingInfo),
+                checks: {
+                    submissions: manager.submissionChecks === undefined ? 0 : manager.submissionChecks.length,
+                    comments: manager.commentChecks === undefined ? 0 : manager.commentChecks.length,
+                },
+                wikiRevisionHuman: manager.lastWikiRevision === undefined ? 'N/A' : `${dayjs.duration(dayjs().diff(manager.lastWikiRevision)).humanize()} ago`,
+                wikiRevision: manager.lastWikiRevision === undefined ? 'N/A' : manager.lastWikiRevision.local().format('MMMM D, YYYY h:mm A Z'),
+                wikiLastCheckHuman: `${dayjs.duration(dayjs().diff(manager.lastWikiCheck)).humanize()} ago`,
+                wikiLastCheck: manager.lastWikiCheck.local().format('MMMM D, YYYY h:mm A Z'),
+                stats: await manager.getStats(),
+                startedAt: 'Not Started',
+                startedAtHuman: 'Not Started',
+                delayBy: manager.delayBy === undefined ? 'No' : `Delayed by ${manager.delayBy} sec`,
+            };
+            // TODO replace indicator data with js on client page
+            let indicator;
+            if (manager.botState.state === RUNNING && manager.queueState.state === RUNNING && manager.eventsState.state === RUNNING) {
+                indicator = 'green';
+            } else if (manager.botState.state === STOPPED && manager.queueState.state === STOPPED && manager.eventsState.state === STOPPED) {
+                indicator = 'red';
+            } else {
+                indicator = 'yellow';
+            }
+            sd.indicator = indicator;
+            if (manager.startedAt !== undefined) {
+                const dur = dayjs.duration(dayjs().diff(manager.startedAt));
+                sd.startedAtHuman = `${dur.humanize()} ago`;
+                sd.startedAt = manager.startedAt.local().format('MMMM D, YYYY h:mm A Z');
+
+                if (sd.stats.cache.totalRequests > 0) {
+                    const minutes = dur.asMinutes();
+                    if (minutes < 10) {
+                        sd.stats.cache.requestRate = formatNumber((10 / minutes) * sd.stats.cache.totalRequests, {
+                            toFixed: 0,
+                            round: {enable: true, indicate: true}
+                        });
+                    } else {
+                        sd.stats.cache.requestRate = formatNumber(sd.stats.cache.totalRequests / (minutes / 10), {
+                            toFixed: 0,
+                            round: {enable: true, indicate: true}
+                        });
+                    }
+                } else {
+                    sd.stats.cache.requestRate = 0;
+                }
+            }
+
+            return res.json(sd);
+        }
+    }
+    return [...middleware, response];
+}
+
+export default liveStats;

src/Web/Server/routes/authenticated/user/logs.ts

@@ -1,19 +1,29 @@
 import {Router} from '@awaitjs/express';
 import {Request, Response} from 'express';
-import {filterLogBySubreddit, isLogLineMinLevel, LogEntry, parseSubredditLogName} from "../../../../../util";
+import {
+    filterLogBySubreddit,
+    filterLogs,
+    isLogLineMinLevel,
+    LogEntry,
+    parseSubredditLogName
+} from "../../../../../util";
 import {Transform} from "stream";
 import winston from "winston";
 import pEvent from "p-event";
 import {getLogger} from "../../../../../Utils/loggerFactory";
 import {booleanMiddle} from "../../../../Common/middleware";
-import {authUserCheck, botRoute} from "../../../middleware";
+import {authUserCheck, botRoute, subredditRoute} from "../../../middleware";
 import {LogInfo} from "../../../../../Common/interfaces";
 import {MESSAGE} from "triple-beam";
+import {Manager} from "../../../../../Subreddit/Manager";
+import Bot from "../../../../../Bot";
 
 // TODO update logs api
-const logs = (subLogMap: Map<string, LogEntry[]>) => {
+const logs = () => {
     const middleware = [
         authUserCheck(),
+        botRoute(false),
+        subredditRoute(false),
         booleanMiddle([{
             name: 'stream',
             defaultVal: false
@@ -24,26 +34,35 @@ const logs = (subLogMap: Map<string, LogEntry[]>) => {
 
         const logger = winston.loggers.get('app');
 
-        const {name: userName, realManagers = [], isOperator} = req.user as Express.User;
-        const {level = 'verbose', stream, limit = 200, sort = 'descending', streamObjects = false} = req.query;
+        const userName = req.user?.name as string;
+        const isOperator = req.user?.isInstanceOperator(req.botApp);
+        const realManagers = req.botApp.bots.map(x => req.user?.accessibleSubreddits(x).map(x => x.displayLabel)).flat() as string[];
+        const {level = 'verbose', stream, limit = 200, sort = 'descending', streamObjects = false, formatted = true} = req.query;
         if (stream) {
             const origin = req.header('X-Forwarded-For') ?? req.header('host');
             try {
                 logger.stream().on('log', (log: LogInfo) => {
                     if (isLogLineMinLevel(log, level as string)) {
-                        const {subreddit: subName} = log;
-                        if (isOperator || (subName !== undefined && (realManagers.includes(subName) || subName.includes(userName)))) {
+                        const {subreddit: subName, user} = log;
+                        if (isOperator || (subName !== undefined && (realManagers.includes(subName) || (user !== undefined && user.includes(userName))))) {
                             if(streamObjects) {
-                                res.write(`${JSON.stringify(log)}\r\n`);
-                            } else {
+                                let obj: any = log;
+                                if(!formatted) {
+                                    const {[MESSAGE]: fMessage, ...rest} = log;
+                                    obj = rest;
+                                }
+                                res.write(`${JSON.stringify(obj)}\r\n`);
+                            } else if(formatted) {
                                 res.write(`${log[MESSAGE]}\r\n`)
+                            } else {
+                                res.write(`${log.message}\r\n`)
                             }
                         }
                     }
                 });
                 logger.info(`${userName} from ${origin} => CONNECTED`);
                 await pEvent(req, 'close');
-                console.log('Request closed detected with "close" listener');
+                //logger.debug('Request closed detected with "close" listener');
                 res.destroy();
                 return;
             } catch (e: any) {
@@ -55,18 +74,61 @@ const logs = (subLogMap: Map<string, LogEntry[]>) => {
                 res.destroy();
             }
         } else {
-            const logs = filterLogBySubreddit(subLogMap, realManagers, {
-                level: (level as string),
-                operator: isOperator,
-                user: userName,
-                sort: sort as 'descending' | 'ascending',
-                limit: Number.parseInt((limit as string))
-            });
-            const subArr: any = [];
-            logs.forEach((v: string[], k: string) => {
-                subArr.push({name: k, logs: v.join('')});
-            });
-            return res.json(subArr);
+            let bots: Bot[] = [];
+            if(req.serverBot !== undefined) {
+                bots = [req.serverBot];
+            } else {
+                bots = req.user?.accessibleBots(req.botApp.bots) as Bot[];
+            }
+
+            const allReq = req.query.subreddit !== undefined && (req.query.subreddit as string).toLowerCase() === 'all';
+
+            const botArr: any = [];
+            for(const b of bots) {
+                const managerLogs = new Map<string, LogInfo[]>();
+                const managers = req.manager !== undefined ? [req.manager] : req.user?.accessibleSubreddits(b) as Manager[];
+                for (const m of managers) {
+                    const logs = filterLogs(m.logs, {
+                        level: (level as string),
+                        // @ts-ignore
+                        sort,
+                        limit: Number.parseInt((limit as string)),
+                        returnType: 'object'
+                    }) as LogInfo[];
+                    managerLogs.set(m.getDisplay(), logs);
+                }
+                const allLogs = filterLogs([...[...managerLogs.values()].flat(), ...(req.user?.isInstanceOperator(req.botApp) ? b.logs : b.logs.filter(x => x.user === req.user?.name))], {
+                    level: (level as string),
+                    // @ts-ignore
+                    sort,
+                    limit: limit as string,
+                    returnType: 'object'
+                }) as LogInfo[];
+                const systemLogs = filterLogs(req.user?.isInstanceOperator(req.botApp) ? b.logs : b.logs.filter(x => x.user === req.user?.name), {
+                    level: (level as string),
+                    // @ts-ignore
+                    sort,
+                    limit: limit as string,
+                    returnType: 'object'
+                }) as LogInfo[];
+                botArr.push({
+                    name: b.getBotName(),
+                    system: systemLogs,
+                    all: formatted ? allLogs.map(x => {
+                        const {[MESSAGE]: fMessage, ...rest} = x;
+                        return {...rest, formatted: fMessage};
+                    }) : allLogs,
+                    subreddits: allReq ? [] : [...managerLogs.entries()].reduce((acc: any[], curr) => {
+                        const l = formatted ? curr[1].map(x => {
+                            const {[MESSAGE]: fMessage, ...rest} = x;
+                            return {...rest, formatted: fMessage};
+                            }) : curr[1];
+                        acc.push({name: curr[0], logs: l});
+                        return acc;
+                    }, [])
+                });
+            }
+            return res.json(botArr);
         }
     };
 

src/Web/Server/routes/authenticated/user/status.ts

@@ -2,26 +2,27 @@ import {Request, Response} from 'express';
 import {
     boolToString,
     cacheStats,
-    filterLogBySubreddit,
+    filterLogBySubreddit, filterLogs,
     formatNumber,
     intersect,
-    LogEntry,
+    LogEntry, logSortFunc,
     pollingInfo
 } from "../../../../../util";
 import {Manager} from "../../../../../Subreddit/Manager";
 import dayjs from "dayjs";
-import {ResourceStats, RUNNING, STOPPED, SYSTEM} from "../../../../../Common/interfaces";
+import {LogInfo, ResourceStats, RUNNING, STOPPED, SYSTEM} from "../../../../../Common/interfaces";
 import {BotStatusResponse} from "../../../../Common/interfaces";
 import winston from "winston";
 import {opStats} from "../../../../Common/util";
-import {authUserCheck, botRoute} from "../../../middleware";
+import {authUserCheck, botRoute, subredditRoute} from "../../../middleware";
 import Bot from "../../../../../Bot";
 
 const status = () => {
 
     const middleware = [
         authUserCheck(),
-        //botRoute(),
+        botRoute(false),
+        subredditRoute(false)
     ];
 
     const response = async (req: Request, res: Response) => {
@@ -32,29 +33,17 @@ const status = () => {
             sort = 'descending',
         } = req.query;
 
-        // @ts-ignore
-        const botLogMap = req.botLogs as Map<string, Map<string, LogEntry[]>>;
-        // @ts-ignore
-        const systemLogs = req.systemLogs as LogEntry[];
+        bots = req.user?.accessibleBots(req.botApp.bots) as Bot[];
 
-
-        if(req.serverBot !== undefined) {
-            bots = [req.serverBot];
-        } else {
-            bots = (req.user as Express.User).isOperator ? req.botApp.bots : req.botApp.bots.filter(x => {
-                const i = intersect(req.user?.subreddits as string[], x.subManagers.map(y => y.subreddit.display_name));
-                return i.length > 0;
-            });
-        }
         const botResponses: BotStatusResponse[] = [];
+        let index = 1;
         for(const b of bots) {
-            botResponses.push(await botStatResponse(b, req, botLogMap));
+            botResponses.push(await botStatResponse(b, req, index));
+            index++;
         }
         const system: any = {};
-        if((req.user as Express.User).isOperator) {
-            // @ts-ignore
-            system.logs = filterLogBySubreddit(new Map([['app', systemLogs]]), [], {level, sort, limit, operator: true}).get('app');
-        }
+        // @ts-ignore
+        system.logs = filterLogs(req.sysLogs, {level, sort, limit, user: req.user?.isInstanceOperator(req.botApp) ? undefined : req.user?.name, returnType: 'object' }) as LogInfo[];
         const response = {
             bots: botResponses,
             system: system,
@@ -62,7 +51,7 @@ const status = () => {
         return res.json(response);
     }
 
-    const botStatResponse = async (bot: Bot, req: Request, botLogMap: Map<string, Map<string, LogEntry[]>>) => {
+    const botStatResponse = async (bot: Bot, req: Request, index: number) => {
         const {
             //subreddits = [],
             //user: userVal,
@@ -72,37 +61,26 @@ const status = () => {
             lastCheck
         } = req.query;
 
-        const {name: userName, realManagers = [], isOperator} = req.user as Express.User;
-        const user = userName as string;
-        const subreddits = realManagers;
-        //const isOperator = opNames.includes(user.toLowerCase())
-
-        const logs = filterLogBySubreddit(botLogMap.get(bot.botName as string) || new Map(), realManagers, {
-            level: (level as string),
-            operator: isOperator,
-            user,
-            // @ts-ignore
-            sort,
-            limit: Number.parseInt((limit as string))
-        });
+        const allReq = req.query.subreddit !== undefined && (req.query.subreddit as string).toLowerCase() === 'all';
 
         const subManagerData = [];
-        for (const s of subreddits) {
-            const m = bot.subManagers.find(x => x.displayLabel === s) as Manager;
-            if(m === undefined) {
-                continue;
-            }
-            if(!(req.user as Express.User).isOperator && !(req.user?.subreddits as string[]).includes(m.subreddit.display_name)) {
-                continue;
-            }
+        for (const m of req.user?.accessibleSubreddits(bot) as Manager[]) {
+            const logs = req.manager === undefined || allReq || req.manager.getDisplay() === m.getDisplay() ? filterLogs(m.logs, {
+                    level: (level as string),
+                    // @ts-ignore
+                    sort,
+                    limit: limit as string,
+                    returnType: 'object'
+                }) as LogInfo[]: [];
             const sd = {
-                name: s,
+                name: m.displayLabel,
                 //linkName: s.replace(/\W/g, ''),
-                logs: logs.get(s) || [], // provide a default empty value in case we truly have not logged anything for this subreddit yet
+                logs: logs || [], // provide a default empty value in case we truly have not logged anything for this subreddit yet
                 botState: m.botState,
                 eventsState: m.eventsState,
                 queueState: m.queueState,
                 indicator: 'gray',
+                permissions: [],
                 queuedActivities: m.queue.length(),
                 runningActivities: m.queue.running(),
                 maxWorkers: m.queue.concurrency,
@@ -242,12 +220,22 @@ const status = () => {
         const cacheMiss = subManagerData.reduce((acc, curr) => acc + curr.stats.cache.totalMiss, 0);
         const sharedSub = subManagerData.find(x => x.stats.cache.isShared);
         const sharedCount = sharedSub !== undefined ? sharedSub.stats.cache.currentKeyCount : 0;
+        const scopes = req.user?.isInstanceOperator(bot) ? bot.client.scope : [];
+        const allSubLogs = subManagerData.map(x => x.logs).flat().sort(logSortFunc(sort as string)).slice(0, (limit as number) + 1);
+        const allLogs = filterLogs([...allSubLogs, ...(req.user?.isInstanceOperator(req.botApp) ? bot.logs : bot.logs.filter(x => x.user === req.user?.name))], {
+            level: (level as string),
+            // @ts-ignore
+            sort,
+            limit: limit as string,
+            returnType: 'object'
+        }) as LogInfo[];
         let allManagerData: any = {
             name: 'All',
             status: bot.running ? 'RUNNING' : 'NOT RUNNING',
             indicator: bot.running ? 'green' : 'grey',
             maxWorkers,
             globalMaxWorkers,
+            scopes: scopes === null || !Array.isArray(scopes) ? [] : scopes,
             subMaxWorkers,
             runningActivities,
             queuedActivities,
@@ -256,7 +244,7 @@ const status = () => {
                 causedBy: SYSTEM
             },
             dryRun: boolToString(bot.dryRun === true),
-            logs: logs.get('all'),
+            logs: allLogs,
             checks: checks,
             softLimit: bot.softLimit,
             hardLimit: bot.hardLimit,
@@ -308,11 +296,11 @@ const status = () => {
                 startedAt: bot.startedAt.local().format('MMMM D, YYYY h:mm A Z'),
                 running: bot.running,
                 error: bot.error,
-                account: bot.botAccount as string,
-                name: bot.botName as string,
+                account: (bot.botAccount as string) ?? `Bot ${index}`,
+                name: (bot.botName as string) ?? `Bot ${index}`,
                 ...opStats(bot),
             },
-            subreddits: [allManagerData, ...subManagerData],
+            subreddits: [allManagerData, ...(allReq ? subManagerData.map(({logs, ...x}) => ({...x, logs: []})) : subManagerData)],
 
         };
 

src/Web/Server/server.ts

@@ -1,5 +1,5 @@
 import {addAsync, Router} from '@awaitjs/express';
-import express, {Request, Response} from 'express';
+import express, {Request, Response, NextFunction, RequestHandler} from 'express';
 import bodyParser from 'body-parser';
 import {App} from "../../App";
 import {Transform} from "stream";
@@ -9,26 +9,23 @@ import {Strategy as JwtStrategy, ExtractJwt} from 'passport-jwt';
 import passport from 'passport';
 import tcpUsed from 'tcp-port-used';
 
-import {
-    intersect,
-    LogEntry, parseBotLogName,
-    parseSubredditLogName
-} from "../../util";
 import {getLogger} from "../../Utils/loggerFactory";
 import LoggedError from "../../Utils/LoggedError";
-import {Invokee, LogInfo, OperatorConfig} from "../../Common/interfaces";
+import {Invokee, LogInfo, OperatorConfigWithFileContext} from "../../Common/interfaces";
 import http from "http";
-import SimpleError from "../../Utils/SimpleError";
 import {heartbeat} from "./routes/authenticated/applicationRoutes";
 import logs from "./routes/authenticated/user/logs";
 import status from './routes/authenticated/user/status';
+import liveStats from './routes/authenticated/user/liveStats';
 import {actionedEventsRoute, actionRoute, configRoute, configLocationRoute, deleteInviteRoute, addInviteRoute, getInvitesRoute} from "./routes/authenticated/user";
 import action from "./routes/authenticated/user/action";
 import {authUserCheck, botRoute} from "./middleware";
 import {opStats} from "../Common/util";
 import Bot from "../../Bot";
 import addBot from "./routes/authenticated/user/addBot";
-import dayjs from "dayjs";
+import ServerUser from "../Common/User/ServerUser";
+import {SimpleError} from "../../Utils/Errors";
+import {ErrorWithCause} from "pony-cause";
 
 const server = addAsync(express());
 server.use(bodyParser.json());
@@ -45,13 +42,9 @@ declare module 'express-session' {
     }
 }
 
-const subLogMap: Map<string, LogEntry[]> = new Map();
-const systemLogs: LogEntry[] = [];
-const botLogMap: Map<string, Map<string, LogEntry[]>> = new Map();
+let sysLogs: LogInfo[] = [];
 
-const botSubreddits: Map<string, string[]> = new Map();
-
-const rcbServer = async function (options: OperatorConfig) {
+const rcbServer = async function (options: OperatorConfigWithFileContext) {
 
     const {
         operator: {
@@ -71,36 +64,12 @@ const rcbServer = async function (options: OperatorConfig) {
     const logger = getLogger({...options.logging});
 
     logger.stream().on('log', (log: LogInfo) => {
-        const logEntry: LogEntry = [dayjs(log.timestamp).unix(), log];
 
         const {bot: botName, subreddit: subName} = log;
 
-        if(botName === undefined) {
-            systemLogs.unshift(logEntry);
-            systemLogs.slice(0, 201);
-        } else {
-            const botLog = botLogMap.get(botName) || new Map();
-
-            if(subName === undefined) {
-                const appLogs = botLog.get('app') || [];
-                appLogs.unshift(logEntry);
-                botLog.set('app', appLogs.slice(0, 200 + 1));
-            } else {
-                let botSubs = botSubreddits.get(botName) || [];
-                if(botSubs.length === 0 && app !== undefined) {
-                    const b = app.bots.find(x => x.botName === botName);
-                    if(b !== undefined) {
-                        botSubs = b.subManagers.map(x => x.displayLabel);
-                        botSubreddits.set(botName, botSubs);
-                    }
-                }
-                if(botSubs.length === 0 || botSubs.includes(subName)) {
-                    const subLogs = botLog.get(subName) || [];
-                    subLogs.unshift(logEntry);
-                    botLog.set(subName, subLogs.slice(0, 200 + 1));
-                }
-            }
-            botLogMap.set(botName, botLog);
+        if(botName === undefined && subName === undefined) {
+            sysLogs.unshift(log);
+            sysLogs = sysLogs.slice(0, 201);
         }
     })
 
@@ -115,9 +84,7 @@ const rcbServer = async function (options: OperatorConfig) {
         httpServer = await server.listen(port);
         io = new SocketServer(httpServer);
     } catch (err: any) {
-        logger.error('Error occurred while initializing web or socket.io server', err);
-        err.logged = true;
-        throw err;
+        throw new ErrorWithCause('[Server] Error occurred while initializing web or socket.io server', {cause: err});
     }
 
     logger.info(`API started => localhost:${port}`);
@@ -128,31 +95,35 @@ const rcbServer = async function (options: OperatorConfig) {
     }, function (jwtPayload, done) {
         const {name, subreddits = [], machine = true} = jwtPayload.data;
         if (machine) {
-            return done(null, {machine});
+            const user = new ServerUser(name, subreddits, true, false);
+            return done(null, user);
+            //return done(null, {machine});
         }
         const isOperator = opNames.includes(name.toLowerCase());
-        let moderatedBots: string[] = [];
-        let moderatedManagers: string[] = [];
-        let realBots: string[] = [];
-        let realManagers: string[] = [];
-        if(app !== undefined) {
-            const modBots =  app.bots.filter(x => intersect(subreddits, x.subManagers.map(y => y.subreddit.display_name)));
-            moderatedBots = modBots.map(x => x.botName as string);
-            moderatedManagers = [...new Set(modBots.map(x => x.subManagers.map(y => y.displayLabel)).flat())];
-            realBots = isOperator ? app.bots.map(x => x.botName as string) : moderatedBots;
-            realManagers = isOperator ? [...new Set(app.bots.map(x => x.subManagers.map(y => y.displayLabel)).flat())] : moderatedManagers
-        }
+        // let moderatedBots: string[] = [];
+        // let moderatedManagers: string[] = [];
+        // let realBots: string[] = [];
+        // let realManagers: string[] = [];
+        // if(app !== undefined) {
+        //     const modBots =  app.bots.filter(x => intersect(subreddits, x.subManagers.map(y => y.subreddit.display_name)).length > 0);
+        //     moderatedBots = modBots.map(x => x.botName as string);
+        //     moderatedManagers = [...new Set(modBots.map(x => x.subManagers).flat().filter(x => subreddits.includes(x.subreddit.display_name)).map(x => x.displayLabel))];
+        //     realBots = isOperator ? app.bots.map(x => x.botName as string) : moderatedBots;
+        //     realManagers = isOperator ? [...new Set(app.bots.map(x => x.subManagers.map(y => y.displayLabel)).flat())] : moderatedManagers
+        // }
 
-        return done(null, {
-            name,
-            subreddits,
-            isOperator,
-            machine: false,
-            moderatedManagers,
-            realManagers,
-            moderatedBots,
-            realBots,
-        });
+        const user = new ServerUser(name, subreddits, false, isOperator);
+        return done(null, user);
+        // return done(null, {
+        //     name,
+        //     subreddits,
+        //     isOperator,
+        //     machine: false,
+        //     moderatedManagers,
+        //     realManagers,
+        //     moderatedBots,
+        //     realBots,
+        // });
     }));
 
     server.use(passport.authenticate('jwt', {session: false}));
@@ -163,30 +134,32 @@ const rcbServer = async function (options: OperatorConfig) {
 
     server.getAsync('/heartbeat', ...heartbeat({name, display, friendly}));
 
-    server.getAsync('/logs', ...logs(subLogMap));
+    server.getAsync('/logs', ...logs());
 
     server.getAsync('/stats', [authUserCheck(), botRoute(false)], async (req: Request, res: Response) => {
         let bots: Bot[] = [];
         if(req.serverBot !== undefined) {
             bots = [req.serverBot];
-        } else {
-            bots = (req.user as Express.User).isOperator ? req.botApp.bots : req.botApp.bots.filter(x => intersect(req.user?.subreddits as string[], x.subManagers.map(y => y.subreddit.display_name)));
+        } else if(req.user !== undefined) {
+            bots = req.user.accessibleBots(req.botApp.bots);
         }
         const resp = [];
+        let index = 1;
         for(const b of bots) {
-            resp.push({name: b.botName, data: await opStats(b)});
+            resp.push({name: b.botName ?? `Bot ${index}`, data: await opStats(b)});
+            index++;
         }
         return res.json(resp);
     });
     const passLogs = async (req: Request, res: Response, next: Function) => {
         // @ts-ignore
-        req.botLogs = botLogMap;
-        // @ts-ignore
-        req.systemLogs = systemLogs;
+        req.sysLogs = sysLogs;
         next();
     }
     server.getAsync('/status', passLogs, ...status())
 
+    server.getAsync('/liveStats', ...liveStats())
+
     server.getAsync('/config', ...configRoute);
 
     server.getAsync('/config/location', ...configLocationRoute);
@@ -197,7 +170,7 @@ const rcbServer = async function (options: OperatorConfig) {
 
     server.getAsync('/check', ...actionRoute);
 
-    server.getAsync('/addBot', ...addBot());
+    server.postAsync('/bot', ...addBot());
 
     server.getAsync('/bot/invite', ...getInvitesRoute);
 

src/Web/assets/public/app.css

@@ -152,3 +152,7 @@ a {
 #saveTip .tooltip:hover {
     transition-delay: 1s;
 }
+
+#redditStatus .iconify-inline {
+    display: inline;
+}

src/Web/assets/views/callback.ejs

@@ -9,19 +9,29 @@
                 <div class="p-6 md:px-10 md:py-6">
                     <div class="text-xl mb-4">Congrats! You did the thing.</div>
                     <div class="space-y-3">
+                        <div>These are the credentials ContextMod will use to act as your bot, <b><%= userName %></b></div>
                         <ul class="list-inside list-disc">
                             <li>Access Token: <b><%= accessToken %></b></li>
                             <li>Refresh Token: <b><%= refreshToken %></b></li>
                         </ul>
-                        <% if(locals.addResult !== undefined) { %>
-                            <div>Result of trying to add bot automatically: <%= addResult %></div>
+                        <div>Note: You can revoke ContextMod's access to this account at any time by visiting the <a href="https://www.reddit.com/prefs/apps">reddit app preferences</a> while logged in as the account and clicking the <b>revoke access</b> link under ContextMod</div>
+                    </div>
+                    <div class="text-xl my-4">What Do I Do Now?</div>
+                    <div class="space-y-3">
+                        <% if(locals.stored === true) { %>
+                            <% if(locals.success === true) { %>
+                                <div>Credentials were successfully persisted to the application and the bot was automatically started! You may now <a href="/">login with your normal/moderator account</a> to view the web dashboard where your bot can be monitored.</div>
+                            <% } else { %>
+                                <div>The bot was successfully saved to the application but it could not be started automatically. Please inform the operator so they can restart the application.</div>
+                            <% } %>
                         <% } else { %>
-                            <div>Bot was not automatically added to an instance and will need to manually appended to configuration...</div>
+                                <div>These credentials were <b>not automatically added</b> to an instance and will need to be <b>manually added by the operator</b>:</div>
+                            <ul class="list-inside list-disc">
+                                <li>If you are a <b>Moderator</b> then copy the above <b>Tokens</b> and pass them on to the Operator of this ContextMod instance.</li>
+                                <li>If you are an <b>Operator</b> copy these somewhere and then restart the application providing these as either arguments, environmental variables, or in a config as described in the <a
+                                            href="https://github.com/FoxxMD/context-mod/blob/master/docs/operatorConfiguration.md#defining-configuration">configuration guide</a></li>
+                            </ul>
                         <% } %>
-                        <div>If you are a <b>Moderator</b> then copy the above <b>Tokens</b> and pass them on to the Operator of this ContextMod instance.</div>
-                        <div>If you are an <b>Operator</b> copy these somewhere and then restart the application providing these as either arguments, environmental variables, or in a json config as described in the <a
-                                    href="https://github.com/FoxxMD/context-mod/blob/master/docs/operatorConfiguration.md#defining-configuration">configuration guide</a>
-                        </div>
                     </div>
                 </div>
             </div>

src/Web/assets/views/helper.ejs

@@ -46,7 +46,43 @@
                                class="text-black placeholder-gray-500 rounded mt-2 mb-3 p-2"
                                placeholder="<%= locals.clientSecret !== undefined ? 'Use Provided Client Secret' : 'Client Secret Not Provided' %>">
                     </div>
-                    <div class="text-lg text-semibold my-3">3. Select permissions</div>
+                    <div class="text-lg text-semibold my-3">3. Select Instance</div>
+                    <div class="my-2 ml-5">
+                        <div class="space-y-3">
+                            <div>Specify the ContextMod instance to add this bot to.</div>
+                            <select id="instanceSelect" style="max-width:400px;" class="form-select
+      block
+      w-full
+      px-3
+      py-1.5
+      text-base
+      font-normal
+      text-gray-700
+      bg-white bg-clip-padding bg-no-repeat
+      border border-solid border-gray-300
+      rounded
+      transition
+      ease-in-out
+      m-0
+      focus:text-gray-700 focus:bg-white focus:border-blue-600 focus:outline-none" aria-label="Default select example">
+                                <% instances.forEach(function (name, index){ %>
+                                    <option selected="<%= index === 0 ? 'true' : 'false' %>" value="<%= name %>"><%= name %></option>
+                                    <%= name %>
+                                <% }) %>
+                            </select>
+                        </div>
+                    </div>
+                    <div class="text-lg text-semibold my-3">4. Optionally, restrict to Subreddits</div>
+                    <div class="my-2 ml-5">
+                        <div class="space-y-3">
+                        <div>Specify which subreddits, out of all the subreddits the bot moderates, CM should run on.</div>
+                        <div>Subreddits should be seperated with a comma. Leave blank to run on all moderated subreddits</div>
+                        <input id="subreddits" style="max-width:800px; display: block;"
+                               class="text-black placeholder-gray-500 rounded mt-2 mb-3 p-2 w-full"
+                               placeholder="aSubreddit,aSecondSubreddit,aThirdSubreddit">
+                        </div>
+                    </div>
+                    <div class="text-lg text-semibold my-3">5. Select permissions</div>
                     <div class="my-2 ml-5">
                         <div class="space-y-3">
                             <div>These are permissions to allow the bot account to perform these actions, <b>in
@@ -220,6 +256,8 @@
                 clientSecret: document.querySelector('#clientSecret').value,
                 code: document.querySelector("#inviteCode").value === '' ? undefined : document.querySelector("#inviteCode").value,
                 permissions,
+                instance: document.querySelector('#instanceSelect').value,
+                subreddits: document.querySelector('#subreddits').value
             })
         }).then((resp) => {
             if(!resp.ok) {

src/Web/assets/views/partials/botsTab.ejs

@@ -1,10 +1,19 @@
 <div class="space-x-4 py-1 md:px-10 leading-6 font-semibold bg-gray-700">
     <div class="container mx-auto">
-        <% if(locals.bots !== undefined) { %>
         <ul id="botTabs" class="inline-flex flex-wrap">
+            <% if(locals.system !== undefined && locals.system.logs !== undefined) {%>
+                <li class="my-3 px-3">
+                        <span data-bot="system" class="rounded-md py-2 px-3 tabSelectWrapper">
+                        <a class="tabSelect instance font-normal pointer hover:font-bold" data-bot="system">
+                            System
+                        </a>
+                        </span>
+                </li>
+            <% } %>
+        <% if(locals.bots !== undefined) { %>
             <% bots.forEach(function (data){ %>
                 <li class="my-3 px-3">
-                        <span data-bot="<%= data.system.name %>" class="rounded-md py-2 px-3 tabSelectWrapper">
+                        <span data-bot="<%= data.system.name %>" class="rounded-md py-2 px-3 tabSelectWrapper real">
                         <a class="tabSelect font-normal pointer hover:font-bold"
                            data-bot="<%= data.system.name %>">
                             <%= data.system.name %>
@@ -24,7 +33,7 @@
                         </span>
                     </li>
             <% } %>
-        </ul>
         <% } %>
+        </ul>
     </div>
 </div>

src/Web/assets/views/partials/footer.ejs

@@ -1,5 +1,29 @@
 <div class="py-3 flex items-center justify-around font-semibold">
     <div>
-        <a href="https://github.com/FoxxMD/context-mod">ContextMod Web</a> created by /u/FoxxMD
+        <a href="https://github.com/FoxxMD/context-mod">ContextMod <%= locals.applicationIdentifier%></a> created by /u/FoxxMD
     </div>
 </div>
+<script type="text/javascript" src="https://cdn.statuspage.io/se-v2.js"></script>
+<script>
+    // https://www.redditstatus.com/api#status
+    var sp = new StatusPage.page({ page : '2kbc0d48tv3j' });
+    sp.status({
+        success : function(data) {
+            console.log(data.status.indicator);
+            switch(data.status.indicator){
+                case 'minor':
+                    document.querySelector('#redditStatus').innerHTML = '<span class="iconify-inline yellow" data-icon="ep:warning-filled"></span>';
+                    break;
+                case 'none':
+                    document.querySelector('#redditStatus').innerHTML = '<span class="iconify-inline green" data-icon="ep:circle-check-filled"></span>';
+                    break;
+                default:
+                    document.querySelector('#redditStatus').innerHTML = '<span class="iconify-inline red" data-icon="ep:warning-filled"></span>';
+                    break;
+            }
+            // data.page.updated_at
+            // data.status.indicator => none, minor, major, or critical
+            // data.status.description
+        }
+    });
+</script>

src/Web/assets/views/partials/head.ejs

@@ -1,6 +1,6 @@
 <head>
     <link rel="stylesheet" href="/public/tailwind.min.css"/>
-    <script src="https://code.iconify.design/2/2.1.0/iconify.min.js"></script>
+    <script src="https://code.iconify.design/2/2.1.2/iconify.min.js"></script>
     <link rel="stylesheet" href="/public/themeToggle.css">
     <link rel="stylesheet" href="/public/app.css">
     <title><%= locals.title !== undefined ? title : `${locals.botName !== undefined ? `CM for ${botName}` : 'ContextMod'}`%></title>

src/Web/assets/views/partials/header.ejs

@@ -33,7 +33,12 @@
                 </ul>
                 <% } %>
             </div>
-            <div class="flex items-center flex-end text-sm">
+            <div class="flex items-center mr-8 text-sm">
+                <a href="https://redditstatus.com" target="_blank">
+                    <span>Reddit Status: <span id="redditStatus" class="ml-2"><span class="iconify-inline" data-icon="ep:question-filled"></span></span></span>
+                </a>
+            </div>
+            <div class="flex items-center text-sm">
                 <a href="logout">Logout</a>
             </div>
         </div>

src/Web/assets/views/status.ejs

@@ -9,6 +9,19 @@
         <div class="grid">
             <div class="">
                 <div class="pb-6 md:px-7">
+                    <% if(isOperator) { %>
+                        <div class="sub" data-bot="system" data-subreddit="All">
+                            <div class="grid grid-cols-1 lg:grid-cols-2 xl:grid-cols-3 2xl:grid-cols-3 gap-5">
+                            </div>
+                            <br/>
+                            <%- include('partials/loadingIcon') %>
+                            <div data-subreddit="All" class="logs font-mono text-sm">
+                                <% system.logs.forEach(function (logEntry){ %>
+                                    <%- logEntry %>
+                                <% }) %>
+                            </div>
+                        </div>
+                    <% } %>
                     <% bots.forEach(function (bot){ %>
                         <% bot.subreddits.forEach(function (data){ %>
                         <div class="sub <%= bot.system.running ? '' : 'offline' %>" data-subreddit="<%= data.name %>" data-bot="<%= bot.system.name %>">
@@ -147,6 +160,19 @@
                                                         </div>
                                                     </div>
                                                 </div>
+                                                <label>Mod Perms</label>
+                                                <span class="has-tooltip">
+                                                    <span class='tooltip rounded shadow-lg p-1 bg-gray-100 text-black -mt-2'>
+                                                        <span>
+                                                            <ul class="list-inside list-disc modPermissionsList">
+                                                                <% data.permissions.forEach(function (i){ %>
+                                                                    <li class="font-mono"><%= i %></li>
+                                                                <% }) %>
+                                                            </ul>
+                                                        </span>
+                                                    </span>
+                                                    <span class="cursor-help underline modPermissionsCount" style="text-decoration-style: dotted"><%= data.permissions.length %></span>
+                                                </span>
                                                 <label>Slow Mode</label>
                                                 <span><%= data.delayBy %></span>
                                             <% } %>
@@ -210,6 +236,19 @@
                                             <% if (data.name === 'All' && isOperator) { %>
                                                 <label>Operators</label>
                                                 <span><%= operators %></span>
+                                                <label>Oauth Scopes</label>
+                                                    <span class="has-tooltip">
+                                                    <span class='tooltip rounded shadow-lg p-1 bg-gray-100 text-black -mt-2'>
+                                                        <span>
+                                                            <ul class="list-inside list-disc">
+                                                                <% data.scopes.forEach(function (i){ %>
+                                                                    <li class="font-mono"><%= i %></li>
+                                                                <% }) %>
+                                                            </ul>
+                                                        </span>
+                                                    </span>
+                                                    <span class="cursor-help underline" style="text-decoration-style: dotted"><%= data.scopes.length %></span>
+                                                </span>
                                             <% } else %>
                                         </div>
                                         <% if (data.name !== 'All') { %>
@@ -611,12 +650,9 @@
                                     <input data-subreddit="<%= data.name %>" style="min-width: 420px;"
                                            class="border-gray-50 placeholder-gray-500 rounded mt-2 mb-3 p-2 text-black checkUrl"
                                            placeholder="<%= data.name === 'All' ? 'Run Bot on a permalink from any moderated Subreddit' : `Run Bot on a permalink using this Subreddit's config` %>"/>
-                                    <span class="mx-2">
-                                        <input type="checkbox" class="dryrunCheck" data-subreddit="<%= data.name %>"
-                                               name="dryrunCheck">
-                                        <label for="dryrunCheck">Dry Run?</label>
-                                    </span>
-                                    <a class="runCheck" data-subreddit="<%= data.name %>" href="">Run</a>
+
+                                    <a class="hover:bg-gray-700 pointer-events-none opacity-20 no-underline rounded-md mx-4 py-2 px-3 border checkActions dryRunCheck" data-subreddit="<%= data.name %>" href="">Dry Run</a>
+                                    <a class="hover:bg-gray-700 pointer-events-none opacity-20 no-underline rounded-md py-2 px-3 border checkActions runCheck" data-subreddit="<%= data.name %>" href="">Run</a>
                                 </div>
                                 <%- include('partials/logSettings') %>
                             </div>
@@ -641,6 +677,48 @@
 </div>
 <%- include('partials/instanceTabJs') %>
 <%- include('partials/logSettingsJs') %>
+<script src="https://unpkg.com/autolinker@3.14.3/dist/Autolinker.min.js" crossorigin="anonymous"></script>
+<script src="https://unpkg.com/dayjs@1.10.7/dayjs.min.js" crossorigin="anonymous"></script>
+<script src="https://unpkg.com/dayjs@1.10.7/plugin/advancedFormat.js"></script>
+<script src="https://unpkg.com/dayjs@1.10.7/plugin/timezone.js"></script>
+<script>
+    dayjs.extend(window.dayjs_plugin_timezone)
+    dayjs.extend(window.dayjs_plugin_advancedFormat)
+    window.formattedTime = (short, full) => `<span class="has-tooltip"><span style="margin-top:35px" class='tooltip rounded shadow-lg p-1 bg-gray-100 text-black space-y-3 p-2 text-left'>${full}</span><span>${short}</span></span>`;
+    window.formatLogLineToHtml = (log, timestamp = undefined) => {
+        const val = typeof log === 'string' ? log : log['MESSAGE'];
+        const logContent = Autolinker.link(val, {
+            email: false,
+            phone: false,
+            mention: false,
+            hashtag: false,
+            stripPrefix: false,
+            sanitizeHtml: true,
+        })
+            .replace(/(\s*debug\s*):/i, '<span class="debug blue">$1</span>:')
+            .replace(/(\s*warn\s*):/i, '<span class="warn yellow">$1</span>:')
+            .replace(/(\s*info\s*):/i, '<span class="info green">$1</span>:')
+            .replace(/(\s*error\s*):/i, '<span class="error red">$1</span>:')
+            .replace(/(\s*verbose\s*):/i, '<span class="error purple">$1</span>:')
+            .replaceAll('\n', '<br />');
+        //.replace(HYPERLINK_REGEX, '<a target="_blank" href="$&">$&</a>');
+        let line = '';
+
+        let timestampString = timestamp;
+        if(timestamp === undefined && typeof log !== 'string') {
+            timestampString = log.timestamp;
+        }
+
+        if(timestampString !== undefined) {
+            const timeStampReplacement = formattedTime(dayjs(timestampString).format('HH:mm:ss z'), timestampString);
+            const splitLine = logContent.split(timestampString);
+            line = `<div class="logLine">${splitLine[0]}${timeStampReplacement}<span style="white-space: pre-wrap">${splitLine[1]}</span></div>`;
+        } else {
+            line = `<div style="white-space: pre-wrap" class="logLine">${logContent}</div>`
+        }
+        return line;
+    }
+</script>
 <script>
     window.sort = 'desc';
 
@@ -658,23 +736,42 @@
         });
     })
 
-    document.querySelectorAll(".runCheck").forEach(el => {
+    document.querySelectorAll(".checkUrl").forEach(el => {
+        const toggleButtons = (e) => {
+            const subFilter = `.sub[data-subreddit="${e.target.dataset.subreddit}"]`;
+            const inputVal = document.querySelector(`${subFilter} .checkUrl`).value;
+            if (inputVal.length > 0) {
+                document.querySelectorAll(`${subFilter} .checkActions`).forEach(el => {
+                    el.classList.remove('pointer-events-none', 'opacity-20');
+                });
+            } else {
+                document.querySelectorAll(`${subFilter} .checkActions`).forEach(el => {
+                    el.classList.add('pointer-events-none', 'opacity-20');
+                });
+            }
+        }
+        el.addEventListener('keyup', toggleButtons, false);
+        el.addEventListener('change', toggleButtons, false);
+    });
+
+    document.querySelectorAll(".checkActions").forEach(el => {
         el.addEventListener('click', e => {
             e.preventDefault();
             const subreddit = e.target.dataset.subreddit;
-            const urlInput = document.querySelector(`[data-subreddit="${subreddit}"].checkUrl`);
-            const dryRunCheck = document.querySelector(`[data-subreddit="${subreddit}"].dryrunCheck`);
+            const subFilter = `.sub[data-subreddit="${subreddit}"]`;
+            const urlInput = document.querySelector(`${subFilter} .checkUrl`);
+
+            const isDryun = e.target.classList.contains('dryRunCheck');
 
             const subSection = e.target.closest('div.sub');
             bot = subSection.dataset.bot;
             const url = urlInput.value;
-            const dryRun = dryRunCheck.checked ? 1 : 0;
 
-            const fetchUrl = `/api/check?instance=<%= instanceId %>&bot=${bot}&url=${url}&dryRun=${dryRun}&subreddit=${subreddit}`;
+            const fetchUrl = `/api/check?instance=<%= instanceId %>&bot=${bot}&url=${url}&dryRun=${isDryun ? 1 : 0}&subreddit=${subreddit}`;
             fetch(fetchUrl);
 
             urlInput.value = '';
-            dryRunCheck.checked = false;
+            urlInput.dispatchEvent(new Event('change'));
         });
     });
 
@@ -711,23 +808,31 @@
             const firstSub = document.querySelectorAll(`[data-bot="${bot}"].sub`)[0];
             firstSub.classList.add('active');
 
+            let firstSubWrapper;
             const firstSubTab = document.querySelector(`ul[data-bot="${bot}"] [data-subreddit="${firstSub.dataset.subreddit}"].tabSelect`);
-            firstSubTab.classList.add('font-bold', 'no-underline', 'pointer-events-none');
-            const firstSubWrapper = firstSubTab.closest('.tabSelectWrapper');
-            //document.querySelector(`[data-subreddit="${subreddit}"][data-bot="${bot}"].sub`).classList.add('active');
+            if(firstSubTab !== null) {
+                firstSubTab.classList.add('font-bold', 'no-underline', 'pointer-events-none');
+                firstSubWrapper = firstSubTab.closest('.tabSelectWrapper');
+                //document.querySelector(`[data-subreddit="${subreddit}"][data-bot="${bot}"].sub`).classList.add('active');
+            }
 
             document.querySelectorAll('.tabSelectWrapper').forEach(el => {
                 el.classList.remove('border-2');
                 el.classList.add('border');
             })
 
-            firstSubWrapper.classList.remove('border');
-            firstSubWrapper.classList.add('border-2');
+            if(firstSubWrapper !== undefined) {
+                firstSubWrapper.classList.remove('border');
+                firstSubWrapper.classList.add('border-2');
+            }
 
             document.querySelectorAll('[data-bot].subreddit.nestedTabs').forEach(el => {
                 el.classList.remove('active');
             })
-            document.querySelector(`[data-bot="${bot}"].subreddit.nestedTabs`).classList.add('active');
+            const nested = document.querySelector(`[data-bot="${bot}"].subreddit.nestedTabs`);
+            if(nested !== null) {
+                nested.classList.add('active');
+            }
 
             const wrapper = e.target.closest('.tabSelectWrapper');//document.querySelector(`[data-subreddit="${subreddit}"].tabSelectWrapper`);
             wrapper.classList.remove('border');
@@ -736,7 +841,9 @@
             if ('URLSearchParams' in window) {
                 var searchParams = new URLSearchParams(window.location.search)
                 searchParams.set("bot", bot);
-                searchParams.set("sub", firstSub.dataset.subreddit);
+                if(firstSub.dataset.subreddit !== undefined) {
+                    searchParams.set("sub", firstSub.dataset.subreddit);
+                }
                 var newRelativePathQuery = window.location.pathname + '?' + searchParams.toString();
                 history.pushState(null, '', newRelativePathQuery);
             }
@@ -772,6 +879,44 @@
                 var newRelativePathQuery = window.location.pathname + '?' + searchParams.toString();
                 history.pushState(null, '', newRelativePathQuery);
             }
+            const activeSub = document.querySelector(`[data-subreddit="${subreddit}"][data-bot="${bot}"].sub`);
+            if(!activeSub.classList.contains('seen')) {
+                //firstSub.classList.add('seen');
+
+                //subreddit = firstSub.dataset.subreddit;
+                //bot = subSection.dataset.bot;
+                level = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="level"]`).value;
+                sort = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="sort"]`).value;
+                limitSel = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="limit"]`).value;
+
+                fetch(`/api/logs?instance=<%= instanceId %>&bot=${bot}&subreddit=${subreddit}&level=${level}&sort=${sort}&limit=${limitSel}&stream=false&formatted=true`).then((resp) => {
+                    if (!resp.ok) {
+                        console.error('Response was not OK from logs GET');
+                    } else {
+                        resp.json().then((data) => {
+                            const logContainer = document.querySelector(`[data-subreddit="${subreddit}"] .logs`);
+                            const logLines = (subreddit.toLowerCase() === 'all' ? data[0].all : data[0].subreddits[0].logs).map(x => {
+                                let fString = x.formatted;
+                                if(x.bot !== undefined) {
+                                    fString = fString.replace(`~${x.bot}~ `, '');
+                                }
+                                if(x.subreddit !== undefined && subreddit !== 'All') {
+                                    fString = fString.replace(`{${x.subreddit}} `, '');
+                                }
+                                return window.formatLogLineToHtml(fString, x.timestamp)
+                            }).join('');
+                            logContainer.insertAdjacentHTML('afterbegin', logLines);
+                            activeSub.classList.add('seen');
+                        });
+                    }
+                }).catch((err) => {
+                    console.log(err);
+                });
+
+            }
+            if(window.socket !== undefined) {
+                window.socket.emit('viewing', {bot, subreddit});
+            }
         });
     });
 
@@ -781,7 +926,10 @@
     let shownBot = searchParams.get('bot');
     if(shownBot === null) {
         // show the first bot listed if none is specified
-        const firstBot = document.querySelector('.tabSelectWrapper[data-bot]');
+        let firstBot = document.querySelector('.real.tabSelectWrapper[data-bot]');
+        if(firstBot === null) {
+
+        }
         if(firstBot !== null) {
             shownBot = firstBot.dataset.bot;
             searchParams.set('bot', shownBot);
@@ -791,17 +939,27 @@
     }
 
     document.querySelector(`[data-bot="${shownBot}"].tabSelect`).classList.add('font-bold', 'no-underline', 'pointer-events-none');
-    document.querySelector(`ul[data-bot="${shownBot}"] [data-subreddit="${shownSub}"].tabSelect`).classList.add('font-bold', 'no-underline', 'pointer-events-none');
+    const tabSelect = document.querySelector(`ul[data-bot="${shownBot}"] [data-subreddit="${shownSub}"].tabSelect`);
+    if(tabSelect !== null) {
+        tabSelect.classList.add('font-bold', 'no-underline', 'pointer-events-none');
+    }
     document.querySelectorAll('.tabSelectWrapper').forEach(el => el.classList.add('border'));
-    document.querySelector(`[data-bot="${shownBot}"][data-subreddit="${shownSub}"].sub`).classList.add('active');
+    document.querySelector(`[data-bot="${shownBot}"][data-subreddit="${shownSub}"].sub`).classList.add('active', 'seen');
     const subWrapper = document.querySelector(`ul[data-bot="${shownBot}"] [data-subreddit="${shownSub}"].tabSelectWrapper`);
-    subWrapper.classList.remove('border');
-    subWrapper.classList.add('border-2');
+    if(subWrapper !== null) {
+        subWrapper.classList.remove('border');
+        subWrapper.classList.add('border-2');
+    }
     const wrapper = document.querySelector(`[data-bot="${shownBot}"].tabSelectWrapper`);
-    wrapper.classList.remove('border');
-    wrapper.classList.add('border-2');
+    if(wrapper !== null) {
+        wrapper.classList.remove('border');
+        wrapper.classList.add('border-2');
+    }
 
-    document.querySelector(`[data-bot="${shownBot}"].subreddit.nestedTabs`).classList.add('active');
+    const nestedTabs = document.querySelector(`[data-bot="${shownBot}"].subreddit.nestedTabs`);
+    if(nestedTabs !== null) {
+        nestedTabs.classList.add('active');
+    }
 
     document.querySelectorAll('.stats.reloadStats').forEach(el => el.classList.add('hidden'));
     document.querySelectorAll('.allStatsToggle').forEach(el => el.classList.add('font-bold', 'no-underline', 'pointer-events-none'));
@@ -809,23 +967,11 @@
 
 <script src="https://cdn.socket.io/4.1.2/socket.io.min.js" integrity="sha384-toS6mmwu70G0fw54EGlWWeA4z3dyJ+dlXBtSURSKN4vyRFOcxd3Bzjj/AoOwY+Rg" crossorigin="anonymous"></script>
 <script>
-    const SUBREDDIT_NAME_LOG_REGEX = /{(.+?)}/;
-    const BOT_NAME_LOG_REGEX = /~(.+?)~/;
-    const parseALogName = (reg) => {
-        return (val) => {
-            const matches = val.match(reg);
-            if (matches === null) {
-                return undefined;
-            }
-            return matches[1];
-        }
-    }
-    const parseSubredditLogName = parseALogName(SUBREDDIT_NAME_LOG_REGEX);
-    const parseBotLogName = parseALogName(BOT_NAME_LOG_REGEX);
 
     let socket = io({
         reconnectionAttempts: 5, // bail after 5 attempts
     });
+    window.socket = socket;
 
     // get all bots
     let bots = [];
@@ -843,26 +989,35 @@
 
     socket.on("connect", () => {
         document.body.classList.add('connected')
-        socket.on("log", data => {
-            bufferedBot.set('All', bufferedBot.get('All').concat(data));
 
-            const bot = parseBotLogName(data);
+        const shownSub = searchParams.get('sub') || 'All'
+        let shownBot = searchParams.get('bot');
+        window.socket.emit('viewing', {bot: shownBot, subreddit: shownSub});
+
+        socket.on("log", data => {
+            const {
+                subreddit,
+                bot,
+                subredditMessage,
+                allMessage,
+                formattedMessage
+            } = data;
+            if(bot === undefined && subreddit === undefined) {
+                const sys = bufferedBot.get('system');
+                if(sys !== undefined) {
+                    sys.set('All', sys.get('All').concat(formattedMessage));
+                    bufferedBot.set('system', sys);
+                }
+            }
             if(bot !== undefined) {
+                bufferedBot.set('All', bufferedBot.get('All').concat(allMessage));
+
                 const buffBot = bufferedBot.get(bot) || newBufferedLogs();
-                buffBot.set('All', buffBot.get('All').concat(data));
-                const sub = parseSubredditLogName(data);
-                if (sub !== undefined) {
-                    buffBot.set(sub, (buffBot.get(sub) || []).concat(data));
+                buffBot.set('All', buffBot.get('All').concat(allMessage));
+                if (subreddit !== undefined) {
+                    buffBot.set(subreddit, (buffBot.get(subreddit) || []).concat(subredditMessage));
                 }
                 bufferedBot.set(bot, buffBot);
-            } else {
-                bufferedBot.forEach((logs, botName) => {
-                    if(botName === 'All') {
-                        return;
-                    }
-                    logs.set('All', logs.get('All').concat(data));
-                    bufferedBot.set(botName, logs);
-                });
             }
 
 
@@ -873,12 +1028,12 @@
                     }
                     subLogs.forEach((logs, subKey) => {
                         // check sub exists -- may be a web log
-                        const el = document.querySelector(`[data-subreddit="${subKey}"][data-bot="${botName}"].sub`);
+                        const el = document.querySelector(`[data-subreddit="${subKey}"][data-bot="${botName}"].sub.seen`);
                         if(null !== el) {
                             const limit = Number.parseInt(document.querySelector(`[data-subreddit="${subKey}"] [data-type="limit"]`).value);
                             const logContainer = el.querySelector(`.logs`);
                             let existingLogs;
-                            if(window.sort === 'desc') {
+                            if(window.sort === 'desc' || window.sort === 'descending') {
                                 logs.forEach((l) => {
                                     logContainer.insertAdjacentHTML('afterbegin', l);
                                 })
@@ -930,6 +1085,29 @@
             }
 
         });
+        socket.on('liveStats', (resp) => {
+            let el;
+            let isAll = resp.system !== undefined;
+            if(isAll) {
+                // got all
+                el = document.querySelector(`[data-subreddit="All"][data-bot="${resp.bot}"].sub`);
+            } else {
+                // got subreddit
+                el = document.querySelector(`[data-subreddit="${resp.name}"].sub`);
+            }
+
+            if(isAll) {
+
+            } else {
+                if(el.querySelector('.modPermissionsCount').innerHTML != resp.permissions.length) {
+                    el.querySelector('.modPermissionsCount').innerHTML = resp.permissions.length;
+                    el.querySelector('.modPermissionsList').innerHTML = '';
+                    el.querySelector('.modPermissionsList').insertAdjacentHTML('afterbegin', resp.permissions.map(x => `<li class="font-mono">${x}</li>`).join(''));
+                }
+            }
+
+            console.log(resp);
+        });
     });
 
     socket.on('disconnect', () => {

src/Web/interfaces.ts

@@ -1,5 +1,6 @@
 import { URL } from "url";
 import {BotConnection} from "../Common/interfaces";
+import {Logger} from "winston";
 
 export interface BotInstance {
     botName: string
@@ -8,15 +9,16 @@ export interface BotInstance {
     subreddits: string[]
     nanny?: string
     running: boolean
+    instance: CMInstanceInterface
 }
 
-export interface CMInstance extends BotConnection {
-    friendly: string
+export interface CMInstanceInterface extends BotConnection {
+    friendly?: string
     operators: string[]
     operatorDisplay: string
     url: URL,
     normalUrl: string,
-    lastCheck: number
+    lastCheck?: number
     online: boolean
     subreddits: string[]
     bots: BotInstance[]

src/Web/types/express/index.d.ts

@@ -1,30 +1,20 @@
 import {App} from "../../../App";
 import Bot from "../../../Bot";
-import {BotInstance, CMInstance} from "../../interfaces";
+import {BotInstance, CMInstanceInterface} from "../../interfaces";
 import {Manager} from "../../../Subreddit/Manager";
+import CMUser from "../../Common/User/CMUser";
 
 declare global {
     declare namespace Express {
         interface Request {
             botApp: App;
             token?: string,
-            instance?: CMInstance,
+            instance?: CMInstanceInterface,
             bot?: BotInstance,
             serverBot: Bot,
             manager?: Manager,
         }
-        interface User {
-            name: string
-            subreddits: string[]
-            machine?: boolean
-            isOperator?: boolean
-            realManagers?: string[]
-            moderatedManagers?: string[]
-            realBots?: string[]
-            moderatedBots?: string[]
-            scope?: string[]
-            token?: string
-            tokenExpiresAt?: number
+        class User extends CMUser<any, any, any> {
         }
     }
 }

src/index.ts

@@ -64,7 +64,8 @@ const program = new Command();
             .allowUnknownOption();
         runCommand = addOptions(runCommand, getUniversalWebOptions());
         runCommand.action(async (interfaceVal, opts) => {
-            const config = buildOperatorConfigWithDefaults(await parseOperatorConfigFromSources({...opts, mode: interfaceVal}));
+            const [opConfig, fileConfig] = await parseOperatorConfigFromSources({...opts, mode: interfaceVal});
+            const config = buildOperatorConfigWithDefaults(opConfig);
             const {
                 mode,
             } = config;
@@ -73,7 +74,7 @@ const program = new Command();
                     await clientServer(config);
                 }
                 if(mode === 'all' || mode === 'server') {
-                    await apiServer(config);
+                    await apiServer({...config, fileConfig});
                 }
             } catch (err: any) {
                 throw err;
@@ -92,9 +93,10 @@ const program = new Command();
         checkCommand
             .addOption(checks)
             .action(async (activityIdentifier, type, botVal, commandOptions = {}) => {
-                const config = buildOperatorConfigWithDefaults(await parseOperatorConfigFromSources(commandOptions));
+                const [opConfig, fileConfig] = await parseOperatorConfigFromSources(commandOptions);
+                const config = buildOperatorConfigWithDefaults(opConfig);
                 const {checks = []} = commandOptions;
-                app = new App(config);
+                app = new App({...config, fileConfig});
 
                 let a;
                 const commentId = commentReg(activityIdentifier);
@@ -168,7 +170,8 @@ const program = new Command();
         unmodCommand
             .addOption(checks)
             .action(async (subreddits = [], botVal, opts = {}) => {
-                const config = buildOperatorConfigWithDefaults(await parseOperatorConfigFromSources(opts));
+                const [opConfig, fileConfig] = await parseOperatorConfigFromSources(opts);
+                const config = buildOperatorConfigWithDefaults(opConfig);
                 const {checks = []} = opts;
                 const logger = winston.loggers.get('app');
                 let bots: Bot[] = [];
@@ -201,10 +204,7 @@ const program = new Command();
 
     } catch (err: any) {
         if (!err.logged && !(err instanceof LoggedError)) {
-            const logger = winston.loggers.get('app');
-            if(isScopeError(err)) {
-                logger.error('Reddit responded with a 403 insufficient_scope which means the bot is lacking necessary OAUTH scopes to perform general actions.');
-            }
+            const logger = winston.loggers.has('app') ? winston.loggers.get('app') : winston.loggers.get('init');
             logger.error(err);
         }
         process.kill(process.pid, 'SIGTERM');

src/util.ts

@@ -1,13 +1,11 @@
 import winston, {Logger} from "winston";
 import jsonStringify from 'safe-stable-stringify';
 import dayjs, {Dayjs, OpUnitType} from 'dayjs';
-import {FormattedRuleResult, isRuleSetResult, RulePremise, RuleResult, RuleSetResult} from "./Rule";
+import {FormattedRuleResult, isRuleSetResult, RulePremise, RuleResult, RuleSetResult, UserNoteCriteria} from "./Rule";
 import deepEqual from "fast-deep-equal";
 import {Duration} from 'dayjs/plugin/duration.js';
 import Ajv from "ajv";
 import {InvalidOptionArgumentError} from "commander";
-import Submission from "snoowrap/dist/objects/Submission";
-import {Comment} from "snoowrap";
 import {inflateSync, deflateSync} from "zlib";
 import pixelmatch from 'pixelmatch';
 import os from 'os';
@@ -15,7 +13,7 @@ import {
     ActivityWindowCriteria, ActivityWindowType,
     CacheOptions,
     CacheProvider,
-    DurationComparison, DurationVal,
+    DurationComparison, DurationVal, FilterCriteriaPropertyResult, FilterCriteriaResult,
     GenericComparison,
     HistoricalStats,
     HistoricalStatsDisplay, ImageComparisonResult,
@@ -23,42 +21,46 @@ import {
     ImageDetection,
     //ImageDownloadOptions,
     LogInfo,
-    NamedGroup,
+    NamedGroup, OperatorJsonConfig,
     PollingOptionsStrong,
     RedditEntity,
     RedditEntityType,
     RegExResult, RepostItem, RepostItemResult,
     ResourceStats, SearchAndReplaceRegExp,
-    StatusCodeError, StringComparisonOptions,
+    StringComparisonOptions,
     StringOperator,
     StrongSubredditState,
     SubredditState
 } from "./Common/interfaces";
-import JSON5 from "json5";
-import yaml, {JSON_SCHEMA} from "js-yaml";
-import SimpleError from "./Utils/SimpleError";
+import { Document as YamlDocument } from 'yaml'
 import InvalidRegexError from "./Utils/InvalidRegexError";
 import {constants, promises} from "fs";
-import {cacheOptDefaults} from "./Common/defaults";
+import {cacheOptDefaults, VERSION} from "./Common/defaults";
 import cacheManager, {Cache} from "cache-manager";
 import redisStore from "cache-manager-redis-store";
 import crypto from "crypto";
 import Autolinker from 'autolinker';
 import {create as createMemoryStore} from './Utils/memoryStore';
-import {MESSAGE} from "triple-beam";
-import {RedditUser} from "snoowrap/dist/objects";
+import {MESSAGE, LEVEL} from "triple-beam";
+import {RedditUser,Comment,Submission} from "snoowrap/dist/objects";
 import reRegExp from '@stdlib/regexp-regexp';
 import fetch, {Response} from "node-fetch";
 import { URL } from "url";
 import ImageData from "./Common/ImageData";
 import {Sharp, SharpOptions} from "sharp";
-// @ts-ignore
-import {blockhashData, hammingDistance} from 'blockhash';
-import {SetRandomInterval} from "./Common/types";
+import {ErrorWithCause, stackWithCauses} from "pony-cause";
+import {ConfigFormat, SetRandomInterval} from "./Common/types";
 import stringSimilarity from 'string-similarity';
 import calculateCosineSimilarity from "./Utils/StringMatching/CosineSimilarity";
 import levenSimilarity from "./Utils/StringMatching/levenSimilarity";
-import {isRequestError, isStatusError} from "./Utils/Errors";
+import {SimpleError, isRateLimitError, isRequestError, isScopeError, isStatusError, CMError} from "./Utils/Errors";
+import {parse} from "path";
+import JsonConfigDocument from "./Common/Config/JsonConfigDocument";
+import YamlConfigDocument from "./Common/Config/YamlConfigDocument";
+import AbstractConfigDocument, {ConfigDocumentInterface} from "./Common/Config/AbstractConfigDocument";
+import LoggedError from "./Utils/LoggedError";
+
+
 //import {ResembleSingleCallbackComparisonResult} from "resemblejs";
 
 // want to guess how many concurrent image comparisons we should be doing
@@ -85,36 +87,78 @@ const CWD = process.cwd();
 //     }
 // }
 const errorAwareFormat = {
-    transform: (info: any, opts: any) => {
-        // don't need to log stack trace if we know the error is just a simple message (we handled it)
-        const stack = !(info instanceof SimpleError) && !(info.message instanceof SimpleError);
-        const {name, response, message, stack: errStack, error, statusCode} = info;
-        if(name === 'StatusCodeError' && response !== undefined && response.headers !== undefined && response.headers['content-type'].includes('html')) {
-            // reddit returns html even when we specify raw_json in the querystring (via snoowrap)
-            // which means the html gets set as the message for the error AND gets added to the stack as the message
-            // and we end up with a h u g e log statement full of noisy html >:(
+    transform: (einfo: any, {stack = true}: any = {}) => {
 
-            const errorSample = error.slice(0, 10);
-            const messageBeforeIndex = message.indexOf(errorSample);
-            let newMessage = `Status Error ${statusCode} from Reddit`;
-            if(messageBeforeIndex > 0) {
-                newMessage = `${message.slice(0, messageBeforeIndex)} - ${newMessage}`;
-            }
-            let cleanStack = errStack;
+        // because winston logger.child() re-assigns its input to an object ALWAYS the object we recieve here will never actually be of type Error
+        const includeStack = stack && (!isProbablyError(einfo, 'simpleerror') && !isProbablyError(einfo.message, 'simpleerror'));
 
-            // try to get just stacktrace by finding beginning of what we assume is the actual trace
-            if(errStack) {
-                cleanStack = `${newMessage}\n${errStack.slice(errStack.indexOf('at new StatusCodeError'))}`;
-            }
-            // now put it all together so its nice and clean
-            info.message = newMessage;
-            info.stack = cleanStack;
+        if (!isProbablyError(einfo.message) && !isProbablyError(einfo)) {
+            return einfo;
         }
-        return errors().transform(info, { stack });
+
+        let info: any = {};
+
+        if (isProbablyError(einfo)) {
+            const tinfo = transformError(einfo);
+            info = Object.assign({}, tinfo, {
+                // @ts-ignore
+                level: einfo.level,
+                // @ts-ignore
+                [LEVEL]: einfo[LEVEL] || einfo.level,
+                message: tinfo.message,
+                // @ts-ignore
+                [MESSAGE]: tinfo[MESSAGE] || tinfo.message
+            });
+            if(includeStack) {
+                // so we have to create a dummy error and re-assign all error properties from our info object to it so we can get a proper stack trace
+                const dummyErr = new ErrorWithCause('');
+                for(const k in tinfo) {
+                    if(dummyErr.hasOwnProperty(k) || k === 'cause') {
+                        // @ts-ignore
+                        dummyErr[k] = tinfo[k];
+                    }
+                }
+                // @ts-ignore
+                info.stack = stackWithCauses(dummyErr);
+            }
+        } else {
+            const err = transformError(einfo.message);
+            info = Object.assign(einfo, err);
+            // @ts-ignore
+            info.message = err.message;
+            // @ts-ignore
+            info[MESSAGE] = err.message;
+
+            if(includeStack) {
+                const dummyErr = new ErrorWithCause('');
+                for(const k in err) {
+                    if(dummyErr.hasOwnProperty(k) || k === 'cause') {
+                        // @ts-ignore
+                        dummyErr[k] = info[k];
+                    }
+                }
+                // @ts-ignore
+                info.stack = stackWithCauses(dummyErr);
+            }
+        }
+
+        // remove redundant message from stack and make stack causes easier to read
+        if(info.stack !== undefined) {
+            let cleanedStack = info.stack.replace(info.message, '');
+            cleanedStack = `${cleanedStack}`;
+            cleanedStack = cleanedStack.replaceAll('caused by:', '\ncaused by:');
+            info.stack = cleanedStack;
+        }
+
+        return info;
     }
 }
 
-export const PASS = '✔';
+const isProbablyError = (val: any, errName = 'error') => {
+    return typeof val === 'object' && val.name !== undefined && val.name.toLowerCase().includes(errName);
+}
+
+export const PASS = '✓';
 export const FAIL = '✘';
 
 export const truncateStringToLength = (length: number, truncStr = '...') => (str: string) => str.length > length ? `${str.slice(0, length - truncStr.length - 1)}${truncStr}` : str;
@@ -289,13 +333,39 @@ export const mergeArr = (objValue: [], srcValue: []): (any[] | undefined) => {
     }
 }
 
+export const removeFromSourceIfKeysExistsInDestination = (destinationArray: any[], sourceArray: any[], options: any): any[] => {
+    // get all keys from objects in destination
+    const destKeys = destinationArray.reduce((acc: string[], curr) => {
+        // can only get keys for objects, skip for everything else
+        if(curr !== null && typeof curr === 'object') {
+            const keys = Object.keys(curr).map(x => x.toLowerCase());
+            for(const k of keys) {
+                if(!acc.includes(k)) {
+                    acc.push(k);
+                }
+            }
+        }
+        return acc;
+    }, []);
+    const sourceItemsToKeep = sourceArray.filter(x => {
+        if(x !== null && typeof x === 'object') {
+            const sourceKeys = Object.keys(x).map(x => x.toLowerCase());
+            // only keep if keys from this object do not appear anywhere in destination items
+            return intersect(sourceKeys, destKeys).length === 0;
+        }
+        // keep if item is not an object since we can't test for keys anyway
+        return true;
+    });
+    return sourceItemsToKeep.concat(destinationArray);
+}
+
 export const ruleNamesFromResults = (results: RuleResult[]) => {
     return results.map(x => x.name || x.premise.kind).join(' | ')
 }
 
-export const triggeredIndicator = (val: boolean | null): string => {
+export const triggeredIndicator = (val: boolean | null, nullResultIndicator = '-'): string => {
     if(val === null) {
-        return '-';
+        return nullResultIndicator;
     }
     return val ? PASS : FAIL;
 }
@@ -312,6 +382,40 @@ export const resultsSummary = (results: (RuleResult|RuleSetResult)[], topLevelCo
     //return results.map(x => x.name || x.premise.kind).join(' | ')
 }
 
+export const filterCriteriaSummary = (val: FilterCriteriaResult<any>): [string, string[]] => {
+    // summarize properties relevant to result
+    const passedProps = {props: val.propertyResults.filter(x => x.passed === true), name: 'Passed'};
+    const failedProps = {props: val.propertyResults.filter(x => x.passed === false), name: 'Failed'};
+    const skippedProps = {props: val.propertyResults.filter(x => x.passed === null), name: 'Skipped'};
+    const dnrProps = {props: val.propertyResults.filter(x => x.passed === undefined), name: 'DNR'};
+
+    const propSummary = [passedProps, failedProps];
+    if (skippedProps.props.length > 0) {
+        propSummary.push(skippedProps);
+    }
+    if (dnrProps.props.length > 0) {
+        propSummary.push(dnrProps);
+    }
+    const propSummaryStrArr = propSummary.map(x => `${x.props.length} ${x.name}${x.props.length > 0 ? ` (${x.props.map(y => y.property as string)})` : ''}`);
+    return [propSummaryStrArr.join(' | '), val.propertyResults.map(filterCriteriaPropertySummary)]
+}
+
+export const filterCriteriaPropertySummary = (val: FilterCriteriaPropertyResult<any>): string => {
+    let passResult: string;
+    switch (val.passed) {
+        case undefined:
+            passResult = 'DNR'
+            break;
+        case null:
+        case true:
+        case false:
+            passResult = triggeredIndicator(val.passed, 'Skipped');
+            break;
+    }
+    const found = val.passed === null || val.passed === undefined ? '' : ` => Found: ${val.found}${val.reason !== undefined ? ` -- ${val.reason}` : ''}${val.behavior === 'exclude' ? ' (Exclude passes when Expected is not Found)' : ''}`;
+    return `${val.property as string} => ${passResult} => Expected: ${val.expected}${found}`;
+}
+
 export const createAjvFactory = (logger: Logger) => {
     return  new Ajv({logger: logger, verbose: true, strict: "log", allowUnionTypes: true});
 }
@@ -458,34 +562,64 @@ export const isActivityWindowCriteria = (val: any): val is ActivityWindowCriteri
     return false;
 }
 
-export const parseFromJsonOrYamlToObject = (content: string): [object?, Error?, Error?] => {
+export interface ConfigToObjectOptions {
+    location?: string,
+    jsonDocFunc?: (content: string, location?: string) => AbstractConfigDocument<OperatorJsonConfig>,
+    yamlDocFunc?: (content: string, location?: string) => AbstractConfigDocument<YamlDocument>
+}
+
+export const parseFromJsonOrYamlToObject = (content: string, options?: ConfigToObjectOptions): [ConfigFormat, ConfigDocumentInterface<YamlDocument | object>?, Error?, Error?] => {
     let obj;
+    let configFormat: ConfigFormat = 'yaml';
     let jsonErr,
         yamlErr;
 
+    const likelyType = likelyJson5(content) ? 'json' : 'yaml';
+
+    const {
+        location,
+        jsonDocFunc = (content: string, location?: string) => new JsonConfigDocument(content, location),
+        yamlDocFunc = (content: string, location?: string) => new YamlConfigDocument(content, location),
+    } = options || {};
+
     try {
-        obj = JSON5.parse(content);
-        const oType = obj === null ? 'null' : typeof obj;
+        const jsonObj = jsonDocFunc(content, location);
+        const output = jsonObj.toJS();
+        const oType = output === null ? 'null' : typeof output;
         if (oType !== 'object') {
             jsonErr = new SimpleError(`Parsing as json produced data of type '${oType}' (expected 'object')`);
             obj = undefined;
+        } else {
+            obj = jsonObj;
+            configFormat = 'json';
         }
     } catch (err: any) {
         jsonErr = err;
     }
-    if (obj === undefined) {
-        try {
-            obj = yaml.load(content, {schema: JSON_SCHEMA, json: true});
-            const oType = obj === null ? 'null' : typeof obj;
-            if (oType !== 'object') {
-                yamlErr = new SimpleError(`Parsing as yaml produced data of type '${oType}' (expected 'object')`);
-                obj = undefined;
+
+    try {
+        const yamlObj = yamlDocFunc(content, location)
+        const output = yamlObj.toJS();
+        const oType = output === null ? 'null' : typeof output;
+        if (oType !== 'object') {
+            yamlErr = new SimpleError(`Parsing as yaml produced data of type '${oType}' (expected 'object')`);
+            obj = undefined;
+        } else if (obj === undefined && (likelyType !== 'json' || yamlObj.parsed.errors.length === 0)) {
+            configFormat = 'yaml';
+            if(yamlObj.parsed.errors.length !== 0) {
+                yamlErr = new Error(yamlObj.parsed.errors.join('\n'))
+            } else {
+                obj = yamlObj;
             }
-        } catch (err: any) {
-            yamlErr = err;
         }
+    } catch (err: any) {
+        yamlErr = err;
     }
-    return [obj, jsonErr, yamlErr];
+
+    if (obj === undefined) {
+        configFormat = likelyType;
+    }
+    return [configFormat, obj, jsonErr, yamlErr];
 }
 
 export const comparisonTextOp = (val1: number, strOp: string, val2: number): boolean => {
@@ -662,6 +796,126 @@ export const parseExternalUrl = (val: string) => {
     return (matches.groups as any).url as string;
 }
 
+export const dummyLogger = {
+    debug: (v: any) => null,
+    error: (v: any) => null,
+    warn: (v: any) => null,
+    info: (v: any) => null
+}
+
+const GIST_REGEX = new RegExp(/.*gist\.github\.com\/.+\/(.+)/i)
+const GH_BLOB_REGEX = new RegExp(/.*github\.com\/(.+)\/(.+)\/blob\/(.+)/i);
+const REGEXR_REGEX = new RegExp(/^.*((regexr\.com)\/[\w\d]+).*$/i);
+const REGEXR_PAGE_REGEX = new RegExp(/(.|[\n\r])+"expression":"(.+)","text"/g);
+export const fetchExternalUrl = async (url: string, logger: (any) = dummyLogger): Promise<string> => {
+    let hadError = false;
+    logger.debug(`Attempting to detect resolvable URL for ${url}`);
+    let match = url.match(GIST_REGEX);
+    if (match !== null) {
+        const gistApiUrl = `https://api.github.com/gists/${match[1]}`;
+        logger.debug(`Looks like a non-raw gist URL! Trying to resolve ${gistApiUrl}`);
+
+        try {
+            const response = await fetch(gistApiUrl);
+            if (!response.ok) {
+                logger.error(`Response was not OK from Gist API (${response.statusText}) -- will return response from original URL instead`);
+                if (response.size > 0) {
+                    logger.error(await response.text())
+                }
+                hadError = true;
+            } else {
+                const data = await response.json();
+                // get first found file
+                const fileKeys = Object.keys(data.files);
+                if (fileKeys.length === 0) {
+                    logger.error(`No files found in gist!`);
+                } else {
+                    if (fileKeys.length > 1) {
+                        logger.warn(`More than one file found in gist! Using first found: ${fileKeys[0]}`);
+                    } else {
+                        logger.debug(`Using file ${fileKeys[0]}`);
+                    }
+                    const file = data.files[fileKeys[0]];
+                    if (file.truncated === false) {
+                        return file.content;
+                    }
+                    const rawUrl = file.raw_url;
+                    logger.debug(`File contents was truncated, retrieving full contents from ${rawUrl}`);
+                    try {
+                        const rawUrlResponse = await fetch(rawUrl);
+                        return await rawUrlResponse.text();
+                    } catch (err: any) {
+                        logger.error('Gist Raw URL Response returned an error, will return response from original URL instead');
+                        logger.error(err);
+                    }
+                }
+            }
+        } catch (err: any) {
+            logger.error('Response returned an error, will return response from original URL instead');
+            logger.error(err);
+        }
+    }
+    match = url.match(GH_BLOB_REGEX);
+
+    if (match !== null) {
+        const rawUrl = `https://raw.githubusercontent.com/${match[1]}/${match[2]}/${match[3]}`
+        logger.debug(`Looks like a single file github URL! Resolving to ${rawUrl}`);
+        try {
+            const response = await fetch(rawUrl);
+            if (!response.ok) {
+                logger.error(`Response was not OK (${response.statusText}) -- will return response from original URL instead`);
+                if (response.size > 0) {
+                    logger.error(await response.text())
+                }
+                hadError = true;
+            } else {
+                return await response.text();
+            }
+        } catch (err: any) {
+            logger.error('Response returned an error, will return response from original URL instead');
+            logger.error(err);
+        }
+    }
+
+    match = url.match(REGEXR_REGEX);
+    if(match !== null) {
+        logger.debug(`Looks like a Regexr URL! Trying to get expression from page HTML`);
+        try {
+            const response = await fetch(url);
+            if (!response.ok) {
+                if (response.size > 0) {
+                    logger.error(await response.text())
+                }
+                throw new Error(`Response was not OK: ${response.statusText}`);
+            } else {
+                const page = await response.text();
+                const pageMatch = [...page.matchAll(REGEXR_PAGE_REGEX)];
+                if(pageMatch.length > 0) {
+                    const unescaped = JSON.parse(`{"value": "${pageMatch[0][2]}"}`)
+                    return unescaped.value;
+                } else {
+                    throw new Error('Could not parse regex expression from page HTML');
+                }
+            }
+        } catch (err: any) {
+            logger.error('Response returned an error');
+            throw err;
+        }
+    }
+
+    if(!hadError) {
+        logger.debug('URL was not special (gist, github blob, etc...) so will retrieve plain contents');
+    }
+    const response = await fetch(url);
+    if(!response.ok) {
+        if (response.size > 0) {
+            logger.error(await response.text())
+        }
+        throw new Error(`Response was not OK: ${response.statusText}`);
+    }
+    return await response.text();
+}
+
 export interface RetryOptions {
     maxRequestRetry: number,
     maxOtherRetry: number,
@@ -685,6 +939,11 @@ export const createRetryHandler = (opts: RetryOptions, logger: Logger) => {
 
         lastErrorAt = dayjs();
 
+        if(isRateLimitError(err)) {
+            logger.error('Will not retry because error was due to ratelimit exhaustion');
+            return false;
+        }
+
         const redditApiError = isRequestError(err) || isStatusError(err);
 
         if(redditApiError) {
@@ -722,6 +981,119 @@ export const createRetryHandler = (opts: RetryOptions, logger: Logger) => {
     }
 }
 
+type StringReturn = (err:any) => string;
+
+export interface LogMatch {
+    [key: string | number]: string | StringReturn
+}
+
+export interface logExceptionOptions {
+    context?: string
+    logIfNotMatched?: boolean
+    logStackTrace?: boolean
+    match?: LogMatch
+}
+
+export const parseMatchMessage = (err: any, match: LogMatch, matchTypes: (string | number)[], defaultMatch: string): [string, boolean] => {
+    for(const m of matchTypes) {
+        if(match[m] !== undefined) {
+            if(typeof match[m] === 'string') {
+                return [match[m] as string, true];
+            }
+            return [(match[m] as Function)(err), true];
+        }
+    }
+    return [defaultMatch, false];
+}
+
+export const getExceptionMessage = (err: any, match: LogMatch = {}): string | undefined => {
+
+    let matched = false,
+        matchMsg;
+
+    if (isRequestError(err)) {
+        if (isRateLimitError(err)) {
+            ([matchMsg, matched] = parseMatchMessage(err, match, ['ratelimit', err.statusCode], 'Ratelimit Exhausted'));
+        } else if (isScopeError(err)) {
+            ([matchMsg, matched] = parseMatchMessage(err, match, ['scope', err.statusCode], 'Missing OAUTH scope required for this request'));
+        } else {
+            ([matchMsg, matched] = parseMatchMessage(err, match, [err.statusCode], err.message));
+        }
+    } else {
+        ([matchMsg, matched] = parseMatchMessage(err, match, ['any'], err.message));
+    }
+
+    if (matched) {
+        return matchMsg;
+    }
+}
+
+const _transformError = (err: Error, seen: Set<Error>, matchOptions?: LogMatch) => {
+    if (!err || !isProbablyError(err)) {
+        return '';
+    }
+    if (seen.has(err)) {
+        return err;
+    }
+
+    // @ts-ignore
+    let mOpts = err.matchOptions ?? matchOptions;
+
+    if (isRequestError(err)) {
+        const errMsgParts = [`Reddit responded with a NOT OK status (${err.statusCode})`];
+
+        if (err.response.headers['content-type'].includes('html')) {
+            // reddit returns html even when we specify raw_json in the querystring (via snoowrap)
+            // which means the html gets set as the message for the error AND gets added to the stack as the message
+            // and we end up with a h u g e log statement full of noisy html >:(
+
+            const {error, statusCode, message, stack: errStack} = err;
+
+            const errorSample = (error as unknown as string).slice(0, 10);
+            const messageBeforeIndex = message.indexOf(errorSample);
+            let newMessage = `Status Error ${statusCode} from Reddit`;
+            if (messageBeforeIndex > 0) {
+                newMessage = `${message.slice(0, messageBeforeIndex)} - ${newMessage}`;
+            }
+            let cleanStack = errStack;
+
+            // try to get just stacktrace by finding beginning of what we assume is the actual trace
+            if (errStack) {
+                cleanStack = `${newMessage}\n${errStack.slice(errStack.indexOf('at new StatusCodeError'))}`;
+            }
+            // now put it all together so its nice and clean
+            err.message = newMessage;
+            err.stack = cleanStack;
+        }
+
+        const msg = getExceptionMessage(err, mOpts);
+        if (msg !== undefined) {
+            errMsgParts.push(msg);
+        }
+
+        // we don't care about stack trace for this error because we know where it came from so truncate to two lines for now...maybe remove all together later
+        if(err.stack !== undefined) {
+            err.stack = err.stack.split('\n').slice(0, 2).join('\n');
+        }
+
+        const normalizedError = new ErrorWithCause(errMsgParts.join(' => '), {cause: err});
+        normalizedError.stack = normalizedError.message;
+        return normalizedError;
+    }
+
+    // @ts-ignore
+    const cause = err.cause as unknown;
+
+    if (cause !== undefined && cause instanceof Error) {
+        // @ts-ignore
+        err.cause = _transformError(cause, seen, mOpts);
+    }
+
+    return err;
+}
+
+export const transformError = (err: Error): any => _transformError(err, new Set());
+
 const LABELS_REGEX: RegExp = /(\[.+?])*/g;
 export const parseLabels = (log: string): string[] => {
     return Array.from(log.matchAll(LABELS_REGEX), m => m[0]).map(x => x.substring(1, x.length - 1));
@@ -767,7 +1139,8 @@ export const isLogLineMinLevel = (log: string | LogInfo, minLevelText: string):
 
 // https://regexr.com/3e6m0
 const HYPERLINK_REGEX: RegExp = /(http(s)?:\/\/.)?(www\.)?[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)/;
-export const formatLogLineToHtml = (log: string | LogInfo) => {
+const formattedTime = (short: string, full: string) => `<span class="has-tooltip"><span style="margin-top:35px" class='tooltip rounded shadow-lg p-1 bg-gray-100 text-black space-y-3 p-2 text-left'>${full}</span><span>${short}</span></span>`;
+export const formatLogLineToHtml = (log: string | LogInfo, timestamp?: string) => {
     const val = typeof log === 'string' ? log : log[MESSAGE];
     const logContent = Autolinker.link(val, {
         email: false,
@@ -784,31 +1157,48 @@ export const formatLogLineToHtml = (log: string | LogInfo) => {
         .replace(/(\s*verbose\s*):/i, '<span class="error purple">$1</span>:')
         .replaceAll('\n', '<br />');
         //.replace(HYPERLINK_REGEX, '<a target="_blank" href="$&">$&</a>');
-    return `<div class="logLine">${logContent}</div>`
+    let line = '';
+
+    let timestampString = timestamp;
+    if(timestamp === undefined && typeof log !== 'string') {
+        timestampString = (log as LogInfo).timestamp;
+    }
+
+    if(timestampString !== undefined) {
+        const timeStampReplacement = formattedTime(dayjs(timestampString).format('HH:mm:ss z'), timestampString);
+        const splitLine = logContent.split(timestampString);
+        line = `<div class="logLine">${splitLine[0]}${timeStampReplacement}<span style="white-space: pre-wrap">${splitLine[1]}</span></div>`;
+    } else {
+        line = `<div style="white-space: pre-wrap" class="logLine">${logContent}</div>`
+    }
+    return line;
 }
 
 export type LogEntry = [number, LogInfo];
 export interface LogOptions {
-    limit: number,
+    limit: number | string,
     level: string,
     sort: 'ascending' | 'descending',
     operator?: boolean,
     user?: string,
     allLogsParser?: Function
-    allLogName?: string
+    allLogName?: string,
+    returnType?: 'string' | 'object'
 }
 
-export const filterLogBySubreddit = (logs: Map<string, LogEntry[]>, validLogCategories: string[] = [], options: LogOptions): Map<string, string[]> => {
+export const filterLogBySubreddit = (logs: Map<string, LogEntry[]>, validLogCategories: string[] = [], options: LogOptions): Map<string, (string|LogInfo)[]> => {
     const {
-        limit,
+        limit: limitVal,
         level,
         sort,
         operator = false,
         user,
         allLogsParser = parseSubredditLogInfoName,
-        allLogName = 'app'
+        allLogName = 'app',
+        returnType = 'string',
     } = options;
 
+    let limit = typeof limitVal === 'number' ? limitVal : Number.parseInt(limitVal);
     // get map of valid logs categories
     const validSubMap: Map<string, LogEntry[]> = new Map();
     for(const [k, v] of logs) {
@@ -840,19 +1230,53 @@ export const filterLogBySubreddit = (logs: Map<string, LogEntry[]>, validLogCate
 
     const sortFunc = sort === 'ascending' ? (a: LogEntry, b: LogEntry) => a[0] - b[0] : (a: LogEntry, b: LogEntry) => b[0] - a[0];
 
-    const preparedMap: Map<string, string[]> = new Map();
+    const preparedMap: Map<string, (string|LogInfo)[]> = new Map();
     // iterate each entry and
     // sort, filter by level, slice to limit, then map to html string
     for(const [k,v] of validSubMap.entries()) {
         let preparedEntries = v.filter(([time, l]) => isLogLineMinLevel(l, level));
         preparedEntries.sort(sortFunc);
-        preparedMap.set(k, preparedEntries.slice(0, limit + 1).map(([time, l]) => formatLogLineToHtml(l)));
+        const entriesSlice = preparedEntries.slice(0, limit + 1);
+        if(returnType === 'string') {
+            preparedMap.set(k, entriesSlice.map(([time, l]) => formatLogLineToHtml(l)));
+        } else {
+            preparedMap.set(k, entriesSlice.map(([time, l]) => l));
+        }
     }
 
 
     return preparedMap;
 }
 
+export const logSortFunc = (sort: string = 'ascending') => sort === 'ascending' ? (a: LogInfo, b: LogInfo) => (dayjs(a.timestamp).isSameOrAfter(b.timestamp) ? 1 : -1) : (a: LogInfo, b: LogInfo) => (dayjs(a.timestamp).isSameOrBefore(b.timestamp) ? 1 : -1);
+
+export const filterLogs= (logs: LogInfo[], options: LogOptions): LogInfo[] | string[] => {
+    const {
+        limit: limitVal,
+        level,
+        sort,
+        operator = false,
+        user,
+        allLogsParser = parseSubredditLogInfoName,
+        allLogName = 'app',
+        returnType = 'string',
+    } = options;
+
+    let limit = typeof limitVal === 'number' ? limitVal : Number.parseInt(limitVal);
+    let leveledLogs = logs.filter(x => isLogLineMinLevel(x, level));
+    if(user !== undefined) {
+        leveledLogs = logs.filter(x => x.user !== undefined && x.user === user);
+    }
+    leveledLogs.sort(logSortFunc(sort));
+    leveledLogs = leveledLogs.slice(0, limit + 1);
+
+    if(returnType === 'string') {
+        return leveledLogs.map(x => formatLogLineToHtml(x));
+    } else {
+        return leveledLogs;
+    }
+}
+
 export const logLevels = {
     error: 0,
     warn: 1,
@@ -996,6 +1420,8 @@ export const toStrongSubredditState = (s: SubredditState, opts?: StrongSubreddit
 
     if (generateDescription && stateDescription === undefined) {
         strongState.stateDescription = objectToStringSummary(strongState);
+    } else {
+        strongState.stateDescription = stateDescription;
     }
 
     return strongState;
@@ -1008,19 +1434,25 @@ export const convertSubredditsRawToStrong = (x: (SubredditState | string), opts:
     return toStrongSubredditState(x, opts);
 }
 
-export async function readConfigFile(path: string, opts: any) {
+export async function readConfigFile(path: string, opts: any): Promise<[string?, ConfigFormat?]> {
     const {log, throwOnNotFound = true} = opts;
+    let extensionHint: ConfigFormat | undefined;
+    const fileInfo = parse(path);
+    if(fileInfo.ext !== undefined) {
+        switch(fileInfo.ext) {
+            case '.json':
+            case '.json5':
+                extensionHint = 'json';
+                break;
+            case '.yaml':
+                extensionHint = 'yaml';
+                break;
+        }
+    }
     try {
         await promises.access(path, constants.R_OK);
         const data = await promises.readFile(path);
-        const [configObj, jsonErr, yamlErr] = parseFromJsonOrYamlToObject(data as unknown as string);
-        if(configObj !== undefined) {
-            return configObj as object;
-        }
-        log.error(`Could not parse file contents at ${path} as JSON or YAML:`);
-        log.error(jsonErr);
-        log.error(yamlErr);
-        throw new SimpleError(`Could not parse file contents at ${path} as JSON or YAML`);
+        return [(data as any).toString(), extensionHint]
     } catch (e: any) {
         const {code} = e;
         if (code === 'ENOENT') {
@@ -1028,14 +1460,16 @@ export async function readConfigFile(path: string, opts: any) {
                 if (log) {
                     log.warn('No file found at given path', {filePath: path});
                 }
+                e.extension = extensionHint;
                 throw e;
             } else {
-                return;
+                return [];
             }
         } else if (log) {
             log.warn(`Encountered error while parsing file`, {filePath: path});
             log.error(e);
         }
+        e.extension = extensionHint;
         throw e;
     }
 }
@@ -1044,6 +1478,29 @@ export async function readConfigFile(path: string, opts: any) {
 //     return (item && typeof item === 'object' && !Array.isArray(item));
 // }
 
+export const fileOrDirectoryIsWriteable = async (location: string) => {
+    const pathInfo = parse(location);
+    try {
+        await promises.access(location, constants.R_OK | constants.W_OK);
+        return true;
+    } catch (err: any) {
+        const {code} = err;
+        if (code === 'ENOENT') {
+            // file doesn't exist, see if we can write to directory in which case we are good
+            try {
+                await promises.access(pathInfo.dir, constants.R_OK | constants.W_OK)
+                // we can write to dir
+                return true;
+            } catch (accessError: any) {
+                // also can't access directory :(
+                throw new SimpleError(`No file exists at ${location} and application does not have permission to write to that directory`);
+            }
+        } else {
+            throw new SimpleError(`File exists at ${location} but application does have permission to write to it.`);
+        }
+    }
+}
+
 export const overwriteMerge = (destinationArray: any[], sourceArray: any[], options: any): any[] => sourceArray;
 
 export const removeUndefinedKeys = (obj: any) => {
@@ -1153,13 +1610,45 @@ export const snooLogWrapper = (logger: Logger) => {
  * Cached activities lose type information when deserialized so need to check properties as well to see if the object is the shape of a Submission
  * */
 export const isSubmission = (value: any) => {
-    return value instanceof Submission || value.domain !== undefined;
+    return value instanceof Submission || value.name.includes('t3_');
 }
 
 export const asSubmission = (value: any): value is Submission => {
     return isSubmission(value);
 }
 
+export const isComment = (value: any) => {
+    return value instanceof Comment || value.name.includes('t1_');
+}
+
+export const asComment = (value: any): value is Comment => {
+    return isComment(value);
+}
+
+export const asActivity = (value: any): value is (Submission | Comment) => {
+    return asComment(value) || asSubmission(value);
+}
+
+export const isUser = (value: any) => {
+    return value instanceof RedditUser || value.name.includes('t2_');
+}
+
+export const asUser = (value: any): value is RedditUser => {
+    return isUser(value);
+}
+
+export const isUserNoteCriteria = (value: any) => {
+    return value !== null && typeof value === 'object' && value.type !== undefined;
+}
+
+export const asUserNoteCriteria = (value: any): value is UserNoteCriteria => {
+    return isUserNoteCriteria(value);
+}
+
+export const userNoteCriteriaSummary = (val: UserNoteCriteria): string => {
+    return `${val.count === undefined ? '>= 1' : val.count} of ${val.search === undefined ? 'current' : val.search} notes is ${val.type}`;
+}
+
 /**
  * Serialized activities store subreddit and user properties as their string representations (instead of proxy)
  * */
@@ -1584,3 +2073,46 @@ export const likelyJson5 = (str: string): boolean => {
     }
     return validStart;
 }
+
+const defaultScanOptions = {
+    COUNT: '100',
+    MATCH: '*'
+}
+/**
+ * Frankenstein redis scan generator
+ *
+ * Cannot use the built-in scan iterator because it is only available in > v4 of redis client but node-cache-manager-redis is using v3.x --
+ * So combining the async iterator defined in v4 from here https://github.com/redis/node-redis/blob/master/packages/client/lib/client/index.ts#L587
+ * with the scan example from v3 https://github.com/redis/node-redis/blob/8a43dea9bee11e41d33502850f6989943163020a/examples/scan.js
+ *
+ * */
+export async function* redisScanIterator(client: any, options: any = {}): AsyncIterable<string> {
+    let cursor: string = '0';
+    const scanOpts = {...defaultScanOptions, ...options};
+    do {
+        const iterScan = new Promise((resolve, reject) => {
+            client.scan(cursor, 'MATCH', scanOpts.MATCH, 'COUNT', scanOpts.COUNT, (err: any, res: any) => {
+                if(err) {
+                    return reject(err);
+                } else {
+                    const newCursor = res[0];
+                    let keys = res[1];
+                    resolve([newCursor, keys]);
+                }
+            });
+        }) as Promise<[any, string[]]>;
+        const [newCursor, keys] = await iterScan;
+        cursor = newCursor;
+        for (const key of keys) {
+            yield key;
+        }
+    } while (cursor !== '0');
+}
+
+export const getUserAgent = (val: string, fragment?: string) => {
+    return `${replaceApplicationIdentifier(val, fragment)} (developed by /u/FoxxMD)`;
+}
+
+export const replaceApplicationIdentifier = (val: string, fragment?: string) => {
+    return val.replace('{VERSION}', `v${VERSION}`).replace('{FRAG}', (fragment !== undefined ? `-${fragment}` : ''));
+}

tsconfig.json

@@ -3,7 +3,11 @@
   "compilerOptions": {
     "sourceMap": true,
     "resolveJsonModule": true,
-    "typeRoots": ["./src/Web/types"]
+    "typeRoots": [
+      "./node_modules/@types",
+      "./src/Web/types",
+      "./src/Common/typings"
+    ]
   },
 //  "compilerOptions": {
 //    "module": "es6",