Merge branch 'edge'

Bump version
fix(migrations): Remove unnecessary log warning for all logs on live stats
2026-01-14 16:08:02 -05:00 · 2022-02-22 11:11:46 -05:00 · 2022-02-22 11:11:29 -05:00 · 2022-02-22 11:10:51 -05:00 · 2022-02-22 10:43:17 -05:00 · 2022-02-21 16:47:17 -05:00
57 changed files with 1424 additions and 413 deletions
--- a/docs/gettingStartedOperator.md
+++ b/docs/gettingStartedOperator.md
@@ -22,13 +22,14 @@ PROTIP: Using a container management tool like [Portainer.io CE](https://www.por

 ### [Dockerhub](https://hub.docker.com/r/foxxmd/context-mod)

-```
-foxxmd/context-mod:latest
-```
+An example of starting the container using the [minimum configuration](/docs/operatorConfiguration.md#minimum-config) with a [configuration file](/docs/operatorConfiguration.md#defining-configuration-via-file):
+
+* Bind the folder where the config is located on your host machine into the container `-v /host/path/folder:/config`
+* Tell CM where to find the config using an env `-e "OPERATOR_CONFIG=/config/myConfig.yaml"`
+* Expose the web interface using the container port `8085`

-Adding **environmental variables** to your `docker run` command will pass them through to the app EX:
 ```
-docker run -d -e "CLIENT_ID=myId" ... foxxmd/context-mod
+docker run -d -e "OPERATOR_CONFIG=/config/myConfig.yaml" -v /host/path/folder:/config -p 8085:8085 foxxmd/context-mod
 ```

 ### Locally
@@ -47,6 +48,12 @@ npm install
 tsc -p .
 ```

+An example of running CM using the [minimum configuration](/docs/operatorConfiguration.md#minimum-config) with a [configuration file](/docs/operatorConfiguration.md#defining-configuration-via-file):
+
+```bash
+node src/index.js run
+```
+
 ### [Heroku Quick Deploy](https://heroku.com/about)
 [![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://dashboard.heroku.com/new?template=https://github.com/FoxxMD/context-mod)

--- a/docs/operatorConfiguration.md
+++ b/docs/operatorConfiguration.md
@@ -41,8 +41,10 @@ configuration.
 **Note:** When reading the **schema** if the variable is available at a level of configuration other than **FILE** it will be
 noted with the same symbol as above. The value shown is the default.

-* To load a JSON configuration (for **FILE**) **from the command line** use the `-c` cli argument EX: `node src/index.js -c /path/to/JSON/config.json`
-* To load a JSON configuration (for **FILE**) **using an environmental variable** use `OPERATOR_CONFIG` EX: `OPERATOR_CONFIG=/path/to/JSON/config.json`
+## Defining Configuration Via File
+
+* **from the command line**  use the `-c` cli argument EX: `node src/index.js -c /path/to/JSON/config.json`
+* **using an environmental variable**  use `OPERATOR_CONFIG` EX: `OPERATOR_CONFIG=/path/to/JSON/config.json`

 [**See the Operator Config Schema here**](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FOperatorConfig.json)

@@ -121,28 +123,41 @@ Below are examples of the minimum required config to run the application using a
 Using **FILE**
 <details>

+CM will look for a file configuration at `PROJECT_DIR/config.yaml` by default [or you can specify your own location.](#defining-configuration-via-file)
+
 YAML
 ```yaml
+operator:
+  name: YourRedditUsername
 bots:
  - credentials:
      clientId: f4b4df1c7b2
      clientSecret: 34v5q1c56ub
-      refreshToken: 34_f1w1v4
-      accessToken: p75_1c467b2
+web:
+  credentials:
+    clientId: f4b4df1c7b2
+    clientSecret: 34v5q1c56ub
 ```
 JSON
 ```json5
 {
+  "operator": {
+    "name": "YourRedditUsername"
+  },
  "bots": [
    {
      "credentials": {
        "clientId": "f4b4df1c7b2",
-        "clientSecret": "34v5q1c56ub",
-        "refreshToken": "34_f1w1v4",
-        "accessToken": "p75_1c467b2"
+        "clientSecret": "34v5q1c56ub"
      }
    }
-  ]
+  ],
+  "web": {
+    "credentials": {
+      "clientId": "f4b4df1c7b2",
+      "clientSecret": "34v5q1c56ub"
+    }
+  }
 }
 ```

@@ -153,10 +168,9 @@ Using **ENV** (`.env`)
 <details>

 ```
+OPERATOR=YourRedditUsername
 CLIENT_ID=f4b4df1c7b2
 CLIENT_SECRET=34v5q1c56ub
-REFRESH_TOKEN=34_f1w1v4
-ACCESS_TOKEN=p75_1c467b2
 ```

 </details>
--- a/docs/screenshots/actionsEvents.png
+++ b/docs/screenshots/actionsEvents.png
--- a/docs/screenshots/botOperations.png
+++ b/docs/screenshots/botOperations.png
--- a/docs/screenshots/config/config.jpg
+++ b/docs/screenshots/config/config.jpg
--- a/docs/screenshots/config/configUpdate.png
+++ b/docs/screenshots/config/configUpdate.png
--- a/docs/screenshots/config/correctness.png
+++ b/docs/screenshots/config/correctness.png
--- a/docs/screenshots/config/enable.png
+++ b/docs/screenshots/config/enable.png
--- a/docs/screenshots/config/errors.png
+++ b/docs/screenshots/config/errors.png
--- a/docs/screenshots/config/save.png
+++ b/docs/screenshots/config/save.png
--- a/docs/screenshots/config/syntax.png
+++ b/docs/screenshots/config/syntax.png
--- a/docs/screenshots/logs.png
+++ b/docs/screenshots/logs.png
--- a/docs/screenshots/runInput.png
+++ b/docs/screenshots/runInput.png
--- a/docs/webInterface.md
+++ b/docs/webInterface.md
@@ -0,0 +1,30 @@
+## Editing/Updating Your Config
+
+* Open the editor for your subreddit
+    * In the web dashboard \-> r/YourSubreddit \-> Config -> **View** [(here)](/docs/screenshots/config/config.jpg)
+* Follow the directions on the [link at the top of the window](/docs/screenshots/config/save.png) to enable config editing using your moderator account
+    * After enabling editing just click "save" at any time to save your config
+    * After you have added/edited your config the bot will detect changes within 5 minutes or you can manually trigger it by clicking **Update**
+
+## General Config (Editor) Tips
+
+* The editor will automatically validate your [syntax (formatting)](/docs/screenshots/config/syntax.png) and [config correctness](/docs/screenshots/config/correctness.png) (property names, required properties, etc.)
+    * These show up as squiggly lines like in Microsoft Word and as a [list at the bottom of the editor](/docs/screenshots/config/errors.png)
+* In your config all **Checks** and **Actions** have two properties that control how they behave:
+    * [**Enable**](/docs/screenshots/config/enable.png) (defaults to `enable: true`) -- Determines if the check or action is run, at all
+    * **Dryrun** (defaults to `dryRun: false`) -- When `true` the check or action will run but any **Actions** that may be triggered will "pretend" to execute but not actually talk to the Reddit API.
+      * Use `dryRun` to test your config without the bot making any changes on reddit
+    * When starting out with a new config it is recommended running the bot with remove/ban actions **disabled**
+        * Use `report` actions to get reports in your modqueue from the bot that describe what it detected and what it would do about it
+        * Once the bot is behaving as desired (no false positives or weird behavior) destructive actions can be enabled or turned off of dryrun
+
+## Web Dashboard Tips
+
+* Use the [**Overview** section](/docs/screenshots/botOperations.png) to control the bot at a high-level
+* You can **manually run** the bot on any activity (comment/submission) by pasting its permalink into the [input field below the Overview section](/docs/screenshots/runInput.png) and hitting one of the **run buttons**
+    * **Dry run** will make the bot run on the activity but it will only **pretend** to run actions, if triggered. This is super useful for testing your config without consequences
+    * **Run** will do everything
+* All of the bot's activity is shown in real-time in the [log section](/docs/screenshots/logs.png)
+    * This will output the results of all run checks/rules and any actions that run
+* You can view summaries of all activities that triggered a check (had actions run) by clicking on [Actioned Events](/docs/screenshots/actionsEvents.png)
+    * This includes activities run with dry run
--- a/package-lock.json
+++ b/package-lock.json
@@ -34,6 +34,7 @@
        "express-socket.io-session": "^1.3.5",
        "fast-deep-equal": "^3.1.3",
        "fuse.js": "^6.4.6",
+        "globrex": "^0.1.2",
        "got": "^11.8.2",
        "he": "^1.2.0",
        "http-proxy": "^1.18.1",
@@ -80,6 +81,7 @@
        "@types/express": "^4.17.13",
        "@types/express-session": "^1.17.4",
        "@types/express-socket.io-session": "^1.3.6",
+        "@types/globrex": "^0.1.1",
        "@types/he": "^1.1.1",
        "@types/http-proxy": "^1.17.7",
        "@types/js-yaml": "^4.0.1",
@@ -360,6 +362,15 @@
        "@types/redis": "^2.8.0"
      }
    },
+    "node_modules/@types/cache-manager-redis-store/node_modules/@types/redis": {
+      "version": "2.8.32",
+      "resolved": "https://registry.npmjs.org/@types/redis/-/redis-2.8.32.tgz",
+      "integrity": "sha512-7jkMKxcGq9p242exlbsVzuJb57KqHRhNl4dHoQu2Y5v9bCAbtIXXH0R3HleSQW4CTOqpHIYUW3t6tpUj4BVQ+w==",
+      "dev": true,
+      "dependencies": {
+        "@types/node": "*"
+      }
+    },
    "node_modules/@types/cacheable-request": {
      "version": "6.0.2",
      "resolved": "https://registry.npmjs.org/@types/cacheable-request/-/cacheable-request-6.0.2.tgz",
@@ -456,6 +467,12 @@
        "@types/socket.io": "2.1.13"
      }
    },
+    "node_modules/@types/globrex": {
+      "version": "0.1.1",
+      "resolved": "https://registry.npmjs.org/@types/globrex/-/globrex-0.1.1.tgz",
+      "integrity": "sha512-bce8X5Yb8l8ou2VDaEG8CYY1p6NynmswkaasO1pdAzFASKJ43sjf9MQdVH6VmKNG2bPEEmvI5onJJSH+1qOMOA==",
+      "dev": true
+    },
    "node_modules/@types/he": {
      "version": "1.1.2",
      "resolved": "https://registry.npmjs.org/@types/he/-/he-1.1.2.tgz",
@@ -607,15 +624,6 @@
      "integrity": "sha512-EEhsLsD6UsDM1yFhAvy0Cjr6VwmpMWqFBCb9w07wVugF7w9nfajxLuVmngTIpgS6svCnm6Vaw+MZhoDCKnOfsw==",
      "dev": true
    },
-    "node_modules/@types/redis": {
-      "version": "2.8.32",
-      "resolved": "https://registry.npmjs.org/@types/redis/-/redis-2.8.32.tgz",
-      "integrity": "sha512-7jkMKxcGq9p242exlbsVzuJb57KqHRhNl4dHoQu2Y5v9bCAbtIXXH0R3HleSQW4CTOqpHIYUW3t6tpUj4BVQ+w==",
-      "dev": true,
-      "dependencies": {
-        "@types/node": "*"
-      }
-    },
    "node_modules/@types/responselike": {
      "version": "1.0.0",
      "resolved": "https://registry.npmjs.org/@types/responselike/-/responselike-1.0.0.tgz",
@@ -2147,6 +2155,11 @@
        "url": "https://github.com/sponsors/isaacs"
      }
    },
+    "node_modules/globrex": {
+      "version": "0.1.2",
+      "resolved": "https://registry.npmjs.org/globrex/-/globrex-0.1.2.tgz",
+      "integrity": "sha512-uHJgbwAMwNFf5mLst7IWLNg14x1CkeqglJb/K3doi4dw6q2IvAAmM/Y81kevy83wP+Sst+nutFTYOGg3d1lsxg=="
+    },
    "node_modules/google-auth-library": {
      "version": "7.11.0",
      "resolved": "https://registry.npmjs.org/google-auth-library/-/google-auth-library-7.11.0.tgz",
@@ -4931,6 +4944,17 @@
      "requires": {
        "@types/cache-manager": "*",
        "@types/redis": "^2.8.0"
+      },
+      "dependencies": {
+        "@types/redis": {
+          "version": "2.8.32",
+          "resolved": "https://registry.npmjs.org/@types/redis/-/redis-2.8.32.tgz",
+          "integrity": "sha512-7jkMKxcGq9p242exlbsVzuJb57KqHRhNl4dHoQu2Y5v9bCAbtIXXH0R3HleSQW4CTOqpHIYUW3t6tpUj4BVQ+w==",
+          "dev": true,
+          "requires": {
+            "@types/node": "*"
+          }
+        }
      }
    },
    "@types/cacheable-request": {
@@ -5029,6 +5053,12 @@
        "@types/socket.io": "2.1.13"
      }
    },
+    "@types/globrex": {
+      "version": "0.1.1",
+      "resolved": "https://registry.npmjs.org/@types/globrex/-/globrex-0.1.1.tgz",
+      "integrity": "sha512-bce8X5Yb8l8ou2VDaEG8CYY1p6NynmswkaasO1pdAzFASKJ43sjf9MQdVH6VmKNG2bPEEmvI5onJJSH+1qOMOA==",
+      "dev": true
+    },
    "@types/he": {
      "version": "1.1.2",
      "resolved": "https://registry.npmjs.org/@types/he/-/he-1.1.2.tgz",
@@ -5180,15 +5210,6 @@
      "integrity": "sha512-EEhsLsD6UsDM1yFhAvy0Cjr6VwmpMWqFBCb9w07wVugF7w9nfajxLuVmngTIpgS6svCnm6Vaw+MZhoDCKnOfsw==",
      "dev": true
    },
-    "@types/redis": {
-      "version": "2.8.32",
-      "resolved": "https://registry.npmjs.org/@types/redis/-/redis-2.8.32.tgz",
-      "integrity": "sha512-7jkMKxcGq9p242exlbsVzuJb57KqHRhNl4dHoQu2Y5v9bCAbtIXXH0R3HleSQW4CTOqpHIYUW3t6tpUj4BVQ+w==",
-      "dev": true,
-      "requires": {
-        "@types/node": "*"
-      }
-    },
    "@types/responselike": {
      "version": "1.0.0",
      "resolved": "https://registry.npmjs.org/@types/responselike/-/responselike-1.0.0.tgz",
@@ -6401,6 +6422,11 @@
        "path-is-absolute": "^1.0.0"
      }
    },
+    "globrex": {
+      "version": "0.1.2",
+      "resolved": "https://registry.npmjs.org/globrex/-/globrex-0.1.2.tgz",
+      "integrity": "sha512-uHJgbwAMwNFf5mLst7IWLNg14x1CkeqglJb/K3doi4dw6q2IvAAmM/Y81kevy83wP+Sst+nutFTYOGg3d1lsxg=="
+    },
    "google-auth-library": {
      "version": "7.11.0",
      "resolved": "https://registry.npmjs.org/google-auth-library/-/google-auth-library-7.11.0.tgz",
--- a/package.json
+++ b/package.json
@@ -49,6 +49,7 @@
    "express-socket.io-session": "^1.3.5",
    "fast-deep-equal": "^3.1.3",
    "fuse.js": "^6.4.6",
+    "globrex": "^0.1.2",
    "got": "^11.8.2",
    "he": "^1.2.0",
    "http-proxy": "^1.18.1",
@@ -95,6 +96,7 @@
    "@types/express": "^4.17.13",
    "@types/express-session": "^1.17.4",
    "@types/express-socket.io-session": "^1.3.6",
+    "@types/globrex": "^0.1.1",
    "@types/he": "^1.1.1",
    "@types/http-proxy": "^1.17.7",
    "@types/js-yaml": "^4.0.1",
--- a/src/Action/ApproveAction.ts
+++ b/src/Action/ApproveAction.ts
@@ -36,7 +36,7 @@ export class ApproveAction extends Action {
            }

            // @ts-ignore
-            if (item.approved) {
+            if (targetItem.approved) {
                const msg = `${target === 'self' ? 'Item' : 'Comment\'s parent Submission'} is already approved`;
                this.logger.warn(msg);
                return {
@@ -54,6 +54,16 @@ export class ApproveAction extends Action {
                }
                // @ts-ignore
                touchedEntities.push(await targetItem.approve());
+
+                if(target === 'self') {
+                    // @ts-ignore
+                    item.approved = true;
+                    await this.resources.resetCacheForItem(item);
+                } else if(await this.resources.hasActivity(targetItem)) {
+                    // @ts-ignore
+                    targetItem.approved = true;
+                    await this.resources.resetCacheForItem(targetItem);
+                }
            }
        }

--- a/src/Action/LockAction.ts
+++ b/src/Action/LockAction.ts
@@ -26,6 +26,9 @@ export class LockAction extends Action {
            //snoowrap typing issue, thinks comments can't be locked
            // @ts-ignore
            await item.lock();
+            // @ts-ignore
+            item.locked = true;
+            await this.resources.resetCacheForItem(item);
            touchedEntities.push(item);
        }
        return {
--- a/src/Action/RemoveAction.ts
+++ b/src/Action/RemoveAction.ts
@@ -4,6 +4,8 @@ import Snoowrap, {Comment, Submission} from "snoowrap";
 import {RuleResult} from "../Rule";
 import {activityIsRemoved} from "../Utils/SnoowrapUtils";
 import {ActionProcessResult} from "../Common/interfaces";
+import dayjs from "dayjs";
+import {isSubmission} from "../util";

 export class RemoveAction extends Action {
    spam: boolean;
@@ -26,11 +28,7 @@ export class RemoveAction extends Action {
        // issue with snoowrap typings, doesn't think prop exists on Submission
        // @ts-ignore
        if (activityIsRemoved(item)) {
-            return {
-                dryRun,
-                success: false,
-                result: 'Item is already removed',
-            }
+            this.logger.warn('It looks like this Item is already removed!');
        }
        if (this.spam) {
            this.logger.verbose('Marking as spam on removal');
@@ -38,6 +36,13 @@ export class RemoveAction extends Action {
        if (!dryRun) {
            // @ts-ignore
            await item.remove({spam: this.spam});
+            item.banned_at_utc = dayjs().unix();
+            item.spam = this.spam;
+            if(!isSubmission(item)) {
+                // @ts-ignore
+                item.removed = true;
+            }
+            await this.resources.resetCacheForItem(item);
            touchedEntities.push(item);
        }

--- a/src/Action/ReportAction.ts
+++ b/src/Action/ReportAction.ts
@@ -35,6 +35,7 @@ export class ReportAction extends Action {
            await item.report({reason: truncatedContent});
            // due to reddit not updating this in response (maybe)?? just increment stale activity
            item.num_reports++;
+            await this.resources.resetCacheForItem(item);
            touchedEntities.push(item);
        }

--- a/src/Action/SubmissionAction/FlairAction.ts
+++ b/src/Action/SubmissionAction/FlairAction.ts
@@ -33,16 +33,26 @@ export class FlairAction extends Action {
        if(this.css !== '') {
            flairParts.push(`CSS: ${this.css}`);
        }
+        if(this.flair_template_id !== '') {
+            flairParts.push(`Template: ${this.flair_template_id}`);
+        }
        const flairSummary = flairParts.length === 0 ? 'No flair (unflaired)' : flairParts.join(' | ');
        this.logger.verbose(flairSummary);
        if (item instanceof Submission) {
            if(!this.dryRun) {
                if (this.flair_template_id) {
-                    await item.selectFlair({flair_template_id: this.flair_template_id}).then(() => {});
+                    // typings are wrong for this function, flair_template_id should be accepted
+                    // assignFlair uses /api/flair (mod endpoint)
+                    // selectFlair uses /api/selectflair (self endpoint for user to choose their own flair for submission)
+                    // @ts-ignore
+                    await item.assignFlair({flair_template_id: this.flair_template_id}).then(() => {});
+                    item.link_flair_template_id = this.flair_template_id;
                } else {
                    await item.assignFlair({text: this.text, cssClass: this.css}).then(() => {});
+                    item.link_flair_css_class = this.css;
+                    item.link_flair_text = this.text;
                }
-
+                await this.resources.resetCacheForItem(item);
            }
        } else {
            this.logger.warn('Cannot flair Comment');
--- a/src/Action/UserFlairAction.ts
+++ b/src/Action/UserFlairAction.ts
@@ -50,6 +50,7 @@ export class UserFlairAction extends Action {
            flairTemplateId: this.flair_template_id,
            username: item.author.name,
          });
+          item.author_flair_template_id = this.flair_template_id
        } catch (err: any) {
          this.logger.error('Either the flair template ID is incorrect or you do not have permission to access it.');
          throw err;
@@ -57,6 +58,9 @@ export class UserFlairAction extends Action {
      } else if (this.text === undefined && this.css === undefined) {
        // @ts-ignore
        await item.subreddit.deleteUserFlair(item.author.name);
+        item.author_flair_css_class = null;
+        item.author_flair_text = null;
+        item.author_flair_template_id = null;
      } else {
        // @ts-ignore
        await item.author.assignFlair({
@@ -64,7 +68,11 @@ export class UserFlairAction extends Action {
          cssClass: this.css,
          text: this.text,
        });
+        item.author_flair_text = this.text ?? null;
+        item.author_flair_css_class = this.css ?? null;
      }
+      await this.resources.resetCacheForItem(item);
+      await this.resources.resetCacheForItem(item.author);
    }

    return {
--- a/src/Action/index.ts
+++ b/src/Action/index.ts
@@ -7,6 +7,7 @@ import Author, {AuthorOptions} from "../Author/Author";
 import {mergeArr} from "../util";
 import LoggedError from "../Utils/LoggedError";
 import {ExtendedSnoowrap} from '../Utils/SnoowrapClients';
+import {ErrorWithCause} from "pony-cause";

 export abstract class Action {
    name?: string;
@@ -86,7 +87,8 @@ export abstract class Action {
            return {...actRes, ...results};
        } catch (err: any) {
            if(!(err instanceof LoggedError)) {
-                this.logger.error(`Encountered error while running`, err);
+                const actionError = new ErrorWithCause('Action did not run successfully due to unexpected error', {cause: err});
+                this.logger.error(actionError);
            }
            actRes.success = false;
            actRes.result = err.message;
--- a/src/Bot/index.ts
+++ b/src/Bot/index.ts
@@ -6,7 +6,7 @@ import EventEmitter from "events";
 import {
    BotInstanceConfig,
    FilterCriteriaDefaults,
-    Invokee,
+    Invokee, LogInfo,
    PAUSED,
    PollOn,
    RUNNING,
@@ -16,7 +16,7 @@ import {
 } from "../Common/interfaces";
 import {
    createRetryHandler,
-    formatNumber, getExceptionMessage,
+    formatNumber, getExceptionMessage, getUserAgent,
    mergeArr,
    parseBool,
    parseDuration, parseMatchMessage,
@@ -38,6 +38,7 @@ class Bot {

    client!: ExtendedSnoowrap;
    logger!: Logger;
+    logs: LogInfo[] = [];
    wikiLocation: string;
    dryRun?: true | undefined;
    running: boolean = false;
@@ -98,6 +99,7 @@ class Bot {
                dryRun,
                heartbeatInterval,
            },
+            userAgent,
            credentials: {
                reddit: {
                    clientId,
@@ -151,6 +153,12 @@ class Bot {
            }
        }, mergeArr);

+        this.logger.stream().on('log', (log: LogInfo) => {
+            if(log.bot !== undefined && log.bot === this.getBotName() && log.subreddit === undefined) {
+                this.logs = [log, ...this.logs].slice(0, 301);
+            }
+        });
+
        let mw = maxWorkers;
        if(maxWorkers < 1) {
            this.logger.warn(`Max queue workers must be greater than or equal to 1 (Specified: ${maxWorkers})`);
@@ -166,7 +174,9 @@ class Bot {
        this.excludeSubreddits = exclude.map(parseSubredditName);

        let creds: any = {
-            get userAgent() { return getUserName() },
+            get userAgent() {
+                return getUserAgent(`web:contextBot:{VERSION}{FRAG}:BOT-${getBotName()}`, userAgent)
+            },
            clientId,
            clientSecret,
            refreshToken,
--- a/src/Common/defaults.ts
+++ b/src/Common/defaults.ts
@@ -39,3 +39,5 @@ export const filterCriteriaDefault: FilterCriteriaDefaults = {
        ]
    }
 }
+
+export const VERSION = '0.10.12';
--- a/src/Common/interfaces.ts
+++ b/src/Common/interfaces.ts
@@ -949,6 +949,10 @@ export interface SubmissionState extends ActivityState {
    link_flair_text?: string | string[]
    link_flair_css_class?: string | string[]
    flairTemplate?: string | string[]
+    /**
+     * Is the submission a reddit-hosted image or video?
+     * */
+    isRedditMediaDomain?: boolean
 }

 // properties calculated/derived by CM -- not provided as plain values by reddit
@@ -1686,6 +1690,17 @@ export interface OperatorJsonConfig {

    bots?: BotInstanceJsonConfig[]

+    /**
+     * Added to the User-Agent information sent to reddit
+     *
+     * This string will be added BETWEEN version and your bot name.
+     *
+     * EX: `myBranch` => `web:contextMod:v1.0.0-myBranch:BOT-/u/MyBotUser`
+     *
+     * * ENV => `USER_AGENT`
+     * */
+    userAgent?: string
+
    /**
     * Settings for the web interface
     * */
@@ -1861,6 +1876,7 @@ export interface BotInstanceConfig extends BotInstanceJsonConfig {
        softLimit: number,
        hardLimit: number,
    }
+    userAgent?: string
 }

 export interface OperatorConfig extends OperatorJsonConfig {
@@ -1931,6 +1947,7 @@ export interface LogInfo {
    instance?: string
    labels?: string[]
    bot?: string
+    user?: string
 }

 export interface ActionResult extends ActionProcessResult {
--- a/src/ConfigBuilder.ts
+++ b/src/ConfigBuilder.ts
@@ -666,9 +666,13 @@ export const parseOperatorConfigFromSources = async (args: any): Promise<[Operat
        defaultBotInstance.caching = configFromFile.caching;
    }

-    let botInstances = [];
+    let botInstances: BotInstanceJsonConfig[] = [];
    if (botInstancesFromFile.length === 0) {
-        botInstances = [defaultBotInstance];
+        // only add default bot if user supplied any credentials
+        // otherwise its most likely just default, empty settings
+        if(defaultBotInstance.credentials !== undefined) {
+            botInstances = [defaultBotInstance];
+        }
    } else {
        botInstances = botInstancesFromFile.map(x => merge.all([defaultBotInstance, x], {arrayMerge: overwriteMerge}));
    }
@@ -694,6 +698,7 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
            stream = {},
        } = {},
        caching: opCache,
+        userAgent,
        web: {
            port = 8085,
            maxLogs = 200,
@@ -772,6 +777,10 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
        ...fileRest
    } = file;

+     const defaultWebCredentials = {
+         redirectUri: 'http://localhost:8085/callback'
+     };
+

    const config: OperatorConfig = {
        mode,
@@ -796,6 +805,7 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
            }
        },
        caching: cache,
+        userAgent,
        web: {
            port,
            caching: {
@@ -811,7 +821,7 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
            },
            maxLogs,
            clients: clients === undefined ? [{host: 'localhost:8095', secret: apiSecret}] : clients,
-            credentials: webCredentials as RequiredWebRedditCredentials,
+            credentials: {...defaultWebCredentials, ...webCredentials} as RequiredWebRedditCredentials,
            operators: operators || defaultOperators,
        },
        api: {
@@ -835,7 +845,8 @@ export const buildBotConfig = (data: BotInstanceJsonConfig, opConfig: OperatorCo
            actionedEventsMax: opActionedEventsMax,
            actionedEventsDefault: opActionedEventsDefault = 25,
            provider: defaultProvider,
-        } = {}
+        } = {},
+        userAgent,
    } = opConfig;
    const {
        name: botName,
@@ -976,6 +987,7 @@ export const buildBotConfig = (data: BotInstanceJsonConfig, opConfig: OperatorCo
        },
        credentials: botCreds,
        caching: botCache,
+        userAgent,
        polling: {
            shared: [...new Set(realShared)] as PollOn[],
            stagger,
--- a/src/Rule/RegexRule.ts
+++ b/src/Rule/RegexRule.ts
@@ -95,6 +95,15 @@ export interface RegexCriteria {
     * */
    totalMatchThreshold?: string,

+    /**
+     * When `true` the Activity being checked MUST pass the `matchThreshold` before the Rule considers any history
+     *
+     * For use with `activityMatchThreshold`/`totalMatchThreshold` -- useful to conserve API calls
+     *
+     * @default false
+     * */
+    mustMatchCurrent?: boolean
+
    window?: ActivityWindowType
 }

@@ -140,6 +149,7 @@ export class RegexRule extends Rule {
                matchThreshold = '> 0',
                activityMatchThreshold = '> 0',
                totalMatchThreshold = null,
+                mustMatchCurrent = false,
                window,
            } = criteria;

@@ -184,6 +194,8 @@ export class RegexRule extends Rule {
            if (singleMatched) {
                activitiesMatchedCount++;
            }
+            const singleCriteriaPass = !mustMatchCurrent || (mustMatchCurrent && singleMatched);
+
            if (activityMatchComparison !== undefined) {
                activityThresholdMet = !activityMatchComparison.isPercent && comparisonTextOp(activitiesMatchedCount, activityMatchComparison.operator, activityMatchComparison.value);
            }
@@ -192,7 +204,7 @@ export class RegexRule extends Rule {
            }

            let history: (Submission | Comment)[] = [];
-            if ((activityThresholdMet === false || totalThresholdMet === false) && window !== undefined) {
+            if ((activityThresholdMet === false || totalThresholdMet === false) && window !== undefined && singleCriteriaPass) {
                // our checking activity didn't meet threshold requirements and criteria does define window
                // leh go

@@ -263,7 +275,8 @@ export class RegexRule extends Rule {
                    matchThreshold,
                    activityMatchThreshold,
                    totalMatchThreshold,
-                    window: humanWindow
+                    window: humanWindow,
+                    mustMatchCurrent,
                },
                matches,
                matchCount,
--- a/src/Schema/Action.json
+++ b/src/Schema/Action.json
@@ -290,6 +290,10 @@
                        }
                    ]
                },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                "is_self": {
                    "type": "boolean"
                },
--- a/src/Schema/App.json
+++ b/src/Schema/App.json
@@ -2413,6 +2413,11 @@
                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s+.*)*$",
                    "type": "string"
                },
+                "mustMatchCurrent": {
+                    "default": false,
+                    "description": "When `true` the Activity being checked MUST pass the `matchThreshold` before the Rule considers any history\n\nFor use with `activityMatchThreshold`/`totalMatchThreshold` -- useful to conserve API calls",
+                    "type": "boolean"
+                },
                "name": {
                    "description": "A descriptive name that will be used in logging and be available for templating",
                    "examples": [
@@ -3508,6 +3513,10 @@
                        }
                    ]
                },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                "is_self": {
                    "type": "boolean"
                },
--- a/src/Schema/OperatorConfig.json
+++ b/src/Schema/OperatorConfig.json
@@ -1131,6 +1131,10 @@
                        }
                    ]
                },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                "is_self": {
                    "type": "boolean"
                },
@@ -1380,6 +1384,10 @@
            "$ref": "#/definitions/SnoowrapOptions",
            "description": "Set global snoowrap options as well as default snoowrap config for all bots that don't specify their own"
        },
+        "userAgent": {
+            "description": "Added to the User-Agent information sent to reddit\n\nThis string will be added BETWEEN version and your bot name.\n\nEX: `myBranch` => `web:contextMod:v1.0.0-myBranch:BOT-/u/MyBotUser`\n\n* ENV => `USER_AGENT`",
+            "type": "string"
+        },
        "web": {
            "description": "Settings for the web interface",
            "properties": {
--- a/src/Schema/Rule.json
+++ b/src/Schema/Rule.json
@@ -1286,6 +1286,11 @@
                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s+.*)*$",
                    "type": "string"
                },
+                "mustMatchCurrent": {
+                    "default": false,
+                    "description": "When `true` the Activity being checked MUST pass the `matchThreshold` before the Rule considers any history\n\nFor use with `activityMatchThreshold`/`totalMatchThreshold` -- useful to conserve API calls",
+                    "type": "boolean"
+                },
                "name": {
                    "description": "A descriptive name that will be used in logging and be available for templating",
                    "examples": [
@@ -1978,6 +1983,10 @@
                        }
                    ]
                },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                "is_self": {
                    "type": "boolean"
                },
--- a/src/Schema/RuleSet.json
+++ b/src/Schema/RuleSet.json
@@ -1260,6 +1260,11 @@
                    "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s+.*)*$",
                    "type": "string"
                },
+                "mustMatchCurrent": {
+                    "default": false,
+                    "description": "When `true` the Activity being checked MUST pass the `matchThreshold` before the Rule considers any history\n\nFor use with `activityMatchThreshold`/`totalMatchThreshold` -- useful to conserve API calls",
+                    "type": "boolean"
+                },
                "name": {
                    "description": "A descriptive name that will be used in logging and be available for templating",
                    "examples": [
@@ -1952,6 +1957,10 @@
                        }
                    ]
                },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                "is_self": {
                    "type": "boolean"
                },
--- a/src/Subreddit/Manager.ts
+++ b/src/Subreddit/Manager.ts
@@ -24,7 +24,7 @@ import {
    ActionedEvent,
    ActionResult,
    DEFAULT_POLLING_INTERVAL,
-    DEFAULT_POLLING_LIMIT, FilterCriteriaDefaults, Invokee,
+    DEFAULT_POLLING_LIMIT, FilterCriteriaDefaults, Invokee, LogInfo,
    ManagerOptions, ManagerStateChangeOption, ManagerStats, PAUSED,
    PollingOptionsStrong, PollOn, RUNNING, RunState, STOPPED, SYSTEM, USER
 } from "../Common/interfaces";
@@ -87,6 +87,7 @@ export class Manager extends EventEmitter {
    subreddit: Subreddit;
    client: ExtendedSnoowrap;
    logger: Logger;
+    logs: LogInfo[] = [];
    botName: string;
    pollOptions: PollingOptionsStrong[] = [];
    submissionChecks!: SubmissionCheck[];
@@ -211,6 +212,11 @@ export class Manager extends EventEmitter {
                return getDisplay()
            }
        }, mergeArr);
+        this.logger.stream().on('log', (log: LogInfo) => {
+            if(log.subreddit !== undefined && log.subreddit === this.getDisplay()) {
+                this.logs = [log, ...this.logs].slice(0, 301);
+            }
+        });
        this.globalDryRun = dryRun;
        this.wikiLocation = wikiLocation;
        this.filterCriteriaDefaults = filterCriteriaDefaults;
@@ -270,15 +276,21 @@ export class Manager extends EventEmitter {
        })(this), 10000);
    }

-    protected async getModPermissions(): Promise<string[]> {
+    public async getModPermissions(): Promise<string[]> {
        if(this.modPermissions !== undefined) {
            return this.modPermissions as string[];
        }
        this.logger.debug('Retrieving mod permissions for bot');
-        const userInfo = parseRedditEntity(this.botName, 'user');
-        const mods = this.subreddit.getModerators({name: userInfo.name});
-        // @ts-ignore
-        this.modPermissions = mods[0].mod_permissions;
+        try {
+            const userInfo = parseRedditEntity(this.botName, 'user');
+            const mods = this.subreddit.getModerators({name: userInfo.name});
+            // @ts-ignore
+            this.modPermissions = mods[0].mod_permissions;
+        } catch (e) {
+            const err = new ErrorWithCause('Unable to retrieve moderator permissions', {cause: e});
+            this.logger.error(err);
+            return [];
+        }
        return this.modPermissions as string[];
    }

@@ -735,7 +747,7 @@ export class Manager extends EventEmitter {
                    actionsRun = runActions.length;

                    if(check.notifyOnTrigger) {
-                        const ar = runActions.map(x => x.name).join(', ');
+                        const ar = runActions.filter(x => x.success).map(x => x.name).join(', ');
                        this.notificationManager.handle('eventActioned', 'Check Triggered', `Check "${check.name}" was triggered on Event: \n\n ${ePeek} \n\n with the following actions run: ${ar}`);
                    }
                    break;
--- a/src/Subreddit/SubredditResources.ts
+++ b/src/Subreddit/SubredditResources.ts
@@ -12,6 +12,7 @@ import winston, {Logger} from "winston";
 import as from 'async';
 import fetch from 'node-fetch';
 import {
+    asActivity,
    asSubmission,
    buildCacheOptionsFromProvider,
    buildCachePrefix,
@@ -19,18 +20,18 @@ import {
    compareDurationValue,
    comparisonTextOp,
    createCacheManager,
-    createHistoricalStatsDisplay, FAIL,
+    createHistoricalStatsDisplay, escapeRegex, FAIL,
    fetchExternalUrl, filterCriteriaSummary,
    formatNumber,
    getActivityAuthorName,
    getActivitySubredditName,
-    isStrongSubredditState, isSubmission,
+    isStrongSubredditState, isSubmission, isUser,
    mergeArr,
    parseDurationComparison,
    parseExternalUrl,
    parseGenericValueComparison,
-    parseRedditEntity,
-    parseWikiContext, PASS,
+    parseRedditEntity, parseStringToRegex,
+    parseWikiContext, PASS, redisScanIterator,
    shouldCacheSubredditStateCriteriaResult,
    subredditStateIsNameOnly,
    toStrongSubredditState
@@ -69,6 +70,7 @@ import {check} from "tcp-port-used";
 import {ExtendedSnoowrap} from "../Utils/SnoowrapClients";
 import dayjs from "dayjs";
 import ImageData from "../Common/ImageData";
+import globrex from 'globrex';

 export const DEFAULT_FOOTER = '\r\n*****\r\nThis action was performed by [a bot.]({{botLink}}) Mention a moderator or [send a modmail]({{modmailLink}}) if you any ideas, questions, or concerns about this action.';

@@ -205,24 +207,23 @@ export class SubredditResources {
         const at = await this.cache.wrap(`${this.name}-historical-allTime`, () => createHistoricalDefaults(), {ttl: 0}) as object;
         const rehydratedAt: any = {};
         for(const [k, v] of Object.entries(at)) {
-            if(Array.isArray(v)) {
+             const t = typeof v;
+             if(t === 'number') {
+                 // simple number stat like eventsCheckedTotal
+                 rehydratedAt[k] = v;
+             } else if(Array.isArray(v)) {
+                 // a map stat that we have data for is serialized as an array of KV pairs
                rehydratedAt[k] = new Map(v);
-            } else {
-                rehydratedAt[k] = v;
-            }
+             } else if(v === null || v === undefined || (t === 'object' && Object.keys(v).length === 0)) {
+                 // a map stat that was not serialized (for some reason) or serialized without any data
+                 rehydratedAt[k] = new Map();
+             } else {
+                 // ???? shouldn't get here
+                 this.logger.warn(`Did not recognize rehydrated historical stat "${k}" of type ${t}`);
+                 rehydratedAt[k] = v;
+             }
         }
         this.stats.historical.allTime = rehydratedAt as HistoricalStats;
-
-        // const lr = await this.cache.wrap(`${this.name}-historical-lastReload`, () => createHistoricalDefaults(), {ttl: 0}) as object;
-        // const rehydratedLr: any = {};
-        // for(const [k, v] of Object.entries(lr)) {
-        //     if(Array.isArray(v)) {
-        //         rehydratedLr[k] = new Map(v);
-        //     } else {
-        //         rehydratedLr[k] = v;
-        //     }
-        // }
-        // this.stats.historical.lastReload = rehydratedLr;
    }

    updateHistoricalStats(data: HistoricalStatUpdateData) {
@@ -298,6 +299,88 @@ export class SubredditResources {
        return 0;
    }

+    async interactWithCacheByKeyPattern(pattern: string | RegExp, action: 'get' | 'delete') {
+        let patternIsReg = pattern instanceof RegExp;
+        let regPattern: RegExp;
+        let globPattern = pattern;
+
+        const cacheDict: Record<string, any> = {};
+
+        if (typeof pattern === 'string') {
+            const possibleRegPattern = parseStringToRegex(pattern, 'ig');
+            if (possibleRegPattern !== undefined) {
+                regPattern = possibleRegPattern;
+                patternIsReg = true;
+            } else {
+                if (this.prefix !== undefined && !pattern.includes(this.prefix)) {
+                    // need to add wildcard to beginning of pattern so that the regex will still match a key with a prefix
+                    globPattern = `${this.prefix}${pattern}`;
+                }
+                // @ts-ignore
+                const result = globrex(globPattern, {flags: 'i'});
+                regPattern = result.regex;
+            }
+        } else {
+            regPattern = pattern;
+        }
+
+        if (this.cacheType === 'redis') {
+            // @ts-ignore
+            const redisClient = this.cache.store.getClient();
+            if (patternIsReg) {
+                // scan all and test key by regex
+                for await (const key of redisClient.scanIterator()) {
+                    if (regPattern.test(key) && (this.prefix === undefined || key.includes(this.prefix))) {
+                        if (action === 'delete') {
+                            await redisClient.del(key)
+                        } else {
+                            cacheDict[key] = await redisClient.get(key);
+                        }
+                    }
+                }
+            } else {
+                // not a regex means we can use glob pattern (more efficient!)
+                for await (const key of redisScanIterator(redisClient, { MATCH: globPattern })) {
+                    if (action === 'delete') {
+                        await redisClient.del(key)
+                    } else {
+                        cacheDict[key] = await redisClient.get(key);
+                    }
+                }
+            }
+        } else if (this.cache.store.keys !== undefined) {
+            for (const key of await this.cache.store.keys()) {
+                if (regPattern.test(key) && (this.prefix === undefined || key.includes(this.prefix))) {
+                    if (action === 'delete') {
+                        await this.cache.del(key)
+                    } else {
+                        cacheDict[key] = await this.cache.get(key);
+                    }
+                }
+            }
+        }
+        return cacheDict;
+    }
+
+    async deleteCacheByKeyPattern(pattern: string | RegExp) {
+        return await this.interactWithCacheByKeyPattern(pattern, 'delete');
+    }
+
+    async getCacheByKeyPattern(pattern: string | RegExp) {
+        return await this.interactWithCacheByKeyPattern(pattern, 'get');
+    }
+
+    async resetCacheForItem(item: Comment | Submission | RedditUser) {
+        if (asActivity(item)) {
+            if (this.filterCriteriaTTL !== false) {
+                await this.deleteCacheByKeyPattern(`itemCrit-${item.name}*`);
+            }
+            await this.setActivity(item, false);
+        } else if (isUser(item) && this.filterCriteriaTTL !== false) {
+            await this.deleteCacheByKeyPattern(`authorCrit-*-${getActivityAuthorName(item)}*`);
+        }
+    }
+
    async getStats() {
        const totals = Object.values(this.stats.cache).reduce((acc, curr) => ({
            miss: acc.miss + curr.miss,
@@ -379,11 +462,8 @@ export class SubredditResources {
                    this.logger.debug(`Cache Hit: Submission ${item.name}`);
                    return cachedSubmission;
                }
-                // @ts-ignore
-                const submission = await item.fetch();
                this.stats.cache.submission.miss++;
-                await this.cache.set(hash, submission, {ttl: this.submissionTTL});
-                return submission;
+                return await this.setActivity(item);
            } else if (this.commentTTL !== false) {
                hash = `comm-${item.name}`;
                await this.stats.cache.comment.identifierRequestCount.set(hash, (await this.stats.cache.comment.identifierRequestCount.wrap(hash, () => 0) as number) + 1);
@@ -394,11 +474,8 @@ export class SubredditResources {
                    this.logger.debug(`Cache Hit: Comment ${item.name}`);
                    return cachedComment;
                }
-                // @ts-ignore
-                const comment = await item.fetch();
                this.stats.cache.comment.miss++;
-                await this.cache.set(hash, comment, {ttl: this.commentTTL});
-                return comment;
+                return this.setActivity(item);
            } else {
                // @ts-ignore
                return await item.fetch();
@@ -409,6 +486,37 @@ export class SubredditResources {
        }
    }

+    // @ts-ignore
+    public async setActivity(item: Submission | Comment, tryToFetch = true)
+    {
+        let hash = '';
+        if(this.submissionTTL !== false && isSubmission(item)) {
+            hash = `sub-${item.name}`;
+            if(tryToFetch && item instanceof Submission) {
+                // @ts-ignore
+                const itemToCache = await item.fetch();
+                await this.cache.set(hash, itemToCache, {ttl: this.submissionTTL});
+                return itemToCache;
+            } else {
+                // @ts-ignore
+                await this.cache.set(hash, item, {ttl: this.submissionTTL});
+                return item;
+            }
+        } else if(this.commentTTL !== false){
+            hash = `comm-${item.name}`;
+            if(tryToFetch && item instanceof Comment) {
+                // @ts-ignore
+                const itemToCache = await item.fetch();
+                await this.cache.set(hash, itemToCache, {ttl: this.commentTTL});
+                return itemToCache;
+            } else {
+                // @ts-ignore
+                await this.cache.set(hash, item, {ttl: this.commentTTL});
+                return item;
+            }
+        }
+    }
+
    async hasActivity(item: Submission | Comment) {
        const hash = asSubmission(item) ? `sub-${item.name}` : `comm-${item.name}`;
        const res = await this.cache.get(hash);
@@ -978,6 +1086,20 @@ export class SubredditResources {
                                    return false
                                }
                                break;
+                            case 'isRedditMediaDomain':
+                                if((item instanceof Comment)) {
+                                    log.warn('`isRedditMediaDomain` is not allowed in `itemIs` criteria when the main Activity is a Comment');
+                                    continue;
+                                }
+                                // @ts-ignore
+                                const isRedditDomain = crit[k] as boolean;
+                                // @ts-ignore
+                                if (item.is_reddit_media_domain !== isRedditDomain) {
+                                    // @ts-ignore
+                                    log.debug(`Failed: Expected => ${k}:${crit[k]} | Found => ${k}:${item.is_reddit_media_domain}`)
+                                    return false
+                                }
+                                break;
                            case 'approved':
                            case 'spam':
                                if(!item.can_mod_post) {
--- a/src/Subreddit/UserNotes.ts
+++ b/src/Subreddit/UserNotes.ts
@@ -14,6 +14,7 @@ import Submission from "snoowrap/dist/objects/Submission";
 import {RichContent} from "../Common/interfaces";
 import {Cache} from 'cache-manager';
 import {isScopeError} from "../Utils/Errors";
+import {ErrorWithCause} from "pony-cause";

 interface RawUserNotesPayload {
    ver: number,
@@ -63,6 +64,7 @@ export class UserNotes {
    identifier: string;
    cache: Cache
    cacheCB: Function;
+    mod?: RedditUser;

    users: Map<string, UserNote[]> = new Map();

@@ -110,14 +112,22 @@ export class UserNotes {
        }
    }

+    // @ts-ignore
+    async getMod() {
+        if(this.mod === undefined) {
+            // idgaf
+            // @ts-ignore
+            this.mod = await this.subreddit._r.getMe();
+        }
+        return this.mod as RedditUser;
+    }
+
    async addUserNote(item: (Submission|Comment), type: string | number, text: string = ''): Promise<UserNote>
    {
        const payload = await this.retrieveData();
        const userName = getActivityAuthorName(item.author);

-        // idgaf
-        // @ts-ignore
-        const mod = await this.subreddit._r.getMe();
+        const mod = await this.getMod();
        if(!payload.constants.users.includes(mod.name)) {
            this.logger.info(`Mod ${mod.name} does not exist in UserNote constants, adding them`);
            payload.constants.users.push(mod.name);
@@ -134,11 +144,11 @@ export class UserNotes {
        }
        payload.blob[userName].ns.push(newNote.toRaw(payload.constants));

+        const existingNotes = await this.getUserNotes(item.author);
        await this.saveData(payload);
        if(this.notesTTL > 0) {
-            const currNotes = this.users.get(userName) || [];
-            currNotes.push(newNote);
-            this.users.set(userName, currNotes);
+            existingNotes.push(newNote);
+            this.users.set(userName, existingNotes);
        }
        return newNote;
    }
@@ -150,7 +160,6 @@ export class UserNotes {
    }

    async retrieveData(): Promise<RawUserNotesPayload> {
-        let cacheMiss;
        if (this.notesTTL > 0) {
            const cachedPayload = await this.cache.get(this.identifier);
            if (cachedPayload !== undefined && cachedPayload !== null) {
@@ -158,19 +167,9 @@ export class UserNotes {
                return cachedPayload as unknown as RawUserNotesPayload;
            }
            this.cacheCB(true);
-            cacheMiss = true;
        }

        try {
-            // DISABLED for now because I think its causing issues
-            // if(cacheMiss && this.debounceCB !== undefined) {
-            //     // timeout is still delayed. its our wiki data and we want it now! cm cacheworth 877 cache now
-            //     this.logger.debug(`Detected missed cache on usernotes retrieval while batch (${this.batchCount}) save is in progress, executing save immediately before retrieving new notes...`);
-            //     clearTimeout(this.saveDebounce);
-            //     await this.debounceCB();
-            //     this.debounceCB = undefined;
-            //     this.saveDebounce = undefined;
-            // }
            // @ts-ignore
            const wiki = this.client.getSubreddit(this.subreddit.display_name).getWikiPage('usernotes');
            const wikiContent = await wiki.content_md;
@@ -199,33 +198,6 @@ export class UserNotes {
        try {
            const wiki = this.client.getSubreddit(this.subreddit.display_name).getWikiPage('usernotes');
            if (this.notesTTL !== false) {
-                // DISABLED for now because if it fails throws an uncaught rejection
-                // and need to figured out how to handle this other than just logging (want to interrupt action flow too?)
-                //
-                // debounce usernote save by 5 seconds -- effectively batch usernote saves
-                //
-                // so that if we are processing a ton of checks that write user notes we aren't calling to save the wiki page on every call
-                // since we also have everything in cache (most likely...)
-                //
-                // TODO might want to increase timeout to 10 seconds
-                // if(this.saveDebounce !== undefined) {
-                //     clearTimeout(this.saveDebounce);
-                // }
-                // this.debounceCB = (async function () {
-                //     const p = wikiPayload;
-                //     // @ts-ignore
-                //     const self = this as UserNotes;
-                //     // @ts-ignore
-                //     self.wiki = await self.subreddit.getWikiPage('usernotes').edit(p);
-                //     self.logger.debug(`Batch saved ${self.batchCount} usernotes`);
-                //     self.debounceCB = undefined;
-                //     self.saveDebounce = undefined;
-                //     self.batchCount = 0;
-                // }).bind(this);
-                // this.saveDebounce = setTimeout(this.debounceCB,5000);
-                // this.batchCount++;
-                // this.logger.debug(`Saving Usernotes has been debounced for 5 seconds (${this.batchCount} batched)`)
-
                // @ts-ignore
                await wiki.edit(wikiPayload);
                await this.cache.set(this.identifier, payload, {ttl: this.notesTTL});
@@ -237,15 +209,14 @@ export class UserNotes {

            return payload as RawUserNotesPayload;
        } catch (err: any) {
-            let msg = 'Could not edit usernotes.';
+            let msg = 'Could not edit usernotes!';
            // Make sure at least one moderator has used toolbox and usernotes before and that this account has editing permissions`;
            if(isScopeError(err)) {
                msg = `${msg} The bot account did not have sufficient OAUTH scope to perform this action. You must re-authenticate the bot and ensure it has has 'wikiedit' permissions.`
            } else {
                msg = `${msg} Make sure at least one moderator has used toolbox, created a usernote, and that this account has editing permissions for the wiki page.`;
            }
-            this.logger.error(msg, err);
-            throw new LoggedError(msg);
+            throw new ErrorWithCause(msg, {cause: err});
        }
    }
 }
--- a/src/Utils/SnoowrapUtils.ts
+++ b/src/Utils/SnoowrapUtils.ts
@@ -503,7 +503,7 @@ export const testAuthorCriteria = async (item: (Comment | Submission), authorOpt
                        break;
                    case 'isMod':
                        const mods: RedditUser[] = await item.subreddit.getModerators();
-                        const isModerator = mods.some(x => x.name === authorName);
+                        const isModerator = mods.some(x => x.name === authorName) || authorName.toLowerCase() === 'automoderator';
                        const modMatch = authorOpts.isMod === isModerator;
                        propResultsMap.isMod!.found = isModerator;
                        propResultsMap.isMod!.passed = !((include && !modMatch) || (!include && modMatch));
--- a/src/Web/Client/CMInstance.ts
+++ b/src/Web/Client/CMInstance.ts
@@ -0,0 +1,148 @@
+import {URL} from "url";
+import {Logger} from "winston";
+import {BotInstance, CMInstanceInterface, CMInstanceInterface as CMInterface} from "../interfaces";
+import dayjs from 'dayjs';
+import {BotConnection, LogInfo} from "../../Common/interfaces";
+import normalizeUrl from "normalize-url";
+import {HeartbeatResponse} from "../Common/interfaces";
+import jwt from "jsonwebtoken";
+import got from "got";
+import {ErrorWithCause} from "pony-cause";
+
+export class CMInstance implements CMInterface {
+    friendly?: string;
+    operators: string[] = [];
+    operatorDisplay: string = '';
+    url: URL;
+    normalUrl: string;
+    lastCheck?: number;
+    online: boolean = false;
+    subreddits: string[] = [];
+    bots: BotInstance[] = [];
+    error?: string | undefined;
+    host: string;
+    secret: string;
+
+    logger: Logger;
+    logs: LogInfo[] = [];
+
+    constructor(options: BotConnection, logger: Logger) {
+        const {
+            host,
+            secret
+        } = options;
+
+        this.host = host;
+        this.secret = secret;
+
+        const normalized = normalizeUrl(options.host);
+        this.normalUrl = normalized;
+        this.url = new URL(normalized);
+
+        const name = this.getName;
+
+        this.logger = logger.child({
+            get instance() {
+                return name();
+            }
+        });
+
+        this.logger.stream().on('log', (log: LogInfo) => {
+            if(log.instance !== undefined && log.instance === this.getName()) {
+                this.logs = [log, ...this.logs].slice(0, 301);
+            }
+        });
+    }
+
+    getData(): CMInterface {
+        return {
+            friendly: this.getName(),
+            operators: this.operators,
+            operatorDisplay: this.operatorDisplay,
+            url: this.url,
+            normalUrl: this.normalUrl,
+            lastCheck: this.lastCheck,
+            online: this.online,
+            subreddits: this.subreddits,
+            bots: this.bots,
+            error: this.error,
+            host: this.host,
+            secret: this.secret
+        }
+    }
+
+    getName = () => {
+        if (this.friendly !== undefined) {
+            return this.friendly
+        }
+        return this.url.host;
+    }
+
+    matchesHost = (val: string) => {
+        return normalizeUrl(val) == this.normalUrl;
+    }
+
+    updateFromHeartbeat = (resp: HeartbeatResponse, otherFriendlies: string[] = []) => {
+        this.operators = resp.operators ?? [];
+        this.operatorDisplay = resp.operatorDisplay ?? '';
+
+        const fr = resp.friendly;
+        if (fr !== undefined) {
+            if (otherFriendlies.includes(fr)) {
+                this.logger.warn(`Client returned a friendly name that is not unique (${fr}), will fallback to host as friendly (${this.url.host})`);
+            } else {
+                this.friendly = fr;
+            }
+        }
+
+        this.subreddits = resp.subreddits;
+        this.bots = resp.bots.map(x => ({...x, instance: this}));
+    }
+
+    checkHeartbeat = async (force = false, otherFriendlies: string[] = []) => {
+        let shouldCheck = force;
+        if (!shouldCheck) {
+            if (this.lastCheck === undefined) {
+                shouldCheck = true;
+            } else {
+                const lastCheck = dayjs().diff(dayjs.unix(this.lastCheck), 's');
+                if (!this.online) {
+                    if (lastCheck > 15) {
+                        shouldCheck = true;
+                    }
+                } else if (lastCheck > 60) {
+                    shouldCheck = true;
+                }
+            }
+        }
+        if (shouldCheck) {
+            this.logger.debug('Starting Heartbeat check');
+            this.lastCheck = dayjs().unix();
+            const machineToken = jwt.sign({
+                data: {
+                    machine: true,
+                },
+            }, this.secret, {
+                expiresIn: '1m'
+            });
+
+            try {
+                const resp = await got.get(`${this.normalUrl}/heartbeat`, {
+                    headers: {
+                        'Authorization': `Bearer ${machineToken}`,
+                    }
+                }).json() as CMInstanceInterface;
+
+                this.online = true;
+                this.updateFromHeartbeat(resp as HeartbeatResponse, otherFriendlies);
+                this.logger.verbose(`Heartbeat detected`);
+            } catch (err: any) {
+                this.online = false;
+                this.error = err.message;
+                const badHeartbeat = new ErrorWithCause('Heartbeat response was not ok', {cause: err});
+                this.logger.error(badHeartbeat);
+            }
+        }
+    }
+
+}
--- a/src/Web/Client/index.ts
+++ b/src/Web/Client/index.ts
@@ -9,12 +9,12 @@ import {Strategy as CustomStrategy} from 'passport-custom';
 import {OperatorConfig, BotConnection, LogInfo} from "../../Common/interfaces";
 import {
    buildCachePrefix,
-    createCacheManager, defaultFormat, filterLogBySubreddit,
-    formatLogLineToHtml,
+    createCacheManager, defaultFormat, filterLogBySubreddit, filterLogs,
+    formatLogLineToHtml, getUserAgent,
    intersect, isLogLineMinLevel,
    LogEntry, parseInstanceLogInfoName, parseInstanceLogName, parseRedditEntity,
    parseSubredditLogName, permissions,
-    randomId, sleep, triggeredIndicator
+    randomId, replaceApplicationIdentifier, sleep, triggeredIndicator
 } from "../../util";
 import {Cache} from "cache-manager";
 import session, {Session, SessionData} from "express-session";
@@ -39,7 +39,7 @@ import DelimiterStream from 'delimiter-stream';
 import {pipeline} from 'stream/promises';
 import {defaultBotStatus} from "../Common/defaults";
 import {arrayMiddle, booleanMiddle} from "../Common/middleware";
-import {BotInstance, CMInstance} from "../interfaces";
+import {BotInstance, CMInstanceInterface} from "../interfaces";
 import { URL } from "url";
 import {MESSAGE} from "triple-beam";
 import Autolinker from "autolinker";
@@ -50,6 +50,7 @@ import {BotStatusResponse} from "../Common/interfaces";
 import {TransformableInfo} from "logform";
 import {SimpleError} from "../../Utils/Errors";
 import {ErrorWithCause} from "pony-cause";
+import {CMInstance} from "./CMInstance";

 const emitter = new EventEmitter();

@@ -106,7 +107,7 @@ interface ConnectUserObj {
    [key: string]: ConnectedUserInfo
 }

-const createToken = (bot: CMInstance, user?: Express.User | any, ) => {
+const createToken = (bot: CMInstanceInterface, user?: Express.User | any, ) => {
    const payload = user !== undefined ? {...user, machine: false} : {machine: true};
    return jwt.sign({
        data: payload,
@@ -117,14 +118,13 @@ const createToken = (bot: CMInstance, user?: Express.User | any, ) => {

 const availableLevels = ['error', 'warn', 'info', 'verbose', 'debug'];

-const instanceLogMap: Map<string, LogEntry[]> = new Map();
-
 const webClient = async (options: OperatorConfig) => {
    const {
        operator: {
            name,
            display,
        },
+        userAgent: uaFragment,
        web: {
            port,
            caching,
@@ -149,22 +149,18 @@ const webClient = async (options: OperatorConfig) => {
        },
    } = options;

+    const userAgent = getUserAgent(`web:contextBot:{VERSION}{FRAG}:dashboard`, uaFragment);
+
+    app.use((req, res, next) => {
+        res.locals.applicationIdentifier = replaceApplicationIdentifier('{VERSION}{FRAG}', uaFragment);
+        next();
+    });
+
    const webOps = operators.map(x => x.toLowerCase());

    const logger = getLogger({defaultLabel: 'Web', ...options.logging}, 'Web');

    logger.stream().on('log', (log: LogInfo) => {
-        const logEntry: LogEntry = [dayjs(log.timestamp).unix(), log];
-        const {instance: instanceLogName} = log;
-        if (instanceLogName !== undefined) {
-            const subLogs = instanceLogMap.get(instanceLogName) || [];
-            subLogs.unshift(logEntry);
-            instanceLogMap.set(instanceLogName, subLogs.slice(0, 200 + 1));
-        } else {
-            const appLogs = instanceLogMap.get('web') || [];
-            appLogs.unshift(logEntry);
-            instanceLogMap.set('web', appLogs.slice(0, 200 + 1));
-        }
        emitter.emit('log', log[MESSAGE]);
    });

@@ -330,7 +326,7 @@ const webClient = async (options: OperatorConfig) => {
                userName,
            };
            if(invite.instance !== undefined) {
-                const bot = cmInstances.find(x => x.friendly === invite.instance);
+                const bot = cmInstances.find(x => x.getName() === invite.instance);
                if(bot !== undefined) {
                    const botPayload: any = {
                        overwrite: invite.overwrite === true,
@@ -430,7 +426,7 @@ const webClient = async (options: OperatorConfig) => {
            clientId,
            clientSecret,
            token: req.isAuthenticated() && req.user?.clientData?.webOperator ? token : undefined,
-            instances: cmInstances.filter(x => req.user?.isInstanceOperator(x)).map(x => x.friendly),
+            instances: cmInstances.filter(x => req.user?.isInstanceOperator(x)).map(x => x.getName()),
        });
    });

@@ -544,7 +540,7 @@ const webClient = async (options: OperatorConfig) => {
                delimiter: '\r\n',
            });

-            const currInstance = cmInstances.find(x => x.friendly === sessionData.botId);
+            const currInstance = cmInstances.find(x => x.getName() === sessionData.botId);
            if(currInstance !== undefined) {
                const ac = new AbortController();
                const options = {
@@ -557,7 +553,8 @@ const webClient = async (options: OperatorConfig) => {
                    });

                    if(err !== undefined) {
-                        logger.warn(`Log streaming encountered an error, trying to reconnect (retries: ${retryCount}) -- ${err.code !== undefined ? `(${err.code}) ` : ''}${err.message}`, {instance: currInstance.friendly});
+                        // @ts-ignore
+                        currInstance.logger.warn(new ErrorWithCause(`Log streaming encountered an error, trying to reconnect (retries: ${retryCount})`, {cause: err}), {user: user.name});
                    }
                    const gotStream = got.stream.get(`${currInstance.normalUrl}/logs`, {
                        retry: {
@@ -578,7 +575,7 @@ const webClient = async (options: OperatorConfig) => {

                    if(err !== undefined) {
                        gotStream.once('data', () => {
-                            logger.info('Streaming resumed', {subreddit: currInstance.friendly});
+                            currInstance.logger.info('Streaming resumed', {instance: currInstance.getName(), user: user.name});
                        });
                    }

@@ -592,7 +589,8 @@ const webClient = async (options: OperatorConfig) => {
                    // ECONNRESET
                    s.catch((err) => {
                        if(err.code !== 'ABORT_ERR' && err.code !== 'ERR_STREAM_PREMATURE_CLOSE') {
-                            logger.error(`Unexpected error, or too many retries, occurred while streaming logs -- ${err.code !== undefined ? `(${err.code}) ` : ''}${err.message}`, {instance: currInstance.friendly});
+                            // @ts-ignore
+                            currInstance.logger.error(new ErrorWithCause('Unexpected error, or too many retries, occurred while streaming logs', {cause: err}), {user: user.name});
                        }
                    });

@@ -640,7 +638,7 @@ const webClient = async (options: OperatorConfig) => {
        delete req.session.authBotId;

        const msg = 'Bot does not exist or you do not have permission to access it';
-        const instance = cmInstances.find(x => x.friendly === req.query.instance);
+        const instance = cmInstances.find(x => x.getName() === req.query.instance);
        if (instance === undefined) {
            return res.status(404).render('error', {error: msg});
        }
@@ -653,15 +651,15 @@ const webClient = async (options: OperatorConfig) => {
            return res.status(404).render('error', {error: msg});
        }
        req.instance = instance;
-        req.session.botId = instance.friendly;
+        req.session.botId = instance.getName();
        if(req.user?.canAccessInstance(instance)) {
-            req.session.authBotId = instance.friendly;
+            req.session.authBotId = instance.getName();
        }
        return next();
    }


-    const botWithPermissions = async (req: express.Request, res: express.Response, next: Function) => {
+    const botWithPermissions = (required: boolean = false, setDefault: boolean = false) => async (req: express.Request, res: express.Response, next: Function) => {

        const instance = req.instance;
        if(instance === undefined) {
@@ -670,28 +668,42 @@ const webClient = async (options: OperatorConfig) => {

        const msg = 'Bot does not exist or you do not have permission to access it';
        const botVal = req.query.bot as string;
-        if(botVal === undefined) {
+        if(botVal === undefined && required) {
            return res.status(400).render('error', {error: `"bot" param must be defined`});
        }

-        const botInstance = instance.bots.find(x => x.botName === botVal);
-        if(botInstance === undefined) {
-            return res.status(404).render('error', {error: msg});
+        if(botVal !== undefined || setDefault) {
+
+            let botInstance;
+            if(botVal === undefined) {
+                // find a bot they can access
+                botInstance = instance.bots.find(x => req.user?.canAccessBot(x));
+                if(botInstance !== undefined) {
+                    req.query.bot = botInstance.botName;
+                }
+            } else {
+                botInstance = instance.bots.find(x => x.botName === botVal);
+            }
+
+            if(botInstance === undefined) {
+                return res.status(404).render('error', {error: msg});
+            }
+
+            if (!req.user?.clientData?.webOperator && !req.user?.canAccessBot(botInstance)) {
+                return res.status(404).render('error', {error: msg});
+            }
+
+            if (req.params.subreddit !== undefined && !req.user?.isInstanceOperator(instance) && !req.user?.subreddits.includes(req.params.subreddit)) {
+                return res.status(404).render('error', {error: msg});
+            }
+            req.bot = botInstance;
        }

-        if (!req.user?.clientData?.webOperator && !req.user?.canAccessBot(botInstance)) {
-            return res.status(404).render('error', {error: msg});
-        }
-
-        if (req.params.subreddit !== undefined && !req.user?.isInstanceOperator(instance) && !req.user?.subreddits.includes(req.params.subreddit)) {
-            return res.status(404).render('error', {error: msg});
-        }
-        req.bot = botInstance;
        next();
    }

    const createUserToken = async (req: express.Request, res: express.Response, next: Function) => {
-        req.token = createToken(req.instance as CMInstance, req.user);
+        req.token = createToken(req.instance as CMInstanceInterface, req.user);
        next();
    }

@@ -715,10 +727,10 @@ const webClient = async (options: OperatorConfig) => {
    // botUserRouter.use([ensureAuthenticated, defaultSession, botWithPermissions, createUserToken]);
    // app.use(botUserRouter);

-    app.useAsync('/api/', [ensureAuthenticated, defaultSession, instanceWithPermissions, botWithPermissions, createUserToken], (req: express.Request, res: express.Response) => {
+    app.useAsync('/api/', [ensureAuthenticated, defaultSession, instanceWithPermissions, botWithPermissions(true), createUserToken], (req: express.Request, res: express.Response) => {
        req.headers.Authorization = `Bearer ${req.token}`

-        const instance = req.instance as CMInstance;
+        const instance = req.instance as CMInstanceInterface;
        return proxy.web(req, res, {
            target: {
                protocol: instance.url.protocol,
@@ -744,16 +756,25 @@ const webClient = async (options: OperatorConfig) => {
            });

            if(accessibleInstance === undefined) {
+                logger.warn(`User ${user.name} is not an operator and has no subreddits in common with any *running* bot instances. If you are sure they should have common subreddits then this client may not be able to access all defined CM servers or the bot may be offline.`, {user: user.name});
                return res.render('noAccess');
            }

            return res.redirect(`/?instance=${accessibleInstance.friendly}`);
        }
-        const instance = cmInstances.find(x => x.friendly === req.query.instance);
+        const instance = cmInstances.find(x => x.getName() === req.query.instance);
        req.instance = instance;
        next();
    }

+    const defaultSubreddit = async (req: express.Request, res: express.Response, next: Function) => {
+        if(req.bot !== undefined && req.query.subreddit === undefined) {
+            const firstAccessibleSub = req.bot.subreddits.find(x => req.user?.isInstanceOperator(req.instance) || req.user?.subreddits.includes(x));
+            req.query.subreddit = firstAccessibleSub;
+        }
+        next();
+    }
+
    const initHeartbeat = async (req: express.Request, res: express.Response, next: Function) => {
        if(!init) {
            for(const c of clients) {
@@ -773,7 +794,7 @@ const webClient = async (options: OperatorConfig) => {
        next();
    }

-    app.getAsync('/', [initHeartbeat, redirectBotsNotAuthed, ensureAuthenticated, defaultSession, defaultInstance, instanceWithPermissions, createUserToken], async (req: express.Request, res: express.Response) => {
+    app.getAsync('/', [initHeartbeat, redirectBotsNotAuthed, ensureAuthenticated, defaultSession, defaultInstance, instanceWithPermissions, botWithPermissions(false, true), createUserToken], async (req: express.Request, res: express.Response) => {

        const user = req.user as Express.User;
        const instance = req.instance as CMInstance;
@@ -786,13 +807,13 @@ const webClient = async (options: OperatorConfig) => {
            const isBotOperator = req.user?.isInstanceOperator(curr);
            if(user?.clientData?.webOperator) {
                // @ts-ignore
-                return acc.concat({...curr, canAccessLocation: true, isOperator: isBotOperator});
+                return acc.concat({...curr.getData(), canAccessLocation: true, isOperator: isBotOperator});
            }
            if(!isBotOperator && !req.user?.canAccessInstance(curr)) {
                return acc;
            }
            // @ts-ignore
-            return acc.concat({...curr, canAccessLocation: isBotOperator, isOperator: isBotOperator, botId: curr.friendly});
+            return acc.concat({...curr.getData(), canAccessLocation: isBotOperator, isOperator: isBotOperator, botId: curr.getName()});
        },[]);

        let resp;
@@ -802,6 +823,8 @@ const webClient = async (options: OperatorConfig) => {
                    'Authorization': `Bearer ${req.token}`,
                },
                searchParams: {
+                    bot: req.query.bot as (string | undefined),
+                    subreddit: req.query.sub as (string | undefined) ?? 'all',
                    limit,
                    sort,
                    level,
@@ -810,14 +833,15 @@ const webClient = async (options: OperatorConfig) => {
            }).json() as any;

        } catch(err: any) {
-            logger.error(`Error occurred while retrieving bot information. Will update heartbeat -- ${err.message}`, {instance: instance.friendly});
-            refreshClient(clients.find(x => normalizeUrl(x.host) === instance.normalUrl) as BotConnection);
+            instance.logger.error(new ErrorWithCause(`Could not retrieve instance information. Will attempted to update heartbeat.`, {cause: err}));
+            refreshClient({host: instance.host, secret: instance.secret});
+            const isOp = req.user?.isInstanceOperator(instance);
            return res.render('offline', {
                instances: shownInstances,
-                instanceId: (req.instance as CMInstance).friendly,
-                isOperator: req.user?.isInstanceOperator(instance),
+                instanceId: (req.instance as CMInstance).getName(),
+                isOperator: isOp,
                // @ts-ignore
-                logs: filterLogBySubreddit(instanceLogMap, [instance.friendly], {limit, sort, level, allLogName: 'web', allLogsParser: parseInstanceLogInfoName }).get(instance.friendly),
+                logs: filterLogs((isOp ? instance.logs : instance.logs.filter(x => x.user === undefined || x.user.includes(req.user.name))), {limit, sort, level}),
                logSettings: {
                    limitSelect: [10, 20, 50, 100, 200].map(x => `<option ${limit === x ? 'selected' : ''} class="capitalize ${limit === x ? 'font-bold' : ''}" data-value="${x}">${x}</option>`).join(' | '),
                    sortSelect: ['ascending', 'descending'].map(x => `<option ${sort === x ? 'selected' : ''} class="capitalize ${sort === x ? 'font-bold' : ''}" data-value="${x}">${x}</option>`).join(' '),
@@ -870,8 +894,8 @@ const webClient = async (options: OperatorConfig) => {
                });
                return {...x, subreddits: subredditsWithSimpleLogs};
            }),
-            botId: (req.instance as CMInstance).friendly,
-            instanceId: (req.instance as CMInstance).friendly,
+            botId: (req.instance as CMInstanceInterface).friendly,
+            instanceId: (req.instance as CMInstanceInterface).friendly,
            isOperator: isOp,
            system: isOp ? {
                logs: resp.system.logs,
@@ -901,7 +925,7 @@ const webClient = async (options: OperatorConfig) => {
        });
    });

-    app.postAsync('/config', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions], async (req: express.Request, res: express.Response) => {
+    app.postAsync('/config', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions(true)], async (req: express.Request, res: express.Response) => {
        const {subreddit} = req.query as any;
        const {location, data, create = false} = req.body as any;

@@ -942,9 +966,9 @@ const webClient = async (options: OperatorConfig) => {
        return res.send();
    });

-    app.getAsync('/events', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions, createUserToken], async (req: express.Request, res: express.Response) => {
+    app.getAsync('/events', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions(true), createUserToken], async (req: express.Request, res: express.Response) => {
        const {subreddit} = req.query as any;
-        const resp = await got.get(`${(req.instance as CMInstance).normalUrl}/events`, {
+        const resp = await got.get(`${(req.instance as CMInstanceInterface).normalUrl}/events`, {
            headers: {
                'Authorization': `Bearer ${req.token}`,
            },
@@ -1055,6 +1079,9 @@ const webClient = async (options: OperatorConfig) => {
        const session = socket.handshake.session as (Session & Partial<SessionData> | undefined);
        // @ts-ignore
        const user = session !== undefined ? session?.passport?.user as Express.User : undefined;
+
+        let liveInterval: any = undefined;
+
        if (session !== undefined && user !== undefined) {
            clearSockStreams(socket.id);
            socket.join(session.id);
@@ -1069,8 +1096,44 @@ const webClient = async (options: OperatorConfig) => {
            emitter.on('log', webLogListener);
            socketListeners.set(socket.id, [...(socketListeners.get(socket.id) || []), webLogListener]);

+            socket.on('viewing', (data) => {
+                if(user !== undefined) {
+                    const {subreddit, bot: botVal} = data;
+                    const currBot = cmInstances.find(x => x.getName() === session.botId);
+                    if(currBot !== undefined) {
+
+                        if(liveInterval !== undefined) {
+                            clearInterval(liveInterval)
+                        }
+
+                        const liveEmit = async () => {
+                            try {
+                                const resp = await got.get(`${currBot.normalUrl}/liveStats`, {
+                                    headers: {
+                                        'Authorization': `Bearer ${createToken(currBot, user)}`,
+                                    },
+                                    searchParams: {
+                                        bot: botVal,
+                                        subreddit
+                                    }
+                                });
+                                const stats = JSON.parse(resp.body);
+                                io.to(session.id).emit('liveStats', stats);
+                            } catch (err: any) {
+                                currBot.logger.error(new ErrorWithCause('Could not retrieve live stats', {cause: err}));
+                            }
+                        }
+
+                        // do an initial get
+                        liveEmit();
+                        // and then every 5 seconds after that
+                        liveInterval = setInterval(async () => await liveEmit(), 5000);
+                    }
+                }
+            });
+
            if(session.botId !== undefined) {
-                const bot = cmInstances.find(x => x.friendly === session.botId);
+                const bot = cmInstances.find(x => x.getName() === session.botId);
                if(bot !== undefined) {
                    // web log listener for bot specifically
                    const botWebLogListener = (log: string) => {
@@ -1100,7 +1163,7 @@ const webClient = async (options: OperatorConfig) => {
                                }).json() as object;
                                io.to(session.id).emit('opStats', resp);
                            } catch (err: any) {
-                                logger.error(`Could not retrieve stats ${err.message}`, {instance: bot.friendly});
+                                bot.logger.error(new ErrorWithCause('Could not retrieve stats', {cause: err}));
                                clearInterval(interval);
                            }
                        }, 5000);
@@ -1113,12 +1176,12 @@ const webClient = async (options: OperatorConfig) => {
        socket.on('disconnect', (reason) => {
            clearSockStreams(socket.id);
            clearSockListeners(socket.id);
+            clearInterval(liveInterval);
        });
    });

    const loopHeartbeat = async () => {
        while(true) {
-            logger.debug('Starting heartbeat check');
            for(const c of clients) {
                await refreshClient(c);
            }
@@ -1127,7 +1190,7 @@ const webClient = async (options: OperatorConfig) => {
        }
    }

-    const addBot = async (bot: CMInstance, userPayload: any, botPayload: any) => {
+    const addBot = async (bot: CMInstanceInterface, userPayload: any, botPayload: any) => {
        try {
            const token = createToken(bot, userPayload);
            const resp = await got.post(`${bot.normalUrl}/bot`, {
@@ -1144,81 +1207,13 @@ const webClient = async (options: OperatorConfig) => {
    }

    const refreshClient = async (client: BotConnection, force = false) => {
-        const normalized = normalizeUrl(client.host);
-        const existingClientIndex = cmInstances.findIndex(x => x.normalUrl === normalized);
-        const existingClient = existingClientIndex === -1 ? undefined : cmInstances[existingClientIndex];
+        const existingClientIndex = cmInstances.findIndex(x => x.matchesHost(client.host));
+        const instance = existingClientIndex === -1 ? new CMInstance(client, logger) : cmInstances[existingClientIndex];

-        let shouldCheck = false;
-        if(!existingClient) {
-            shouldCheck = true;
-        } else if(force) {
-            shouldCheck = true;
-        } else  {
-            const lastCheck = dayjs().diff(dayjs.unix(existingClient.lastCheck), 's');
-            if(!existingClient.online) {
-                if(lastCheck > 15) {
-                    shouldCheck = true;
-                }
-            } else if(lastCheck > 60) {
-                shouldCheck = true;
-            }
-        }
-        if(shouldCheck)
-        {
-            const machineToken = jwt.sign({
-                data: {
-                    machine: true,
-                },
-            }, client.secret, {
-                expiresIn: '1m'
-            });
-            //let base = `${c.host}${c.port !== undefined ? `:${c.port}` : ''}`;
-            const normalized = normalizeUrl(client.host);
-            const url = new URL(normalized);
-            let botStat: CMInstance = {
-                ...client,
-                subreddits: [] as string[],
-                operators: [] as string[],
-                operatorDisplay: '',
-                online: false,
-                friendly: url.host,
-                lastCheck: dayjs().unix(),
-                normalUrl: normalized,
-                url,
-                bots: [],
-            };
-            try {
-                const resp = await got.get(`${normalized}/heartbeat`, {
-                    headers: {
-                        'Authorization': `Bearer ${machineToken}`,
-                    }
-                }).json() as CMInstance;
+        await instance.checkHeartbeat(force);

-                const {bots, ...restResp} = resp;
-
-                botStat = {...botStat, ...restResp, bots: bots.map(x => ({...x, instance: botStat})), online: true};
-                const sameNameIndex = cmInstances.findIndex(x => x.friendly === botStat.friendly);
-                if(sameNameIndex > -1 && sameNameIndex !== existingClientIndex) {
-                    logger.warn(`Client returned a friendly name that is not unique (${botStat.friendly}), will fallback to host as friendly (${botStat.normalUrl})`);
-                    botStat.friendly = botStat.normalUrl;
-                }
-                botStat.online = true;
-                // if(botStat.online) {
-                //     botStat.indicator = botStat.running ? 'green' : 'yellow';
-                // } else {
-                //     botStat.indicator = 'red';
-                // }
-                logger.verbose(`Heartbeat detected`, {instance: botStat.friendly});
-            } catch (err: any) {
-                botStat.error = err.message;
-                logger.error(`Heartbeat response from ${botStat.friendly} was not ok: ${err.message}`, {instance: botStat.friendly});
-            } finally {
-                if(existingClientIndex !== -1) {
-                    cmInstances.splice(existingClientIndex, 1, botStat);
-                } else {
-                    cmInstances.push(botStat);
-                }
-            }
+        if(existingClientIndex === -1) {
+            cmInstances.push(instance);
        }
    }
 }
--- a/src/Web/Common/User/ClientUser.ts
+++ b/src/Web/Common/User/ClientUser.ts
@@ -1,14 +1,14 @@
-import {BotInstance, CMInstance} from "../../interfaces";
+import {BotInstance, CMInstanceInterface} from "../../interfaces";
 import CMUser from "./CMUser";
 import {intersect, parseRedditEntity} from "../../../util";

-class ClientUser extends CMUser<CMInstance, BotInstance, string> {
+class ClientUser extends CMUser<CMInstanceInterface, BotInstance, string> {

-    isInstanceOperator(val: CMInstance): boolean {
+    isInstanceOperator(val: CMInstanceInterface): boolean {
        return val.operators.map(x=> x.toLowerCase()).includes(this.name.toLowerCase());
    }

-    canAccessInstance(val: CMInstance): boolean {
+    canAccessInstance(val: CMInstanceInterface): boolean {
        return this.isInstanceOperator(val) || intersect(this.subreddits, val.subreddits.map(x => parseRedditEntity(x).name)).length > 0;
    }

--- a/src/Web/Common/User/ServerUser.ts
+++ b/src/Web/Common/User/ServerUser.ts
@@ -1,4 +1,4 @@
-import {BotInstance, CMInstance} from "../../interfaces";
+import {BotInstance, CMInstanceInterface} from "../../interfaces";
 import CMUser from "./CMUser";
 import {intersect, parseRedditEntity} from "../../../util";
 import {App} from "../../../App";
--- a/src/Web/Common/interfaces.ts
+++ b/src/Web/Common/interfaces.ts
@@ -1,5 +1,6 @@
 import {RunningState} from "../../Subreddit/Manager";
 import {LogInfo, ManagerStats} from "../../Common/interfaces";
+import {BotInstance} from "../interfaces";

 export interface BotStats {
    startedAtHuman: string,
@@ -73,3 +74,11 @@ export interface IUser {
    token?: string
    tokenExpiresAt?: number
 }
+
+export interface HeartbeatResponse {
+    subreddits: string[]
+    operators: string[]
+    operatorDisplay?: string
+    friendly?: string
+    bots: BotInstance[]
+}
--- a/src/Web/Server/middleware.ts
+++ b/src/Web/Server/middleware.ts
@@ -46,19 +46,24 @@ export const subredditRoute = (required = true) => async (req: Request, res: Res
    if(subreddit === undefined && !required) {
        next();
    } else {
-        //const {name: userName} = req.user as Express.User;

-        const manager = bot.subManagers.find(x => x.displayLabel === subreddit);
-        if (manager === undefined) {
-            return res.status(400).send('Cannot access route for subreddit you do not manage or is not run by the bot')
+        if(subreddit.toLowerCase() === 'all') {
+            next();
+        } else {
+            //const {name: userName} = req.user as Express.User;
+
+            const manager = bot.subManagers.find(x => x.displayLabel === subreddit);
+            if (manager === undefined) {
+                return res.status(400).send('Cannot access route for subreddit you do not manage or is not run by the bot')
+            }
+
+            if (!req.user?.canAccessSubreddit(bot, subreddit)) {
+                return res.status(400).send('Cannot access route for subreddit you do not manage or is not run by the bot')
+            }
+
+            req.manager = manager;
+
+            next();
        }
-
-        if (!req.user?.canAccessSubreddit(bot, subreddit)) {
-            return res.status(400).send('Cannot access route for subreddit you do not manage or is not run by the bot')
-        }
-
-        req.manager = manager;
-
-        next();
    }
 }
--- a/src/Web/Server/routes/authenticated/user/action.ts
+++ b/src/Web/Server/routes/authenticated/user/action.ts
@@ -19,7 +19,7 @@ const action = async (req: Request, res: Response) => {

    for (const manager of subreddits) {
        const mLogger = manager.logger;
-        mLogger.info(`/u/${userName} invoked '${action}' action for ${type} on ${manager.displayLabel}`);
+        mLogger.info(`/u/${userName} invoked '${action}' action for ${type} on ${manager.displayLabel}`, {user: userName});
        try {
            switch (action) {
                case 'start':
--- a/src/Web/Server/routes/authenticated/user/index.ts
+++ b/src/Web/Server/routes/authenticated/user/index.ts
@@ -106,7 +106,7 @@ const action = async (req: Request, res: Response) => {
    }

    if (a === undefined) {
-        winston.loggers.get('app').error('Could not parse Comment or Submission ID from given URL', {subreddit: `/u/${userName}`});
+        winston.loggers.get('app').error('Could not parse Comment or Submission ID from given URL', {user: userName});
        return res.send('OK');
    } else {
        // @ts-ignore
@@ -118,15 +118,15 @@ const action = async (req: Request, res: Response) => {
        if (manager === undefined || !req.user?.canAccessSubreddit(req.serverBot, manager.subreddit.display_name)) {
            let msg = 'Activity does not belong to a subreddit you moderate or the bot runs on.';
            if (subreddit === 'All') {
-                msg = `${msg} If you want to test an Activity against a Subreddit\'s config it does not belong to then switch to that Subreddit's tab first.`
+                msg = `${msg} If you want to test an Activity against a Subreddit's config it does not belong to then switch to that Subreddit's tab first.`
            }
-            winston.loggers.get('app').error(msg, {subreddit: `/u/${userName}`});
+            winston.loggers.get('app').error(msg, {user: userName});
            return res.send('OK');
        }

        // will run dryrun if specified or if running activity on subreddit it does not belong to
        const dr: boolean | undefined = (dryRun || manager.subreddit.display_name !== sub) ? true : undefined;
-        manager.logger.info(`/u/${userName} running${dr === true ? ' DRY RUN ' : ' '}check on${manager.subreddit.display_name !== sub ? ' FOREIGN ACTIVITY ' : ' '}${url}`);
+        manager.logger.info(`/u/${userName} running${dr === true ? ' DRY RUN ' : ' '}check on${manager.subreddit.display_name !== sub ? ' FOREIGN ACTIVITY ' : ' '}${url}`, {user: userName, subreddit});
        await manager.runChecks(activity instanceof Submission ? 'Submission' : 'Comment', activity, {dryRun: dr, force: true})
    }
    res.send('OK');
--- a/src/Web/Server/routes/authenticated/user/liveStats.ts
+++ b/src/Web/Server/routes/authenticated/user/liveStats.ts
@@ -0,0 +1,261 @@
+import {authUserCheck, botRoute, subredditRoute} from "../../../middleware";
+import {Request, Response} from "express";
+import Bot from "../../../../../Bot";
+import {boolToString, cacheStats, filterLogs, formatNumber, logSortFunc, pollingInfo} from "../../../../../util";
+import dayjs from "dayjs";
+import {LogInfo, ResourceStats, RUNNING, STOPPED, SYSTEM} from "../../../../../Common/interfaces";
+import {Manager} from "../../../../../Subreddit/Manager";
+import winston from "winston";
+import {opStats} from "../../../../Common/util";
+import {BotStatusResponse} from "../../../../Common/interfaces";
+
+const liveStats = () => {
+    const middleware = [
+        authUserCheck(),
+        botRoute(),
+        subredditRoute(false),
+    ]
+
+    const response = async (req: Request, res: Response) =>
+    {
+        const bot = req.serverBot as Bot;
+        const manager = req.manager;
+        
+        if(manager === undefined) {
+            // getting all
+            const subManagerData: any[] = [];
+            for (const m of req.user?.accessibleSubreddits(bot) as Manager[]) {
+                const sd = {
+                    queuedActivities: m.queue.length(),
+                    runningActivities: m.queue.running(),
+                    maxWorkers: m.queue.concurrency,
+                    subMaxWorkers: m.subMaxWorkers || bot.maxWorkers,
+                    globalMaxWorkers: bot.maxWorkers,
+                    checks: {
+                        submissions: m.submissionChecks === undefined ? 0 : m.submissionChecks.length,
+                        comments: m.commentChecks === undefined ? 0 : m.commentChecks.length,
+                    },
+                    stats: await m.getStats(),
+                }
+            }
+
+            const totalStats = subManagerData.reduce((acc, curr) => {
+                return {
+                    checks: {
+                        submissions: acc.checks.submissions + curr.checks.submissions,
+                        comments: acc.checks.comments + curr.checks.comments,
+                    },
+                    historical: {
+                        allTime: {
+                            eventsCheckedTotal: acc.historical.allTime.eventsCheckedTotal + curr.stats.historical.allTime.eventsCheckedTotal,
+                            eventsActionedTotal: acc.historical.allTime.eventsActionedTotal + curr.stats.historical.allTime.eventsActionedTotal,
+                            checksRunTotal: acc.historical.allTime.checksRunTotal + curr.stats.historical.allTime.checksRunTotal,
+                            checksFromCacheTotal: acc.historical.allTime.checksFromCacheTotal + curr.stats.historical.allTime.checksFromCacheTotal,
+                            checksTriggeredTotal: acc.historical.allTime.checksTriggeredTotal + curr.stats.historical.allTime.checksTriggeredTotal,
+                            rulesRunTotal: acc.historical.allTime.rulesRunTotal + curr.stats.historical.allTime.rulesRunTotal,
+                            rulesCachedTotal: acc.historical.allTime.rulesCachedTotal + curr.stats.historical.allTime.rulesCachedTotal,
+                            rulesTriggeredTotal: acc.historical.allTime.rulesTriggeredTotal + curr.stats.historical.allTime.rulesTriggeredTotal,
+                            actionsRunTotal: acc.historical.allTime.actionsRunTotal + curr.stats.historical.allTime.actionsRunTotal,
+                        }
+                    },
+                    maxWorkers: acc.maxWorkers + curr.maxWorkers,
+                    subMaxWorkers: acc.subMaxWorkers + curr.subMaxWorkers,
+                    globalMaxWorkers: acc.globalMaxWorkers + curr.globalMaxWorkers,
+                    runningActivities: acc.runningActivities + curr.runningActivities,
+                    queuedActivities: acc.queuedActivities + curr.queuedActivities,
+                };
+            }, {
+                checks: {
+                    submissions: 0,
+                    comments: 0,
+                },
+                historical: {
+                    allTime: {
+                        eventsCheckedTotal: 0,
+                        eventsActionedTotal: 0,
+                        checksRunTotal: 0,
+                        checksFromCacheTotal: 0,
+                        checksTriggeredTotal: 0,
+                        rulesRunTotal: 0,
+                        rulesCachedTotal: 0,
+                        rulesTriggeredTotal: 0,
+                        actionsRunTotal: 0,
+                    }
+                },
+                maxWorkers: 0,
+                subMaxWorkers: 0,
+                globalMaxWorkers: 0,
+                runningActivities: 0,
+                queuedActivities: 0,
+            });
+            const {
+                checks,
+                maxWorkers,
+                globalMaxWorkers,
+                subMaxWorkers,
+                runningActivities,
+                queuedActivities,
+                ...rest
+            } = totalStats;
+
+            let cumRaw = subManagerData.reduce((acc, curr) => {
+                Object.keys(curr.stats.cache.types as ResourceStats).forEach((k) => {
+                    acc[k].requests += curr.stats.cache.types[k].requests;
+                    acc[k].miss += curr.stats.cache.types[k].miss;
+                    // @ts-ignore
+                    acc[k].identifierAverageHit += (typeof curr.stats.cache.types[k].identifierAverageHit === 'string' ? Number.parseFloat(curr.stats.cache.types[k].identifierAverageHit) : curr.stats.cache.types[k].identifierAverageHit);
+                    acc[k].averageTimeBetweenHits += curr.stats.cache.types[k].averageTimeBetweenHits === 'N/A' ? 0 : Number.parseFloat(curr.stats.cache.types[k].averageTimeBetweenHits)
+                });
+                return acc;
+            }, cacheStats());
+            cumRaw = Object.keys(cumRaw).reduce((acc, curr) => {
+                const per = acc[curr].miss === 0 ? 0 : formatNumber(acc[curr].miss / acc[curr].requests) * 100;
+                // @ts-ignore
+                acc[curr].missPercent = `${formatNumber(per, {toFixed: 0})}%`;
+                acc[curr].identifierAverageHit = formatNumber(acc[curr].identifierAverageHit);
+                acc[curr].averageTimeBetweenHits = formatNumber(acc[curr].averageTimeBetweenHits)
+                return acc;
+            }, cumRaw);
+            const cacheReq = subManagerData.reduce((acc, curr) => acc + curr.stats.cache.totalRequests, 0);
+            const cacheMiss = subManagerData.reduce((acc, curr) => acc + curr.stats.cache.totalMiss, 0);
+            const sharedSub = subManagerData.find(x => x.stats.cache.isShared);
+            const sharedCount = sharedSub !== undefined ? sharedSub.stats.cache.currentKeyCount : 0;
+            const scopes = req.user?.isInstanceOperator(bot) ? bot.client.scope : [];
+            let allManagerData: any = {
+                name: 'All',
+                status: bot.running ? 'RUNNING' : 'NOT RUNNING',
+                indicator: bot.running ? 'green' : 'grey',
+                maxWorkers,
+                globalMaxWorkers,
+                scopes: scopes === null || !Array.isArray(scopes) ? [] : scopes,
+                subMaxWorkers,
+                runningActivities,
+                queuedActivities,
+                botState: {
+                    state: RUNNING,
+                    causedBy: SYSTEM
+                },
+                dryRun: boolToString(bot.dryRun === true),
+                checks: checks,
+                softLimit: bot.softLimit,
+                hardLimit: bot.hardLimit,
+                stats: {
+                    ...rest,
+                    cache: {
+                        currentKeyCount: sharedCount + subManagerData.reduce((acc, curr) => curr.stats.cache.isShared ? acc : acc + curr.stats.cache.currentKeyCount,0),
+                        isShared: false,
+                        totalRequests: cacheReq,
+                        totalMiss: cacheMiss,
+                        missPercent: `${formatNumber(cacheMiss === 0 || cacheReq === 0 ? 0 : (cacheMiss / cacheReq) * 100, {toFixed: 0})}%`,
+                        types: {
+                            ...cumRaw,
+                        }
+                    }
+                },
+            };
+            // if(isOperator) {
+            allManagerData.startedAt = bot.startedAt.local().format('MMMM D, YYYY h:mm A Z');
+            allManagerData.heartbeatHuman = dayjs.duration({seconds: bot.heartbeatInterval}).humanize();
+            allManagerData.heartbeat = bot.heartbeatInterval;
+            allManagerData = {...allManagerData, ...opStats(bot)};
+            //}
+
+            const botDur = dayjs.duration(dayjs().diff(bot.startedAt))
+            if (allManagerData.stats.cache.totalRequests > 0) {
+                const minutes = botDur.asMinutes();
+                if (minutes < 10) {
+                    allManagerData.stats.cache.requestRate = formatNumber((10 / minutes) * allManagerData.stats.cache.totalRequests, {
+                        toFixed: 0,
+                        round: {enable: true, indicate: true}
+                    });
+                } else {
+                    allManagerData.stats.cache.requestRate = formatNumber(allManagerData.stats.cache.totalRequests / (minutes / 10), {
+                        toFixed: 0,
+                        round: {enable: true, indicate: true}
+                    });
+                }
+            } else {
+                allManagerData.stats.cache.requestRate = 0;
+            }
+
+            const data = {
+                bot: bot.getBotName(),
+                system: {
+                    startedAt: bot.startedAt.local().format('MMMM D, YYYY h:mm A Z'),
+                    running: bot.running,
+                    error: bot.error,
+                    ...opStats(bot),
+                },
+            };
+            return res.json(data);
+        } else {
+            // getting specific subreddit stats
+            const sd = {
+                name: manager.displayLabel,
+                botState: manager.botState,
+                eventsState: manager.eventsState,
+                queueState: manager.queueState,
+                indicator: 'gray',
+                permissions: await manager.getModPermissions(),
+                queuedActivities: manager.queue.length(),
+                runningActivities: manager.queue.running(),
+                maxWorkers: manager.queue.concurrency,
+                subMaxWorkers: manager.subMaxWorkers || bot.maxWorkers,
+                globalMaxWorkers: bot.maxWorkers,
+                validConfig: boolToString(manager.validConfigLoaded),
+                configFormat: manager.wikiFormat,
+                dryRun: boolToString(manager.dryRun === true),
+                pollingInfo: manager.pollOptions.length === 0 ? ['nothing :('] : manager.pollOptions.map(pollingInfo),
+                checks: {
+                    submissions: manager.submissionChecks === undefined ? 0 : manager.submissionChecks.length,
+                    comments: manager.commentChecks === undefined ? 0 : manager.commentChecks.length,
+                },
+                wikiRevisionHuman: manager.lastWikiRevision === undefined ? 'N/A' : `${dayjs.duration(dayjs().diff(manager.lastWikiRevision)).humanize()} ago`,
+                wikiRevision: manager.lastWikiRevision === undefined ? 'N/A' : manager.lastWikiRevision.local().format('MMMM D, YYYY h:mm A Z'),
+                wikiLastCheckHuman: `${dayjs.duration(dayjs().diff(manager.lastWikiCheck)).humanize()} ago`,
+                wikiLastCheck: manager.lastWikiCheck.local().format('MMMM D, YYYY h:mm A Z'),
+                stats: await manager.getStats(),
+                startedAt: 'Not Started',
+                startedAtHuman: 'Not Started',
+                delayBy: manager.delayBy === undefined ? 'No' : `Delayed by ${manager.delayBy} sec`,
+            };
+            // TODO replace indicator data with js on client page
+            let indicator;
+            if (manager.botState.state === RUNNING && manager.queueState.state === RUNNING && manager.eventsState.state === RUNNING) {
+                indicator = 'green';
+            } else if (manager.botState.state === STOPPED && manager.queueState.state === STOPPED && manager.eventsState.state === STOPPED) {
+                indicator = 'red';
+            } else {
+                indicator = 'yellow';
+            }
+            sd.indicator = indicator;
+            if (manager.startedAt !== undefined) {
+                const dur = dayjs.duration(dayjs().diff(manager.startedAt));
+                sd.startedAtHuman = `${dur.humanize()} ago`;
+                sd.startedAt = manager.startedAt.local().format('MMMM D, YYYY h:mm A Z');
+
+                if (sd.stats.cache.totalRequests > 0) {
+                    const minutes = dur.asMinutes();
+                    if (minutes < 10) {
+                        sd.stats.cache.requestRate = formatNumber((10 / minutes) * sd.stats.cache.totalRequests, {
+                            toFixed: 0,
+                            round: {enable: true, indicate: true}
+                        });
+                    } else {
+                        sd.stats.cache.requestRate = formatNumber(sd.stats.cache.totalRequests / (minutes / 10), {
+                            toFixed: 0,
+                            round: {enable: true, indicate: true}
+                        });
+                    }
+                } else {
+                    sd.stats.cache.requestRate = 0;
+                }
+            }
+
+            return res.json(sd);
+        }
+    }
+    return [...middleware, response];
+}
+
+export default liveStats;
--- a/src/Web/Server/routes/authenticated/user/logs.ts
+++ b/src/Web/Server/routes/authenticated/user/logs.ts
@@ -1,19 +1,29 @@
 import {Router} from '@awaitjs/express';
 import {Request, Response} from 'express';
-import {filterLogBySubreddit, isLogLineMinLevel, LogEntry, parseSubredditLogName} from "../../../../../util";
+import {
+    filterLogBySubreddit,
+    filterLogs,
+    isLogLineMinLevel,
+    LogEntry,
+    parseSubredditLogName
+} from "../../../../../util";
 import {Transform} from "stream";
 import winston from "winston";
 import pEvent from "p-event";
 import {getLogger} from "../../../../../Utils/loggerFactory";
 import {booleanMiddle} from "../../../../Common/middleware";
-import {authUserCheck, botRoute} from "../../../middleware";
+import {authUserCheck, botRoute, subredditRoute} from "../../../middleware";
 import {LogInfo} from "../../../../../Common/interfaces";
 import {MESSAGE} from "triple-beam";
+import {Manager} from "../../../../../Subreddit/Manager";
+import Bot from "../../../../../Bot";

 // TODO update logs api
-const logs = (subLogMap: Map<string, LogEntry[]>) => {
+const logs = () => {
    const middleware = [
        authUserCheck(),
+        botRoute(false),
+        subredditRoute(false),
        booleanMiddle([{
            name: 'stream',
            defaultVal: false
@@ -33,8 +43,8 @@ const logs = (subLogMap: Map<string, LogEntry[]>) => {
            try {
                logger.stream().on('log', (log: LogInfo) => {
                    if (isLogLineMinLevel(log, level as string)) {
-                        const {subreddit: subName} = log;
-                        if (isOperator || (subName !== undefined && (realManagers.includes(subName) || subName.includes(userName)))) {
+                        const {subreddit: subName, user} = log;
+                        if (isOperator || (subName !== undefined && (realManagers.includes(subName) || (user !== undefined && user.includes(userName))))) {
                            if(streamObjects) {
                                let obj: any = log;
                                if(!formatted) {
@@ -52,7 +62,7 @@ const logs = (subLogMap: Map<string, LogEntry[]>) => {
                });
                logger.info(`${userName} from ${origin} => CONNECTED`);
                await pEvent(req, 'close');
-                console.log('Request closed detected with "close" listener');
+                //logger.debug('Request closed detected with "close" listener');
                res.destroy();
                return;
            } catch (e: any) {
@@ -64,24 +74,61 @@ const logs = (subLogMap: Map<string, LogEntry[]>) => {
                res.destroy();
            }
        } else {
-            const logs = filterLogBySubreddit(subLogMap, realManagers, {
-                level: (level as string),
-                operator: isOperator,
-                user: userName,
-                sort: sort as 'descending' | 'ascending',
-                limit: Number.parseInt((limit as string)),
-                returnType: 'object',
-            });
-            const subArr: any = [];
-            logs.forEach((v: (string|LogInfo)[], k: string) => {
-                let logs = v as LogInfo[];
-                let output: any[] = formatted ? logs : logs.map((x) => {
-                    const {[MESSAGE]: fMessage, ...rest} = x;
-                    return rest;
-                })
-                subArr.push({name: k, logs: output});
-            });
-            return res.json(subArr);
+            let bots: Bot[] = [];
+            if(req.serverBot !== undefined) {
+                bots = [req.serverBot];
+            } else {
+                bots = req.user?.accessibleBots(req.botApp.bots) as Bot[];
+            }
+
+            const allReq = req.query.subreddit !== undefined && (req.query.subreddit as string).toLowerCase() === 'all';
+
+            const botArr: any = [];
+            for(const b of bots) {
+                const managerLogs = new Map<string, LogInfo[]>();
+                const managers = req.manager !== undefined ? [req.manager] : req.user?.accessibleSubreddits(b) as Manager[];
+                for (const m of managers) {
+                    const logs = filterLogs(m.logs, {
+                        level: (level as string),
+                        // @ts-ignore
+                        sort,
+                        limit: Number.parseInt((limit as string)),
+                        returnType: 'object'
+                    }) as LogInfo[];
+                    managerLogs.set(m.getDisplay(), logs);
+                }
+                const allLogs = filterLogs([...[...managerLogs.values()].flat(), ...(req.user?.isInstanceOperator(req.botApp) ? b.logs : b.logs.filter(x => x.user === req.user?.name))], {
+                    level: (level as string),
+                    // @ts-ignore
+                    sort,
+                    limit: limit as string,
+                    returnType: 'object'
+                }) as LogInfo[];
+                const systemLogs = filterLogs(req.user?.isInstanceOperator(req.botApp) ? b.logs : b.logs.filter(x => x.user === req.user?.name), {
+                    level: (level as string),
+                    // @ts-ignore
+                    sort,
+                    limit: limit as string,
+                    returnType: 'object'
+                }) as LogInfo[];
+                botArr.push({
+                    name: b.getBotName(),
+                    system: systemLogs,
+                    all: formatted ? allLogs.map(x => {
+                        const {[MESSAGE]: fMessage, ...rest} = x;
+                        return {...rest, formatted: fMessage};
+                    }) : allLogs,
+                    subreddits: allReq ? [] : [...managerLogs.entries()].reduce((acc: any[], curr) => {
+                        const l = formatted ? curr[1].map(x => {
+                            const {[MESSAGE]: fMessage, ...rest} = x;
+                            return {...rest, formatted: fMessage};
+                            }) : curr[1];
+                        acc.push({name: curr[0], logs: l});
+                        return acc;
+                    }, [])
+                });
+            }
+            return res.json(botArr);
        }
    };

--- a/src/Web/Server/routes/authenticated/user/status.ts
+++ b/src/Web/Server/routes/authenticated/user/status.ts
@@ -2,26 +2,27 @@ import {Request, Response} from 'express';
 import {
    boolToString,
    cacheStats,
-    filterLogBySubreddit,
+    filterLogBySubreddit, filterLogs,
    formatNumber,
    intersect,
-    LogEntry,
+    LogEntry, logSortFunc,
    pollingInfo
 } from "../../../../../util";
 import {Manager} from "../../../../../Subreddit/Manager";
 import dayjs from "dayjs";
-import {ResourceStats, RUNNING, STOPPED, SYSTEM} from "../../../../../Common/interfaces";
+import {LogInfo, ResourceStats, RUNNING, STOPPED, SYSTEM} from "../../../../../Common/interfaces";
 import {BotStatusResponse} from "../../../../Common/interfaces";
 import winston from "winston";
 import {opStats} from "../../../../Common/util";
-import {authUserCheck, botRoute} from "../../../middleware";
+import {authUserCheck, botRoute, subredditRoute} from "../../../middleware";
 import Bot from "../../../../../Bot";

 const status = () => {

    const middleware = [
        authUserCheck(),
-        //botRoute(),
+        botRoute(false),
+        subredditRoute(false)
    ];

    const response = async (req: Request, res: Response) => {
@@ -32,28 +33,17 @@ const status = () => {
            sort = 'descending',
        } = req.query;

-        // @ts-ignore
-        const botLogMap = req.botLogs as Map<string, Map<string, LogEntry[]>>;
-        // @ts-ignore
-        const systemLogs = req.systemLogs as LogEntry[];
+        bots = req.user?.accessibleBots(req.botApp.bots) as Bot[];

-
-        if(req.serverBot !== undefined) {
-            bots = [req.serverBot];
-        } else {
-            bots = req.user?.accessibleBots(req.botApp.bots) as Bot[];
-        }
        const botResponses: BotStatusResponse[] = [];
        let index = 1;
        for(const b of bots) {
-            botResponses.push(await botStatResponse(b, req, botLogMap, index));
+            botResponses.push(await botStatResponse(b, req, index));
            index++;
        }
        const system: any = {};
-        if(req.user?.isInstanceOperator(req.botApp)) {
-            // @ts-ignore
-            system.logs = filterLogBySubreddit(new Map([['app', systemLogs]]), [], {level, sort, limit, operator: true}).get('all');
-        }
+        // @ts-ignore
+        system.logs = filterLogs(req.sysLogs, {level, sort, limit, user: req.user?.isInstanceOperator(req.botApp) ? undefined : req.user?.name, returnType: 'object' }) as LogInfo[];
        const response = {
            bots: botResponses,
            system: system,
@@ -61,7 +51,7 @@ const status = () => {
        return res.json(response);
    }

-    const botStatResponse = async (bot: Bot, req: Request, botLogMap: Map<string, Map<string, LogEntry[]>>, index: number) => {
+    const botStatResponse = async (bot: Bot, req: Request, index: number) => {
        const {
            //subreddits = [],
            //user: userVal,
@@ -71,28 +61,26 @@ const status = () => {
            lastCheck
        } = req.query;

-        const user = req.user?.name as string;
-
-        const logs = filterLogBySubreddit(botLogMap.get(bot.botName as string) || new Map(), req.user?.accessibleSubreddits(bot).map(x => x.displayLabel) as string[], {
-            level: (level as string),
-            operator: req.user?.isInstanceOperator(req.botApp),
-            user,
-            // @ts-ignore
-            sort,
-            limit: Number.parseInt((limit as string)),
-            returnType: 'object'
-        });
+        const allReq = req.query.subreddit !== undefined && (req.query.subreddit as string).toLowerCase() === 'all';

        const subManagerData = [];
        for (const m of req.user?.accessibleSubreddits(bot) as Manager[]) {
+            const logs = req.manager === undefined || allReq || req.manager.getDisplay() === m.getDisplay() ? filterLogs(m.logs, {
+                    level: (level as string),
+                    // @ts-ignore
+                    sort,
+                    limit: limit as string,
+                    returnType: 'object'
+                }) as LogInfo[]: [];
            const sd = {
                name: m.displayLabel,
                //linkName: s.replace(/\W/g, ''),
-                logs: logs.get(m.displayLabel) || [], // provide a default empty value in case we truly have not logged anything for this subreddit yet
+                logs: logs || [], // provide a default empty value in case we truly have not logged anything for this subreddit yet
                botState: m.botState,
                eventsState: m.eventsState,
                queueState: m.queueState,
                indicator: 'gray',
+                permissions: [],
                queuedActivities: m.queue.length(),
                runningActivities: m.queue.running(),
                maxWorkers: m.queue.concurrency,
@@ -232,12 +220,22 @@ const status = () => {
        const cacheMiss = subManagerData.reduce((acc, curr) => acc + curr.stats.cache.totalMiss, 0);
        const sharedSub = subManagerData.find(x => x.stats.cache.isShared);
        const sharedCount = sharedSub !== undefined ? sharedSub.stats.cache.currentKeyCount : 0;
+        const scopes = req.user?.isInstanceOperator(bot) ? bot.client.scope : [];
+        const allSubLogs = subManagerData.map(x => x.logs).flat().sort(logSortFunc(sort as string)).slice(0, (limit as number) + 1);
+        const allLogs = filterLogs([...allSubLogs, ...(req.user?.isInstanceOperator(req.botApp) ? bot.logs : bot.logs.filter(x => x.user === req.user?.name))], {
+            level: (level as string),
+            // @ts-ignore
+            sort,
+            limit: limit as string,
+            returnType: 'object'
+        }) as LogInfo[];
        let allManagerData: any = {
            name: 'All',
            status: bot.running ? 'RUNNING' : 'NOT RUNNING',
            indicator: bot.running ? 'green' : 'grey',
            maxWorkers,
            globalMaxWorkers,
+            scopes: scopes === null || !Array.isArray(scopes) ? [] : scopes,
            subMaxWorkers,
            runningActivities,
            queuedActivities,
@@ -246,7 +244,7 @@ const status = () => {
                causedBy: SYSTEM
            },
            dryRun: boolToString(bot.dryRun === true),
-            logs: logs.get('all'),
+            logs: allLogs,
            checks: checks,
            softLimit: bot.softLimit,
            hardLimit: bot.hardLimit,
@@ -302,7 +300,7 @@ const status = () => {
                name: (bot.botName as string) ?? `Bot ${index}`,
                ...opStats(bot),
            },
-            subreddits: [allManagerData, ...subManagerData],
+            subreddits: [allManagerData, ...(allReq ? subManagerData.map(({logs, ...x}) => ({...x, logs: []})) : subManagerData)],

        };

--- a/src/Web/Server/server.ts
+++ b/src/Web/Server/server.ts
@@ -9,11 +9,6 @@ import {Strategy as JwtStrategy, ExtractJwt} from 'passport-jwt';
 import passport from 'passport';
 import tcpUsed from 'tcp-port-used';

-import {
-    intersect,
-    LogEntry, parseBotLogName,
-    parseSubredditLogName
-} from "../../util";
 import {getLogger} from "../../Utils/loggerFactory";
 import LoggedError from "../../Utils/LoggedError";
 import {Invokee, LogInfo, OperatorConfigWithFileContext} from "../../Common/interfaces";
@@ -21,13 +16,13 @@ import http from "http";
 import {heartbeat} from "./routes/authenticated/applicationRoutes";
 import logs from "./routes/authenticated/user/logs";
 import status from './routes/authenticated/user/status';
+import liveStats from './routes/authenticated/user/liveStats';
 import {actionedEventsRoute, actionRoute, configRoute, configLocationRoute, deleteInviteRoute, addInviteRoute, getInvitesRoute} from "./routes/authenticated/user";
 import action from "./routes/authenticated/user/action";
 import {authUserCheck, botRoute} from "./middleware";
 import {opStats} from "../Common/util";
 import Bot from "../../Bot";
 import addBot from "./routes/authenticated/user/addBot";
-import dayjs from "dayjs";
 import ServerUser from "../Common/User/ServerUser";
 import {SimpleError} from "../../Utils/Errors";
 import {ErrorWithCause} from "pony-cause";
@@ -47,11 +42,7 @@ declare module 'express-session' {
    }
 }

-const subLogMap: Map<string, LogEntry[]> = new Map();
-const systemLogs: LogEntry[] = [];
-const botLogMap: Map<string, Map<string, LogEntry[]>> = new Map();
-
-const botSubreddits: Map<string, string[]> = new Map();
+let sysLogs: LogInfo[] = [];

 const rcbServer = async function (options: OperatorConfigWithFileContext) {

@@ -73,36 +64,12 @@ const rcbServer = async function (options: OperatorConfigWithFileContext) {
    const logger = getLogger({...options.logging});

    logger.stream().on('log', (log: LogInfo) => {
-        const logEntry: LogEntry = [dayjs(log.timestamp).unix(), log];

        const {bot: botName, subreddit: subName} = log;

-        if(botName === undefined) {
-            systemLogs.unshift(logEntry);
-            systemLogs.slice(0, 201);
-        } else {
-            const botLog = botLogMap.get(botName) || new Map();
-
-            if(subName === undefined) {
-                const appLogs = botLog.get('app') || [];
-                appLogs.unshift(logEntry);
-                botLog.set('app', appLogs.slice(0, 200 + 1));
-            } else {
-                let botSubs = botSubreddits.get(botName) || [];
-                if(app !== undefined && (botSubs.length === 0 || !botSubs.includes(subName))) {
-                    const b = app.bots.find(x => x.botName === botName);
-                    if(b !== undefined) {
-                        botSubs = b.subManagers.map(x => x.displayLabel);
-                        botSubreddits.set(botName, botSubs);
-                    }
-                }
-                if(botSubs.length === 0 || botSubs.includes(subName)) {
-                    const subLogs = botLog.get(subName) || [];
-                    subLogs.unshift(logEntry);
-                    botLog.set(subName, subLogs.slice(0, 200 + 1));
-                }
-            }
-            botLogMap.set(botName, botLog);
+        if(botName === undefined && subName === undefined) {
+            sysLogs.unshift(log);
+            sysLogs = sysLogs.slice(0, 201);
        }
    })

@@ -167,7 +134,7 @@ const rcbServer = async function (options: OperatorConfigWithFileContext) {

    server.getAsync('/heartbeat', ...heartbeat({name, display, friendly}));

-    server.getAsync('/logs', ...logs(subLogMap));
+    server.getAsync('/logs', ...logs());

    server.getAsync('/stats', [authUserCheck(), botRoute(false)], async (req: Request, res: Response) => {
        let bots: Bot[] = [];
@@ -186,13 +153,13 @@ const rcbServer = async function (options: OperatorConfigWithFileContext) {
    });
    const passLogs = async (req: Request, res: Response, next: Function) => {
        // @ts-ignore
-        req.botLogs = botLogMap;
-        // @ts-ignore
-        req.systemLogs = systemLogs;
+        req.sysLogs = sysLogs;
        next();
    }
    server.getAsync('/status', passLogs, ...status())

+    server.getAsync('/liveStats', ...liveStats())
+
    server.getAsync('/config', ...configRoute);

    server.getAsync('/config/location', ...configLocationRoute);
--- a/src/Web/assets/public/app.css
+++ b/src/Web/assets/public/app.css
@@ -152,3 +152,7 @@ a {
 #saveTip .tooltip:hover {
    transition-delay: 1s;
 }
+
+#redditStatus .iconify-inline {
+    display: inline;
+}
--- a/src/Web/assets/views/partials/footer.ejs
+++ b/src/Web/assets/views/partials/footer.ejs
@@ -1,5 +1,29 @@
 <div class="py-3 flex items-center justify-around font-semibold">
    <div>
-        <a href="https://github.com/FoxxMD/context-mod">ContextMod Web</a> created by /u/FoxxMD
+        <a href="https://github.com/FoxxMD/context-mod">ContextMod <%= locals.applicationIdentifier%></a> created by /u/FoxxMD
    </div>
 </div>
+<script type="text/javascript" src="https://cdn.statuspage.io/se-v2.js"></script>
+<script>
+    // https://www.redditstatus.com/api#status
+    var sp = new StatusPage.page({ page : '2kbc0d48tv3j' });
+    sp.status({
+        success : function(data) {
+            console.log(data.status.indicator);
+            switch(data.status.indicator){
+                case 'minor':
+                    document.querySelector('#redditStatus').innerHTML = '<span class="iconify-inline yellow" data-icon="ep:warning-filled"></span>';
+                    break;
+                case 'none':
+                    document.querySelector('#redditStatus').innerHTML = '<span class="iconify-inline green" data-icon="ep:circle-check-filled"></span>';
+                    break;
+                default:
+                    document.querySelector('#redditStatus').innerHTML = '<span class="iconify-inline red" data-icon="ep:warning-filled"></span>';
+                    break;
+            }
+            // data.page.updated_at
+            // data.status.indicator => none, minor, major, or critical
+            // data.status.description
+        }
+    });
+</script>
--- a/src/Web/assets/views/partials/head.ejs
+++ b/src/Web/assets/views/partials/head.ejs
@@ -1,6 +1,6 @@
 <head>
    <link rel="stylesheet" href="/public/tailwind.min.css"/>
-    <script src="https://code.iconify.design/2/2.1.0/iconify.min.js"></script>
+    <script src="https://code.iconify.design/2/2.1.2/iconify.min.js"></script>
    <link rel="stylesheet" href="/public/themeToggle.css">
    <link rel="stylesheet" href="/public/app.css">
    <title><%= locals.title !== undefined ? title : `${locals.botName !== undefined ? `CM for ${botName}` : 'ContextMod'}`%></title>
--- a/src/Web/assets/views/partials/header.ejs
+++ b/src/Web/assets/views/partials/header.ejs
@@ -33,7 +33,12 @@
                </ul>
                <% } %>
            </div>
-            <div class="flex items-center flex-end text-sm">
+            <div class="flex items-center mr-8 text-sm">
+                <a href="https://redditstatus.com" target="_blank">
+                    <span>Reddit Status: <span id="redditStatus" class="ml-2"><span class="iconify-inline" data-icon="ep:question-filled"></span></span></span>
+                </a>
+            </div>
+            <div class="flex items-center text-sm">
                <a href="logout">Logout</a>
            </div>
        </div>
--- a/src/Web/assets/views/status.ejs
+++ b/src/Web/assets/views/status.ejs
@@ -160,6 +160,19 @@
                                                        </div>
                                                    </div>
                                                </div>
+                                                <label>Mod Perms</label>
+                                                <span class="has-tooltip">
+                                                    <span class='tooltip rounded shadow-lg p-1 bg-gray-100 text-black -mt-2'>
+                                                        <span>
+                                                            <ul class="list-inside list-disc modPermissionsList">
+                                                                <% data.permissions.forEach(function (i){ %>
+                                                                    <li class="font-mono"><%= i %></li>
+                                                                <% }) %>
+                                                            </ul>
+                                                        </span>
+                                                    </span>
+                                                    <span class="cursor-help underline modPermissionsCount" style="text-decoration-style: dotted"><%= data.permissions.length %></span>
+                                                </span>
                                                <label>Slow Mode</label>
                                                <span><%= data.delayBy %></span>
                                            <% } %>
@@ -223,6 +236,19 @@
                                            <% if (data.name === 'All' && isOperator) { %>
                                                <label>Operators</label>
                                                <span><%= operators %></span>
+                                                <label>Oauth Scopes</label>
+                                                    <span class="has-tooltip">
+                                                    <span class='tooltip rounded shadow-lg p-1 bg-gray-100 text-black -mt-2'>
+                                                        <span>
+                                                            <ul class="list-inside list-disc">
+                                                                <% data.scopes.forEach(function (i){ %>
+                                                                    <li class="font-mono"><%= i %></li>
+                                                                <% }) %>
+                                                            </ul>
+                                                        </span>
+                                                    </span>
+                                                    <span class="cursor-help underline" style="text-decoration-style: dotted"><%= data.scopes.length %></span>
+                                                </span>
                                            <% } else %>
                                        </div>
                                        <% if (data.name !== 'All') { %>
@@ -651,6 +677,48 @@
 </div>
 <%- include('partials/instanceTabJs') %>
 <%- include('partials/logSettingsJs') %>
+<script src="https://unpkg.com/autolinker@3.14.3/dist/Autolinker.min.js" crossorigin="anonymous"></script>
+<script src="https://unpkg.com/dayjs@1.10.7/dayjs.min.js" crossorigin="anonymous"></script>
+<script src="https://unpkg.com/dayjs@1.10.7/plugin/advancedFormat.js"></script>
+<script src="https://unpkg.com/dayjs@1.10.7/plugin/timezone.js"></script>
+<script>
+    dayjs.extend(window.dayjs_plugin_timezone)
+    dayjs.extend(window.dayjs_plugin_advancedFormat)
+    window.formattedTime = (short, full) => `<span class="has-tooltip"><span style="margin-top:35px" class='tooltip rounded shadow-lg p-1 bg-gray-100 text-black space-y-3 p-2 text-left'>${full}</span><span>${short}</span></span>`;
+    window.formatLogLineToHtml = (log, timestamp = undefined) => {
+        const val = typeof log === 'string' ? log : log['MESSAGE'];
+        const logContent = Autolinker.link(val, {
+            email: false,
+            phone: false,
+            mention: false,
+            hashtag: false,
+            stripPrefix: false,
+            sanitizeHtml: true,
+        })
+            .replace(/(\s*debug\s*):/i, '<span class="debug blue">$1</span>:')
+            .replace(/(\s*warn\s*):/i, '<span class="warn yellow">$1</span>:')
+            .replace(/(\s*info\s*):/i, '<span class="info green">$1</span>:')
+            .replace(/(\s*error\s*):/i, '<span class="error red">$1</span>:')
+            .replace(/(\s*verbose\s*):/i, '<span class="error purple">$1</span>:')
+            .replaceAll('\n', '<br />');
+        //.replace(HYPERLINK_REGEX, '<a target="_blank" href="$&">$&</a>');
+        let line = '';
+
+        let timestampString = timestamp;
+        if(timestamp === undefined && typeof log !== 'string') {
+            timestampString = log.timestamp;
+        }
+
+        if(timestampString !== undefined) {
+            const timeStampReplacement = formattedTime(dayjs(timestampString).format('HH:mm:ss z'), timestampString);
+            const splitLine = logContent.split(timestampString);
+            line = `<div class="logLine">${splitLine[0]}${timeStampReplacement}<span style="white-space: pre-wrap">${splitLine[1]}</span></div>`;
+        } else {
+            line = `<div style="white-space: pre-wrap" class="logLine">${logContent}</div>`
+        }
+        return line;
+    }
+</script>
 <script>
    window.sort = 'desc';

@@ -811,6 +879,44 @@
                var newRelativePathQuery = window.location.pathname + '?' + searchParams.toString();
                history.pushState(null, '', newRelativePathQuery);
            }
+            const activeSub = document.querySelector(`[data-subreddit="${subreddit}"][data-bot="${bot}"].sub`);
+            if(!activeSub.classList.contains('seen')) {
+                //firstSub.classList.add('seen');
+
+                //subreddit = firstSub.dataset.subreddit;
+                //bot = subSection.dataset.bot;
+                level = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="level"]`).value;
+                sort = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="sort"]`).value;
+                limitSel = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="limit"]`).value;
+
+                fetch(`/api/logs?instance=<%= instanceId %>&bot=${bot}&subreddit=${subreddit}&level=${level}&sort=${sort}&limit=${limitSel}&stream=false&formatted=true`).then((resp) => {
+                    if (!resp.ok) {
+                        console.error('Response was not OK from logs GET');
+                    } else {
+                        resp.json().then((data) => {
+                            const logContainer = document.querySelector(`[data-subreddit="${subreddit}"] .logs`);
+                            const logLines = (subreddit.toLowerCase() === 'all' ? data[0].all : data[0].subreddits[0].logs).map(x => {
+                                let fString = x.formatted;
+                                if(x.bot !== undefined) {
+                                    fString = fString.replace(`~${x.bot}~ `, '');
+                                }
+                                if(x.subreddit !== undefined && subreddit !== 'All') {
+                                    fString = fString.replace(`{${x.subreddit}} `, '');
+                                }
+                                return window.formatLogLineToHtml(fString, x.timestamp)
+                            }).join('');
+                            logContainer.insertAdjacentHTML('afterbegin', logLines);
+                            activeSub.classList.add('seen');
+                        });
+                    }
+                }).catch((err) => {
+                    console.log(err);
+                });
+
+            }
+            if(window.socket !== undefined) {
+                window.socket.emit('viewing', {bot, subreddit});
+            }
        });
    });

@@ -838,7 +944,7 @@
        tabSelect.classList.add('font-bold', 'no-underline', 'pointer-events-none');
    }
    document.querySelectorAll('.tabSelectWrapper').forEach(el => el.classList.add('border'));
-    document.querySelector(`[data-bot="${shownBot}"][data-subreddit="${shownSub}"].sub`).classList.add('active');
+    document.querySelector(`[data-bot="${shownBot}"][data-subreddit="${shownSub}"].sub`).classList.add('active', 'seen');
    const subWrapper = document.querySelector(`ul[data-bot="${shownBot}"] [data-subreddit="${shownSub}"].tabSelectWrapper`);
    if(subWrapper !== null) {
        subWrapper.classList.remove('border');
@@ -865,6 +971,7 @@
    let socket = io({
        reconnectionAttempts: 5, // bail after 5 attempts
    });
+    window.socket = socket;

    // get all bots
    let bots = [];
@@ -882,6 +989,11 @@

    socket.on("connect", () => {
        document.body.classList.add('connected')
+
+        const shownSub = searchParams.get('sub') || 'All'
+        let shownBot = searchParams.get('bot');
+        window.socket.emit('viewing', {bot: shownBot, subreddit: shownSub});
+
        socket.on("log", data => {
            const {
                subreddit,
@@ -916,12 +1028,12 @@
                    }
                    subLogs.forEach((logs, subKey) => {
                        // check sub exists -- may be a web log
-                        const el = document.querySelector(`[data-subreddit="${subKey}"][data-bot="${botName}"].sub`);
+                        const el = document.querySelector(`[data-subreddit="${subKey}"][data-bot="${botName}"].sub.seen`);
                        if(null !== el) {
                            const limit = Number.parseInt(document.querySelector(`[data-subreddit="${subKey}"] [data-type="limit"]`).value);
                            const logContainer = el.querySelector(`.logs`);
                            let existingLogs;
-                            if(window.sort === 'desc') {
+                            if(window.sort === 'desc' || window.sort === 'descending') {
                                logs.forEach((l) => {
                                    logContainer.insertAdjacentHTML('afterbegin', l);
                                })
@@ -973,6 +1085,29 @@
            }

        });
+        socket.on('liveStats', (resp) => {
+            let el;
+            let isAll = resp.system !== undefined;
+            if(isAll) {
+                // got all
+                el = document.querySelector(`[data-subreddit="All"][data-bot="${resp.bot}"].sub`);
+            } else {
+                // got subreddit
+                el = document.querySelector(`[data-subreddit="${resp.name}"].sub`);
+            }
+
+            if(isAll) {
+
+            } else {
+                if(el.querySelector('.modPermissionsCount').innerHTML != resp.permissions.length) {
+                    el.querySelector('.modPermissionsCount').innerHTML = resp.permissions.length;
+                    el.querySelector('.modPermissionsList').innerHTML = '';
+                    el.querySelector('.modPermissionsList').insertAdjacentHTML('afterbegin', resp.permissions.map(x => `<li class="font-mono">${x}</li>`).join(''));
+                }
+            }
+
+            console.log(resp);
+        });
    });

    socket.on('disconnect', () => {
--- a/src/Web/interfaces.ts
+++ b/src/Web/interfaces.ts
@@ -1,5 +1,6 @@
 import { URL } from "url";
 import {BotConnection} from "../Common/interfaces";
+import {Logger} from "winston";

 export interface BotInstance {
    botName: string
@@ -8,16 +9,16 @@ export interface BotInstance {
    subreddits: string[]
    nanny?: string
    running: boolean
-    instance: CMInstance
+    instance: CMInstanceInterface
 }

-export interface CMInstance extends BotConnection {
-    friendly: string
+export interface CMInstanceInterface extends BotConnection {
+    friendly?: string
    operators: string[]
    operatorDisplay: string
    url: URL,
    normalUrl: string,
-    lastCheck: number
+    lastCheck?: number
    online: boolean
    subreddits: string[]
    bots: BotInstance[]
--- a/src/Web/types/express/index.d.ts
+++ b/src/Web/types/express/index.d.ts
@@ -1,6 +1,6 @@
 import {App} from "../../../App";
 import Bot from "../../../Bot";
-import {BotInstance, CMInstance} from "../../interfaces";
+import {BotInstance, CMInstanceInterface} from "../../interfaces";
 import {Manager} from "../../../Subreddit/Manager";
 import CMUser from "../../Common/User/CMUser";

@@ -9,7 +9,7 @@ declare global {
        interface Request {
            botApp: App;
            token?: string,
-            instance?: CMInstance,
+            instance?: CMInstanceInterface,
            bot?: BotInstance,
            serverBot: Bot,
            manager?: Manager,
--- a/src/util.ts
+++ b/src/util.ts
@@ -6,8 +6,6 @@ import deepEqual from "fast-deep-equal";
 import {Duration} from 'dayjs/plugin/duration.js';
 import Ajv from "ajv";
 import {InvalidOptionArgumentError} from "commander";
-import Submission from "snoowrap/dist/objects/Submission";
-import {Comment} from "snoowrap";
 import {inflateSync, deflateSync} from "zlib";
 import pixelmatch from 'pixelmatch';
 import os from 'os';
@@ -37,14 +35,14 @@ import {
 import { Document as YamlDocument } from 'yaml'
 import InvalidRegexError from "./Utils/InvalidRegexError";
 import {constants, promises} from "fs";
-import {cacheOptDefaults} from "./Common/defaults";
+import {cacheOptDefaults, VERSION} from "./Common/defaults";
 import cacheManager, {Cache} from "cache-manager";
 import redisStore from "cache-manager-redis-store";
 import crypto from "crypto";
 import Autolinker from 'autolinker';
 import {create as createMemoryStore} from './Utils/memoryStore';
 import {MESSAGE, LEVEL} from "triple-beam";
-import {RedditUser} from "snoowrap/dist/objects";
+import {RedditUser,Comment,Submission} from "snoowrap/dist/objects";
 import reRegExp from '@stdlib/regexp-regexp';
 import fetch, {Response} from "node-fetch";
 import { URL } from "url";
@@ -160,7 +158,7 @@ const isProbablyError = (val: any, errName = 'error') => {
    return typeof val === 'object' && val.name !== undefined && val.name.toLowerCase().includes(errName);
 }

-export const PASS = '✔';
+export const PASS = '✓';
 export const FAIL = '✘';

 export const truncateStringToLength = (length: number, truncStr = '...') => (str: string) => str.length > length ? `${str.slice(0, length - truncStr.length - 1)}${truncStr}` : str;
@@ -1161,9 +1159,14 @@ export const formatLogLineToHtml = (log: string | LogInfo, timestamp?: string) =
        //.replace(HYPERLINK_REGEX, '<a target="_blank" href="$&">$&</a>');
    let line = '';

-    if(timestamp !== undefined) {
-        const timeStampReplacement = formattedTime(dayjs(timestamp).format('HH:mm:ss z'), timestamp);
-        const splitLine = logContent.split(timestamp);
+    let timestampString = timestamp;
+    if(timestamp === undefined && typeof log !== 'string') {
+        timestampString = (log as LogInfo).timestamp;
+    }
+
+    if(timestampString !== undefined) {
+        const timeStampReplacement = formattedTime(dayjs(timestampString).format('HH:mm:ss z'), timestampString);
+        const splitLine = logContent.split(timestampString);
        line = `<div class="logLine">${splitLine[0]}${timeStampReplacement}<span style="white-space: pre-wrap">${splitLine[1]}</span></div>`;
    } else {
        line = `<div style="white-space: pre-wrap" class="logLine">${logContent}</div>`
@@ -1173,7 +1176,7 @@ export const formatLogLineToHtml = (log: string | LogInfo, timestamp?: string) =

 export type LogEntry = [number, LogInfo];
 export interface LogOptions {
-    limit: number,
+    limit: number | string,
    level: string,
    sort: 'ascending' | 'descending',
    operator?: boolean,
@@ -1185,7 +1188,7 @@ export interface LogOptions {

 export const filterLogBySubreddit = (logs: Map<string, LogEntry[]>, validLogCategories: string[] = [], options: LogOptions): Map<string, (string|LogInfo)[]> => {
    const {
-        limit,
+        limit: limitVal,
        level,
        sort,
        operator = false,
@@ -1195,6 +1198,7 @@ export const filterLogBySubreddit = (logs: Map<string, LogEntry[]>, validLogCate
        returnType = 'string',
    } = options;

+    let limit = typeof limitVal === 'number' ? limitVal : Number.parseInt(limitVal);
    // get map of valid logs categories
    const validSubMap: Map<string, LogEntry[]> = new Map();
    for(const [k, v] of logs) {
@@ -1244,6 +1248,35 @@ export const filterLogBySubreddit = (logs: Map<string, LogEntry[]>, validLogCate
    return preparedMap;
 }

+export const logSortFunc = (sort: string = 'ascending') => sort === 'ascending' ? (a: LogInfo, b: LogInfo) => (dayjs(a.timestamp).isSameOrAfter(b.timestamp) ? 1 : -1) : (a: LogInfo, b: LogInfo) => (dayjs(a.timestamp).isSameOrBefore(b.timestamp) ? 1 : -1);
+
+export const filterLogs= (logs: LogInfo[], options: LogOptions): LogInfo[] | string[] => {
+    const {
+        limit: limitVal,
+        level,
+        sort,
+        operator = false,
+        user,
+        allLogsParser = parseSubredditLogInfoName,
+        allLogName = 'app',
+        returnType = 'string',
+    } = options;
+
+    let limit = typeof limitVal === 'number' ? limitVal : Number.parseInt(limitVal);
+    let leveledLogs = logs.filter(x => isLogLineMinLevel(x, level));
+    if(user !== undefined) {
+        leveledLogs = logs.filter(x => x.user !== undefined && x.user === user);
+    }
+    leveledLogs.sort(logSortFunc(sort));
+    leveledLogs = leveledLogs.slice(0, limit + 1);
+
+    if(returnType === 'string') {
+        return leveledLogs.map(x => formatLogLineToHtml(x));
+    } else {
+        return leveledLogs;
+    }
+}
+
 export const logLevels = {
    error: 0,
    warn: 1,
@@ -1577,13 +1610,33 @@ export const snooLogWrapper = (logger: Logger) => {
 * Cached activities lose type information when deserialized so need to check properties as well to see if the object is the shape of a Submission
 * */
 export const isSubmission = (value: any) => {
-    return value instanceof Submission || value.domain !== undefined;
+    return value instanceof Submission || value.name.includes('t3_');
 }

 export const asSubmission = (value: any): value is Submission => {
    return isSubmission(value);
 }

+export const isComment = (value: any) => {
+    return value instanceof Comment || value.name.includes('t1_');
+}
+
+export const asComment = (value: any): value is Comment => {
+    return isComment(value);
+}
+
+export const asActivity = (value: any): value is (Submission | Comment) => {
+    return asComment(value) || asSubmission(value);
+}
+
+export const isUser = (value: any) => {
+    return value instanceof RedditUser || value.name.includes('t2_');
+}
+
+export const asUser = (value: any): value is RedditUser => {
+    return isUser(value);
+}
+
 export const isUserNoteCriteria = (value: any) => {
    return value !== null && typeof value === 'object' && value.type !== undefined;
 }
@@ -2020,3 +2073,46 @@ export const likelyJson5 = (str: string): boolean => {
    }
    return validStart;
 }
+
+const defaultScanOptions = {
+    COUNT: '100',
+    MATCH: '*'
+}
+/**
+ * Frankenstein redis scan generator
+ *
+ * Cannot use the built-in scan iterator because it is only available in > v4 of redis client but node-cache-manager-redis is using v3.x --
+ * So combining the async iterator defined in v4 from here https://github.com/redis/node-redis/blob/master/packages/client/lib/client/index.ts#L587
+ * with the scan example from v3 https://github.com/redis/node-redis/blob/8a43dea9bee11e41d33502850f6989943163020a/examples/scan.js
+ *
+ * */
+export async function* redisScanIterator(client: any, options: any = {}): AsyncIterable<string> {
+    let cursor: string = '0';
+    const scanOpts = {...defaultScanOptions, ...options};
+    do {
+        const iterScan = new Promise((resolve, reject) => {
+            client.scan(cursor, 'MATCH', scanOpts.MATCH, 'COUNT', scanOpts.COUNT, (err: any, res: any) => {
+                if(err) {
+                    return reject(err);
+                } else {
+                    const newCursor = res[0];
+                    let keys = res[1];
+                    resolve([newCursor, keys]);
+                }
+            });
+        }) as Promise<[any, string[]]>;
+        const [newCursor, keys] = await iterScan;
+        cursor = newCursor;
+        for (const key of keys) {
+            yield key;
+        }
+    } while (cursor !== '0');
+}
+
+export const getUserAgent = (val: string, fragment?: string) => {
+    return `${replaceApplicationIdentifier(val, fragment)} (developed by /u/FoxxMD)`;
+}
+
+export const replaceApplicationIdentifier = (val: string, fragment?: string) => {
+    return val.replace('{VERSION}', `v${VERSION}`).replace('{FRAG}', (fragment !== undefined ? `-${fragment}` : ''));
+}
Author	SHA1	Message	Date
FoxxMD	ae8e11feb4	Merge branch 'edge'	2022-02-22 11:11:46 -05:00
FoxxMD	5cd415e300	Bump version	2022-02-22 11:11:29 -05:00
FoxxMD	7cdaa4bf25	fix(migrations): Remove unnecessary log warning for all logs on live stats	2022-02-22 11:10:51 -05:00
FoxxMD	4969cafc97	fix(ui): Add missing dayjs plugins for timestamp formatting	2022-02-22 10:43:17 -05:00
FoxxMD	88bafbc1ac	fix(ui): Fix not clearing intervals on client disconnect	2022-02-21 16:47:17 -05:00
FoxxMD	a5acd6ec83	feat: Refactor client/secret api interaction to improve fetching data and enable live stats * Only return logs for "default viewed" subreddit/bot when fetching instance status, when specified from QS * Greatly reduces amount of data fetched and response time * Return logs with formatted property for non-streaming response * Implement server live stats endpoint to return subreddit/all stats based on QS * Use client websocket connection to return stats for currently viewed subreddit	2022-02-21 16:14:41 -05:00
FoxxMD	d93c8bdef2	Merge branch 'docUpdates' into edge	2022-02-21 12:00:37 -05:00
FoxxMD	8a32bd6485	Merge branch 'edge' into logRefactor # Conflicts: # src/Web/Client/index.ts	2022-02-18 15:45:28 -05:00
FoxxMD	425cbc4826	feat: Improve user agent reporting and version display in ui	2022-02-18 15:16:37 -05:00
FoxxMD	3a2d3f5047	refactor(logging): Use logging from CMInstance instead of client Reduces logging complexity and has better single responsibility	2022-02-18 13:37:07 -05:00
FoxxMD	ae20b85400	refactor(client): Refactor server instance into own class * Move from plain data with interface to class and refactor heartbeat logic into class * Makes logging easier and cleans up client code	2022-02-18 13:09:33 -05:00
FoxxMD	e07b8cc291	Merge branch 'edge'	2022-02-18 11:58:28 -05:00
FoxxMD	e993c5d376	refactor(logging): Move log collection into bot/manager for better single responsibility * Move "sorting" log objects into lists for retrieval from server and into bot/managers for each log object type * Refactor log filtering and aggregration under status/log endpoints to use logs from each entity rather than pulling from server Reduces complexity in historical log data structures at the expense of slightly more runtime data crunching. The trade-off is well worth it and paves the way for easier retrieval of single/subsets of logs	2022-02-18 11:58:13 -05:00
FoxxMD	80fabeac54	fix(usernote): Fix adding new note to user note cache AFTER clearing cache * Fixes an issue where the cached notes for a user only contain the last added note instead of all notes + new * Also reduced api calls by caching moderator adding new note instead of calling each time	2022-02-18 09:54:18 -05:00
FoxxMD	c001be9abf	feat(ui): Add reddit status indicator with link	2022-02-17 16:14:36 -05:00
FoxxMD	639a542fb2	fix(ui): Fix default values for scopes and permissions when not available	2022-02-17 13:53:06 -05:00
FoxxMD	9299258de0	feat(ui): Add moderator permissions list to subreddit overview	2022-02-17 13:37:42 -05:00
FoxxMD	59f8ac6dd4	feat(ui): Add oauth scopes list to bot overview Visible when user is an operator	2022-02-17 13:29:37 -05:00
FoxxMD	f16155bb1f	fix(flair): Fix snoowrap function used for assigning flair template id	2022-02-17 13:17:17 -05:00
FoxxMD	e2d2f73bb3	feat: Add log warning when user has no access	2022-02-15 11:00:09 -05:00
FoxxMD	9ca5d6c8c2	fix: Fix config builder to supply more defaults for a minimal configuration * Provide a default redirect uri * Don't add default bot instance if no credentials were specified	2022-02-14 12:07:54 -05:00
FoxxMD	4f9d1c1ca1	docs: Some clarifications for install/run directions	2022-02-14 10:54:25 -05:00
FoxxMD	d8f673bd26	fix(remove): Only warn if item looks removed, on remove action If the item is not actually removed (it's hard to tell from reddit api) we don't want to prematurely end remove action. Just warn and try to remove anyway	2022-02-14 09:31:50 -05:00
FoxxMD	7e2068d82a	fix(author): Ensure automoderator is always detected as a moderator for author isMod test	2022-02-14 09:30:54 -05:00
FoxxMD	176611dbf3	docs: Add web interface and config onboarding	2022-02-11 23:40:28 -05:00
FoxxMD	3d99406f33	Merge branch 'persistActions' into edge	2022-02-09 17:09:24 -05:00
FoxxMD	ab355977ba	fix(approve): Fix approval check target	2022-02-09 16:41:39 -05:00
FoxxMD	8667fcdef3	fix(stats): Correctly initialize all time historical stats from cache when stat is empty	2022-02-09 13:10:05 -05:00
FoxxMD	ec20445772	refactor(ui): Use checkmark symbol that matches x symbol (no emojis)	2022-02-09 13:09:39 -05:00
FoxxMD	0293928a99	feat(cache): Implement cache key manipulation based on key pattern * Implement glob pattern or regex as argument * Implement scan search for redis for efficiency otherwise iterate keys using generic function * Implement cache reset based on passed item from action -- reset item crit for activities, author crit for users, and overwrite any cached activity	2022-02-08 13:01:09 -05:00
FoxxMD	b56d6dbe7c	fix(actions): Only include successfully run actions in notification summary	2022-02-07 22:21:22 -05:00
FoxxMD	42d269e28d	feat(actions): Mutate activities during actions for immediate use and ensure cache is synced	2022-02-07 16:21:43 -05:00
FoxxMD	8f60a1da53	feat(regex): Add option to stop rule early if current activity does not match In order to prevent history from being pulled (and using api) if user indicates current activity must also match	2022-02-07 15:15:50 -05:00
FoxxMD	f511be7c33	fix(usernote): Throw error with cause when usernote fails instead of logging quietly * Makes error cause easier to see in stack and fixes error now logging during action failure * Use error with cause for logging action error for clearer stack	2022-02-07 12:41:10 -05:00
FoxxMD	ebb426e696	feat(filter): Add isRedditMediaDomain submission state criteria	2022-02-07 10:36:56 -05:00