Merge branch 'edge'

* Only return logs for "default viewed" subreddit/bot when fetching instance status, when specified from QS
  * Greatly reduces amount of data fetched and response time
* Return logs with formatted property for non-streaming response
* Implement server live stats endpoint to return subreddit/all stats based on QS
* Use client websocket connection to return stats for currently viewed subreddit

.github/FUNDING.yml

@@ -0,0 +1,2 @@
+github: [FoxxMD]
+custom: ["bitcoincash:qqmpsh365r8n9jhp4p8ks7f7qdr7203cws4kmkmr8q"]

docs/gettingStartedOperator.md

@@ -22,13 +22,14 @@ PROTIP: Using a container management tool like [Portainer.io CE](https://www.por
 
 ### [Dockerhub](https://hub.docker.com/r/foxxmd/context-mod)
 
-```
-foxxmd/context-mod:latest
-```
+An example of starting the container using the [minimum configuration](/docs/operatorConfiguration.md#minimum-config) with a [configuration file](/docs/operatorConfiguration.md#defining-configuration-via-file):
+
+* Bind the folder where the config is located on your host machine into the container `-v /host/path/folder:/config`
+* Tell CM where to find the config using an env `-e "OPERATOR_CONFIG=/config/myConfig.yaml"`
+* Expose the web interface using the container port `8085`
 
-Adding **environmental variables** to your `docker run` command will pass them through to the app EX:
 ```
-docker run -d -e "CLIENT_ID=myId" ... foxxmd/context-mod
+docker run -d -e "OPERATOR_CONFIG=/config/myConfig.yaml" -v /host/path/folder:/config -p 8085:8085 foxxmd/context-mod
 ```
 
 ### Locally
@@ -47,6 +48,12 @@ npm install
 tsc -p .
 ```
 
+An example of running CM using the [minimum configuration](/docs/operatorConfiguration.md#minimum-config) with a [configuration file](/docs/operatorConfiguration.md#defining-configuration-via-file):
+
+```bash
+node src/index.js run
+```
+
 ### [Heroku Quick Deploy](https://heroku.com/about)
 [![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://dashboard.heroku.com/new?template=https://github.com/FoxxMD/context-mod)
 

docs/operatorConfiguration.md

@@ -41,8 +41,10 @@ configuration.
 **Note:** When reading the **schema** if the variable is available at a level of configuration other than **FILE** it will be
 noted with the same symbol as above. The value shown is the default.
 
-* To load a JSON configuration (for **FILE**) **from the command line** use the `-c` cli argument EX: `node src/index.js -c /path/to/JSON/config.json`
-* To load a JSON configuration (for **FILE**) **using an environmental variable** use `OPERATOR_CONFIG` EX: `OPERATOR_CONFIG=/path/to/JSON/config.json`
+## Defining Configuration Via File
+
+* **from the command line**  use the `-c` cli argument EX: `node src/index.js -c /path/to/JSON/config.json`
+* **using an environmental variable**  use `OPERATOR_CONFIG` EX: `OPERATOR_CONFIG=/path/to/JSON/config.json`
 
 [**See the Operator Config Schema here**](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2FFoxxMD%2Fcontext-mod%2Fmaster%2Fsrc%2FSchema%2FOperatorConfig.json)
 
@@ -121,28 +123,41 @@ Below are examples of the minimum required config to run the application using a
 Using **FILE**
 <details>
 
+CM will look for a file configuration at `PROJECT_DIR/config.yaml` by default [or you can specify your own location.](#defining-configuration-via-file)
+
 YAML
 ```yaml
+operator:
+  name: YourRedditUsername
 bots:
   - credentials:
       clientId: f4b4df1c7b2
       clientSecret: 34v5q1c56ub
-      refreshToken: 34_f1w1v4
-      accessToken: p75_1c467b2
+web:
+  credentials:
+    clientId: f4b4df1c7b2
+    clientSecret: 34v5q1c56ub
 ```
 JSON
 ```json5
 {
+  "operator": {
+    "name": "YourRedditUsername"
+  },
   "bots": [
     {
       "credentials": {
         "clientId": "f4b4df1c7b2",
-        "clientSecret": "34v5q1c56ub",
-        "refreshToken": "34_f1w1v4",
-        "accessToken": "p75_1c467b2"
+        "clientSecret": "34v5q1c56ub"
       }
     }
-  ]
+  ],
+  "web": {
+    "credentials": {
+      "clientId": "f4b4df1c7b2",
+      "clientSecret": "34v5q1c56ub"
+    }
+  }
 }
 ```
 
@@ -153,10 +168,9 @@ Using **ENV** (`.env`)
 <details>
 
 ```
+OPERATOR=YourRedditUsername
 CLIENT_ID=f4b4df1c7b2
 CLIENT_SECRET=34v5q1c56ub
-REFRESH_TOKEN=34_f1w1v4
-ACCESS_TOKEN=p75_1c467b2
 ```
 
 </details>

docs/webInterface.md

@@ -0,0 +1,30 @@
+## Editing/Updating Your Config
+
+* Open the editor for your subreddit
+    * In the web dashboard \-> r/YourSubreddit \-> Config -> **View** [(here)](/docs/screenshots/config/config.jpg)
+* Follow the directions on the [link at the top of the window](/docs/screenshots/config/save.png) to enable config editing using your moderator account
+    * After enabling editing just click "save" at any time to save your config
+    * After you have added/edited your config the bot will detect changes within 5 minutes or you can manually trigger it by clicking **Update**
+
+## General Config (Editor) Tips
+
+* The editor will automatically validate your [syntax (formatting)](/docs/screenshots/config/syntax.png) and [config correctness](/docs/screenshots/config/correctness.png) (property names, required properties, etc.)
+    * These show up as squiggly lines like in Microsoft Word and as a [list at the bottom of the editor](/docs/screenshots/config/errors.png)
+* In your config all **Checks** and **Actions** have two properties that control how they behave:
+    * [**Enable**](/docs/screenshots/config/enable.png) (defaults to `enable: true`) -- Determines if the check or action is run, at all
+    * **Dryrun** (defaults to `dryRun: false`) -- When `true` the check or action will run but any **Actions** that may be triggered will "pretend" to execute but not actually talk to the Reddit API.
+      * Use `dryRun` to test your config without the bot making any changes on reddit
+    * When starting out with a new config it is recommended running the bot with remove/ban actions **disabled**
+        * Use `report` actions to get reports in your modqueue from the bot that describe what it detected and what it would do about it
+        * Once the bot is behaving as desired (no false positives or weird behavior) destructive actions can be enabled or turned off of dryrun
+
+## Web Dashboard Tips
+
+* Use the [**Overview** section](/docs/screenshots/botOperations.png) to control the bot at a high-level
+* You can **manually run** the bot on any activity (comment/submission) by pasting its permalink into the [input field below the Overview section](/docs/screenshots/runInput.png) and hitting one of the **run buttons**
+    * **Dry run** will make the bot run on the activity but it will only **pretend** to run actions, if triggered. This is super useful for testing your config without consequences
+    * **Run** will do everything
+* All of the bot's activity is shown in real-time in the [log section](/docs/screenshots/logs.png)
+    * This will output the results of all run checks/rules and any actions that run
+* You can view summaries of all activities that triggered a check (had actions run) by clicking on [Actioned Events](/docs/screenshots/actionsEvents.png)
+    * This includes activities run with dry run

package-lock.json

@@ -34,6 +34,7 @@
         "express-socket.io-session": "^1.3.5",
         "fast-deep-equal": "^3.1.3",
         "fuse.js": "^6.4.6",
+        "globrex": "^0.1.2",
         "got": "^11.8.2",
         "he": "^1.2.0",
         "http-proxy": "^1.18.1",
@@ -80,6 +81,7 @@
         "@types/express": "^4.17.13",
         "@types/express-session": "^1.17.4",
         "@types/express-socket.io-session": "^1.3.6",
+        "@types/globrex": "^0.1.1",
         "@types/he": "^1.1.1",
         "@types/http-proxy": "^1.17.7",
         "@types/js-yaml": "^4.0.1",
@@ -360,6 +362,15 @@
         "@types/redis": "^2.8.0"
       }
     },
+    "node_modules/@types/cache-manager-redis-store/node_modules/@types/redis": {
+      "version": "2.8.32",
+      "resolved": "https://registry.npmjs.org/@types/redis/-/redis-2.8.32.tgz",
+      "integrity": "sha512-7jkMKxcGq9p242exlbsVzuJb57KqHRhNl4dHoQu2Y5v9bCAbtIXXH0R3HleSQW4CTOqpHIYUW3t6tpUj4BVQ+w==",
+      "dev": true,
+      "dependencies": {
+        "@types/node": "*"
+      }
+    },
     "node_modules/@types/cacheable-request": {
       "version": "6.0.2",
       "resolved": "https://registry.npmjs.org/@types/cacheable-request/-/cacheable-request-6.0.2.tgz",
@@ -456,6 +467,12 @@
         "@types/socket.io": "2.1.13"
       }
     },
+    "node_modules/@types/globrex": {
+      "version": "0.1.1",
+      "resolved": "https://registry.npmjs.org/@types/globrex/-/globrex-0.1.1.tgz",
+      "integrity": "sha512-bce8X5Yb8l8ou2VDaEG8CYY1p6NynmswkaasO1pdAzFASKJ43sjf9MQdVH6VmKNG2bPEEmvI5onJJSH+1qOMOA==",
+      "dev": true
+    },
     "node_modules/@types/he": {
       "version": "1.1.2",
       "resolved": "https://registry.npmjs.org/@types/he/-/he-1.1.2.tgz",
@@ -607,15 +624,6 @@
       "integrity": "sha512-EEhsLsD6UsDM1yFhAvy0Cjr6VwmpMWqFBCb9w07wVugF7w9nfajxLuVmngTIpgS6svCnm6Vaw+MZhoDCKnOfsw==",
       "dev": true
     },
-    "node_modules/@types/redis": {
-      "version": "2.8.32",
-      "resolved": "https://registry.npmjs.org/@types/redis/-/redis-2.8.32.tgz",
-      "integrity": "sha512-7jkMKxcGq9p242exlbsVzuJb57KqHRhNl4dHoQu2Y5v9bCAbtIXXH0R3HleSQW4CTOqpHIYUW3t6tpUj4BVQ+w==",
-      "dev": true,
-      "dependencies": {
-        "@types/node": "*"
-      }
-    },
     "node_modules/@types/responselike": {
       "version": "1.0.0",
       "resolved": "https://registry.npmjs.org/@types/responselike/-/responselike-1.0.0.tgz",
@@ -2147,6 +2155,11 @@
         "url": "https://github.com/sponsors/isaacs"
       }
     },
+    "node_modules/globrex": {
+      "version": "0.1.2",
+      "resolved": "https://registry.npmjs.org/globrex/-/globrex-0.1.2.tgz",
+      "integrity": "sha512-uHJgbwAMwNFf5mLst7IWLNg14x1CkeqglJb/K3doi4dw6q2IvAAmM/Y81kevy83wP+Sst+nutFTYOGg3d1lsxg=="
+    },
     "node_modules/google-auth-library": {
       "version": "7.11.0",
       "resolved": "https://registry.npmjs.org/google-auth-library/-/google-auth-library-7.11.0.tgz",
@@ -4931,6 +4944,17 @@
       "requires": {
         "@types/cache-manager": "*",
         "@types/redis": "^2.8.0"
+      },
+      "dependencies": {
+        "@types/redis": {
+          "version": "2.8.32",
+          "resolved": "https://registry.npmjs.org/@types/redis/-/redis-2.8.32.tgz",
+          "integrity": "sha512-7jkMKxcGq9p242exlbsVzuJb57KqHRhNl4dHoQu2Y5v9bCAbtIXXH0R3HleSQW4CTOqpHIYUW3t6tpUj4BVQ+w==",
+          "dev": true,
+          "requires": {
+            "@types/node": "*"
+          }
+        }
       }
     },
     "@types/cacheable-request": {
@@ -5029,6 +5053,12 @@
         "@types/socket.io": "2.1.13"
       }
     },
+    "@types/globrex": {
+      "version": "0.1.1",
+      "resolved": "https://registry.npmjs.org/@types/globrex/-/globrex-0.1.1.tgz",
+      "integrity": "sha512-bce8X5Yb8l8ou2VDaEG8CYY1p6NynmswkaasO1pdAzFASKJ43sjf9MQdVH6VmKNG2bPEEmvI5onJJSH+1qOMOA==",
+      "dev": true
+    },
     "@types/he": {
       "version": "1.1.2",
       "resolved": "https://registry.npmjs.org/@types/he/-/he-1.1.2.tgz",
@@ -5180,15 +5210,6 @@
       "integrity": "sha512-EEhsLsD6UsDM1yFhAvy0Cjr6VwmpMWqFBCb9w07wVugF7w9nfajxLuVmngTIpgS6svCnm6Vaw+MZhoDCKnOfsw==",
       "dev": true
     },
-    "@types/redis": {
-      "version": "2.8.32",
-      "resolved": "https://registry.npmjs.org/@types/redis/-/redis-2.8.32.tgz",
-      "integrity": "sha512-7jkMKxcGq9p242exlbsVzuJb57KqHRhNl4dHoQu2Y5v9bCAbtIXXH0R3HleSQW4CTOqpHIYUW3t6tpUj4BVQ+w==",
-      "dev": true,
-      "requires": {
-        "@types/node": "*"
-      }
-    },
     "@types/responselike": {
       "version": "1.0.0",
       "resolved": "https://registry.npmjs.org/@types/responselike/-/responselike-1.0.0.tgz",
@@ -6401,6 +6422,11 @@
         "path-is-absolute": "^1.0.0"
       }
     },
+    "globrex": {
+      "version": "0.1.2",
+      "resolved": "https://registry.npmjs.org/globrex/-/globrex-0.1.2.tgz",
+      "integrity": "sha512-uHJgbwAMwNFf5mLst7IWLNg14x1CkeqglJb/K3doi4dw6q2IvAAmM/Y81kevy83wP+Sst+nutFTYOGg3d1lsxg=="
+    },
     "google-auth-library": {
       "version": "7.11.0",
       "resolved": "https://registry.npmjs.org/google-auth-library/-/google-auth-library-7.11.0.tgz",

package.json

@@ -49,6 +49,7 @@
     "express-socket.io-session": "^1.3.5",
     "fast-deep-equal": "^3.1.3",
     "fuse.js": "^6.4.6",
+    "globrex": "^0.1.2",
     "got": "^11.8.2",
     "he": "^1.2.0",
     "http-proxy": "^1.18.1",
@@ -95,6 +96,7 @@
     "@types/express": "^4.17.13",
     "@types/express-session": "^1.17.4",
     "@types/express-socket.io-session": "^1.3.6",
+    "@types/globrex": "^0.1.1",
     "@types/he": "^1.1.1",
     "@types/http-proxy": "^1.17.7",
     "@types/js-yaml": "^4.0.1",

src/Action/ApproveAction.ts

@@ -36,7 +36,7 @@ export class ApproveAction extends Action {
             }
 
             // @ts-ignore
-            if (item.approved) {
+            if (targetItem.approved) {
                 const msg = `${target === 'self' ? 'Item' : 'Comment\'s parent Submission'} is already approved`;
                 this.logger.warn(msg);
                 return {
@@ -54,6 +54,16 @@ export class ApproveAction extends Action {
                 }
                 // @ts-ignore
                 touchedEntities.push(await targetItem.approve());
+
+                if(target === 'self') {
+                    // @ts-ignore
+                    item.approved = true;
+                    await this.resources.resetCacheForItem(item);
+                } else if(await this.resources.hasActivity(targetItem)) {
+                    // @ts-ignore
+                    targetItem.approved = true;
+                    await this.resources.resetCacheForItem(targetItem);
+                }
             }
         }
 

src/Action/LockAction.ts

@@ -26,6 +26,9 @@ export class LockAction extends Action {
             //snoowrap typing issue, thinks comments can't be locked
             // @ts-ignore
             await item.lock();
+            // @ts-ignore
+            item.locked = true;
+            await this.resources.resetCacheForItem(item);
             touchedEntities.push(item);
         }
         return {

src/Action/RemoveAction.ts

@@ -4,6 +4,8 @@ import Snoowrap, {Comment, Submission} from "snoowrap";
 import {RuleResult} from "../Rule";
 import {activityIsRemoved} from "../Utils/SnoowrapUtils";
 import {ActionProcessResult} from "../Common/interfaces";
+import dayjs from "dayjs";
+import {isSubmission} from "../util";
 
 export class RemoveAction extends Action {
     spam: boolean;
@@ -26,11 +28,7 @@ export class RemoveAction extends Action {
         // issue with snoowrap typings, doesn't think prop exists on Submission
         // @ts-ignore
         if (activityIsRemoved(item)) {
-            return {
-                dryRun,
-                success: false,
-                result: 'Item is already removed',
-            }
+            this.logger.warn('It looks like this Item is already removed!');
         }
         if (this.spam) {
             this.logger.verbose('Marking as spam on removal');
@@ -38,6 +36,13 @@ export class RemoveAction extends Action {
         if (!dryRun) {
             // @ts-ignore
             await item.remove({spam: this.spam});
+            item.banned_at_utc = dayjs().unix();
+            item.spam = this.spam;
+            if(!isSubmission(item)) {
+                // @ts-ignore
+                item.removed = true;
+            }
+            await this.resources.resetCacheForItem(item);
             touchedEntities.push(item);
         }
 

src/Action/ReportAction.ts

@@ -35,6 +35,7 @@ export class ReportAction extends Action {
             await item.report({reason: truncatedContent});
             // due to reddit not updating this in response (maybe)?? just increment stale activity
             item.num_reports++;
+            await this.resources.resetCacheForItem(item);
             touchedEntities.push(item);
         }
 

src/Action/SubmissionAction/FlairAction.ts

@@ -33,16 +33,26 @@ export class FlairAction extends Action {
         if(this.css !== '') {
             flairParts.push(`CSS: ${this.css}`);
         }
+        if(this.flair_template_id !== '') {
+            flairParts.push(`Template: ${this.flair_template_id}`);
+        }
         const flairSummary = flairParts.length === 0 ? 'No flair (unflaired)' : flairParts.join(' | ');
         this.logger.verbose(flairSummary);
         if (item instanceof Submission) {
             if(!this.dryRun) {
                 if (this.flair_template_id) {
-                    await item.selectFlair({flair_template_id: this.flair_template_id}).then(() => {});
+                    // typings are wrong for this function, flair_template_id should be accepted
+                    // assignFlair uses /api/flair (mod endpoint)
+                    // selectFlair uses /api/selectflair (self endpoint for user to choose their own flair for submission)
+                    // @ts-ignore
+                    await item.assignFlair({flair_template_id: this.flair_template_id}).then(() => {});
+                    item.link_flair_template_id = this.flair_template_id;
                 } else {
                     await item.assignFlair({text: this.text, cssClass: this.css}).then(() => {});
+                    item.link_flair_css_class = this.css;
+                    item.link_flair_text = this.text;
                 }
-
+                await this.resources.resetCacheForItem(item);
             }
         } else {
             this.logger.warn('Cannot flair Comment');

src/Action/UserFlairAction.ts

@@ -50,6 +50,7 @@ export class UserFlairAction extends Action {
             flairTemplateId: this.flair_template_id,
             username: item.author.name,
           });
+          item.author_flair_template_id = this.flair_template_id
         } catch (err: any) {
           this.logger.error('Either the flair template ID is incorrect or you do not have permission to access it.');
           throw err;
@@ -57,6 +58,9 @@ export class UserFlairAction extends Action {
       } else if (this.text === undefined && this.css === undefined) {
         // @ts-ignore
         await item.subreddit.deleteUserFlair(item.author.name);
+        item.author_flair_css_class = null;
+        item.author_flair_text = null;
+        item.author_flair_template_id = null;
       } else {
         // @ts-ignore
         await item.author.assignFlair({
@@ -64,7 +68,11 @@ export class UserFlairAction extends Action {
           cssClass: this.css,
           text: this.text,
         });
+        item.author_flair_text = this.text ?? null;
+        item.author_flair_css_class = this.css ?? null;
       }
+      await this.resources.resetCacheForItem(item);
+      await this.resources.resetCacheForItem(item.author);
     }
 
     return {

src/Action/index.ts

@@ -7,6 +7,7 @@ import Author, {AuthorOptions} from "../Author/Author";
 import {mergeArr} from "../util";
 import LoggedError from "../Utils/LoggedError";
 import {ExtendedSnoowrap} from '../Utils/SnoowrapClients';
+import {ErrorWithCause} from "pony-cause";
 
 export abstract class Action {
     name?: string;
@@ -86,7 +87,8 @@ export abstract class Action {
             return {...actRes, ...results};
         } catch (err: any) {
             if(!(err instanceof LoggedError)) {
-                this.logger.error(`Encountered error while running`, err);
+                const actionError = new ErrorWithCause('Action did not run successfully due to unexpected error', {cause: err});
+                this.logger.error(actionError);
             }
             actRes.success = false;
             actRes.result = err.message;

src/Bot/index.ts

@@ -6,7 +6,7 @@ import EventEmitter from "events";
 import {
     BotInstanceConfig,
     FilterCriteriaDefaults,
-    Invokee,
+    Invokee, LogInfo,
     PAUSED,
     PollOn,
     RUNNING,
@@ -16,7 +16,7 @@ import {
 } from "../Common/interfaces";
 import {
     createRetryHandler,
-    formatNumber, getExceptionMessage,
+    formatNumber, getExceptionMessage, getUserAgent,
     mergeArr,
     parseBool,
     parseDuration, parseMatchMessage,
@@ -38,6 +38,7 @@ class Bot {
 
     client!: ExtendedSnoowrap;
     logger!: Logger;
+    logs: LogInfo[] = [];
     wikiLocation: string;
     dryRun?: true | undefined;
     running: boolean = false;
@@ -98,6 +99,7 @@ class Bot {
                 dryRun,
                 heartbeatInterval,
             },
+            userAgent,
             credentials: {
                 reddit: {
                     clientId,
@@ -151,6 +153,12 @@ class Bot {
             }
         }, mergeArr);
 
+        this.logger.stream().on('log', (log: LogInfo) => {
+            if(log.bot !== undefined && log.bot === this.getBotName() && log.subreddit === undefined) {
+                this.logs = [log, ...this.logs].slice(0, 301);
+            }
+        });
+
         let mw = maxWorkers;
         if(maxWorkers < 1) {
             this.logger.warn(`Max queue workers must be greater than or equal to 1 (Specified: ${maxWorkers})`);
@@ -166,7 +174,9 @@ class Bot {
         this.excludeSubreddits = exclude.map(parseSubredditName);
 
         let creds: any = {
-            get userAgent() { return getUserName() },
+            get userAgent() {
+                return getUserAgent(`web:contextBot:{VERSION}{FRAG}:BOT-${getBotName()}`, userAgent)
+            },
             clientId,
             clientSecret,
             refreshToken,

src/Common/defaults.ts

@@ -39,3 +39,5 @@ export const filterCriteriaDefault: FilterCriteriaDefaults = {
         ]
     }
 }
+
+export const VERSION = '0.10.12';

src/Common/interfaces.ts

@@ -949,6 +949,10 @@ export interface SubmissionState extends ActivityState {
     link_flair_text?: string | string[]
     link_flair_css_class?: string | string[]
     flairTemplate?: string | string[]
+    /**
+     * Is the submission a reddit-hosted image or video?
+     * */
+    isRedditMediaDomain?: boolean
 }
 
 // properties calculated/derived by CM -- not provided as plain values by reddit
@@ -1686,6 +1690,17 @@ export interface OperatorJsonConfig {
 
     bots?: BotInstanceJsonConfig[]
 
+    /**
+     * Added to the User-Agent information sent to reddit
+     *
+     * This string will be added BETWEEN version and your bot name.
+     *
+     * EX: `myBranch` => `web:contextMod:v1.0.0-myBranch:BOT-/u/MyBotUser`
+     *
+     * * ENV => `USER_AGENT`
+     * */
+    userAgent?: string
+
     /**
      * Settings for the web interface
      * */
@@ -1861,6 +1876,7 @@ export interface BotInstanceConfig extends BotInstanceJsonConfig {
         softLimit: number,
         hardLimit: number,
     }
+    userAgent?: string
 }
 
 export interface OperatorConfig extends OperatorJsonConfig {
@@ -1931,6 +1947,7 @@ export interface LogInfo {
     instance?: string
     labels?: string[]
     bot?: string
+    user?: string
 }
 
 export interface ActionResult extends ActionProcessResult {
@@ -2100,3 +2117,56 @@ export interface FilterResult<T> {
     join: JoinOperands
     passed: boolean
 }
+
+export interface TextTransformOptions {
+    /**
+     * A set of search-and-replace operations to perform on text values before performing a match. Transformations are performed in the order they are defined.
+     *
+     * * If `transformationsActivity` IS NOT defined then these transformations will be performed on BOTH the activity text (submission title or comment) AND the repost candidate text
+     * * If `transformationsActivity` IS defined then these transformations are only performed on repost candidate text
+     * */
+    transformations?: SearchAndReplaceRegExp[]
+
+    /**
+     * Specify a separate set of transformations for the activity text (submission title or comment)
+     *
+     * To perform no transformations when `transformations` is defined set this to an empty array (`[]`)
+     * */
+    transformationsActivity?: SearchAndReplaceRegExp[]
+}
+
+export interface TextMatchOptions {
+    /**
+     * The percentage, as a whole number, of a repost title/comment that must match the title/comment being checked in order to consider both a match
+     *
+     * Note: Setting to 0 will make every candidate considered a match -- useful if you want to match if the URL has been reposted anywhere
+     *
+     * Defaults to `85` (85%)
+     *
+     * @default 85
+     * @example [85]
+     * */
+    matchScore?: number
+
+    /**
+     * The minimum number of words in the activity being checked for which this rule will run on
+     *
+     * If the word count is below the minimum the rule fails
+     *
+     * Defaults to 2
+     *
+     * @default 2
+     * @example [2]
+     * */
+    minWordCount?: number
+
+    /**
+     * Should text matching be case sensitive?
+     *
+     * Defaults to false
+     *
+     * @default false
+     * @example [false]
+     **/
+    caseSensitive?: boolean
+}

src/ConfigBuilder.ts

@@ -666,9 +666,13 @@ export const parseOperatorConfigFromSources = async (args: any): Promise<[Operat
         defaultBotInstance.caching = configFromFile.caching;
     }
 
-    let botInstances = [];
+    let botInstances: BotInstanceJsonConfig[] = [];
     if (botInstancesFromFile.length === 0) {
-        botInstances = [defaultBotInstance];
+        // only add default bot if user supplied any credentials
+        // otherwise its most likely just default, empty settings
+        if(defaultBotInstance.credentials !== undefined) {
+            botInstances = [defaultBotInstance];
+        }
     } else {
         botInstances = botInstancesFromFile.map(x => merge.all([defaultBotInstance, x], {arrayMerge: overwriteMerge}));
     }
@@ -694,6 +698,7 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
             stream = {},
         } = {},
         caching: opCache,
+        userAgent,
         web: {
             port = 8085,
             maxLogs = 200,
@@ -772,6 +777,10 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
         ...fileRest
     } = file;
 
+     const defaultWebCredentials = {
+         redirectUri: 'http://localhost:8085/callback'
+     };
+
 
     const config: OperatorConfig = {
         mode,
@@ -796,6 +805,7 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
             }
         },
         caching: cache,
+        userAgent,
         web: {
             port,
             caching: {
@@ -811,7 +821,7 @@ export const buildOperatorConfigWithDefaults = (data: OperatorJsonConfig): Opera
             },
             maxLogs,
             clients: clients === undefined ? [{host: 'localhost:8095', secret: apiSecret}] : clients,
-            credentials: webCredentials as RequiredWebRedditCredentials,
+            credentials: {...defaultWebCredentials, ...webCredentials} as RequiredWebRedditCredentials,
             operators: operators || defaultOperators,
         },
         api: {
@@ -835,7 +845,8 @@ export const buildBotConfig = (data: BotInstanceJsonConfig, opConfig: OperatorCo
             actionedEventsMax: opActionedEventsMax,
             actionedEventsDefault: opActionedEventsDefault = 25,
             provider: defaultProvider,
-        } = {}
+        } = {},
+        userAgent,
     } = opConfig;
     const {
         name: botName,
@@ -976,6 +987,7 @@ export const buildBotConfig = (data: BotInstanceJsonConfig, opConfig: OperatorCo
         },
         credentials: botCreds,
         caching: botCache,
+        userAgent,
         polling: {
             shared: [...new Set(realShared)] as PollOn[],
             stagger,

src/Rule/RegexRule.ts

@@ -95,6 +95,15 @@ export interface RegexCriteria {
      * */
     totalMatchThreshold?: string,
 
+    /**
+     * When `true` the Activity being checked MUST pass the `matchThreshold` before the Rule considers any history
+     *
+     * For use with `activityMatchThreshold`/`totalMatchThreshold` -- useful to conserve API calls
+     *
+     * @default false
+     * */
+    mustMatchCurrent?: boolean
+
     window?: ActivityWindowType
 }
 
@@ -140,6 +149,7 @@ export class RegexRule extends Rule {
                 matchThreshold = '> 0',
                 activityMatchThreshold = '> 0',
                 totalMatchThreshold = null,
+                mustMatchCurrent = false,
                 window,
             } = criteria;
 
@@ -184,6 +194,8 @@ export class RegexRule extends Rule {
             if (singleMatched) {
                 activitiesMatchedCount++;
             }
+            const singleCriteriaPass = !mustMatchCurrent || (mustMatchCurrent && singleMatched);
+
             if (activityMatchComparison !== undefined) {
                 activityThresholdMet = !activityMatchComparison.isPercent && comparisonTextOp(activitiesMatchedCount, activityMatchComparison.operator, activityMatchComparison.value);
             }
@@ -192,7 +204,7 @@ export class RegexRule extends Rule {
             }
 
             let history: (Submission | Comment)[] = [];
-            if ((activityThresholdMet === false || totalThresholdMet === false) && window !== undefined) {
+            if ((activityThresholdMet === false || totalThresholdMet === false) && window !== undefined && singleCriteriaPass) {
                 // our checking activity didn't meet threshold requirements and criteria does define window
                 // leh go
 
@@ -263,7 +275,8 @@ export class RegexRule extends Rule {
                     matchThreshold,
                     activityMatchThreshold,
                     totalMatchThreshold,
-                    window: humanWindow
+                    window: humanWindow,
+                    mustMatchCurrent,
                 },
                 matches,
                 matchCount,

src/Rule/RepeatActivityRule.ts

@@ -1,17 +1,28 @@
 import {Rule, RuleJSONConfig, RuleOptions, RuleResult} from "./index";
 import {Comment} from "snoowrap";
 import {
-    activityWindowText, asSubmission,
-    comparisonTextOp, FAIL, getActivitySubredditName, isExternalUrlSubmission, isRedditMedia,
-    parseGenericValueComparison, parseSubredditName,
-    parseUsableLinkIdentifier as linkParser, PASS, subredditStateIsNameOnly, toStrongSubredditState
+    activityWindowText,
+    asSubmission,
+    comparisonTextOp,
+    FAIL,
+    getActivitySubredditName,
+    isExternalUrlSubmission,
+    isRedditMedia,
+    parseGenericValueComparison,
+    parseSubredditName,
+    parseUsableLinkIdentifier as linkParser,
+    PASS,
+    searchAndReplace,
+    stringSameness,
+    subredditStateIsNameOnly,
+    toStrongSubredditState
 } from "../util";
 import {
     ActivityWindow,
     ActivityWindowType,
-    ReferenceSubmission,
+    ReferenceSubmission, SearchAndReplaceRegExp,
     StrongSubredditState,
-    SubredditState
+    SubredditState, TextMatchOptions, TextTransformOptions
 } from "../Common/interfaces";
 import Submission from "snoowrap/dist/objects/Submission";
 import dayjs from "dayjs";
@@ -29,27 +40,6 @@ interface RepeatActivityReducer {
     allSets: RepeatActivityData[]
 }
 
-const getActivityIdentifier = (activity: (Submission | Comment), length = 200) => {
-    let identifier: string;
-    if (asSubmission(activity)) {
-        if (activity.is_self) {
-            identifier = `${activity.title}${activity.selftext.slice(0, length)}`;
-        } else if(isRedditMedia(activity)) {
-            identifier = activity.title;
-        } else {
-            identifier = parseUsableLinkIdentifier(activity.url) as string;
-        }
-    } else {
-        identifier = activity.body.slice(0, length);
-    }
-    return identifier;
-}
-
-const fuzzyOptions = {
-    includeScore: true,
-    distance: 15
-};
-
 export class RepeatActivityRule extends Rule {
     threshold: string;
     window: ActivityWindowType;
@@ -62,6 +52,9 @@ export class RepeatActivityRule extends Rule {
     activityFilterFunc: (x: Submission|Comment) => Promise<boolean> = async (x) => true;
     keepRemoved: boolean;
     minWordCount: number;
+    transformations: SearchAndReplaceRegExp[]
+    caseSensitive: boolean
+    matchScore: number
 
     constructor(options: RepeatActivityOptions) {
         super(options);
@@ -75,7 +68,13 @@ export class RepeatActivityRule extends Rule {
             include = [],
             exclude = [],
             keepRemoved = false,
+            transformations = [],
+            caseSensitive = true,
+            matchScore = 85,
         } = options;
+        this.matchScore = matchScore;
+        this.transformations = transformations;
+        this.caseSensitive = caseSensitive;
         this.minWordCount = minWordCount;
         this.keepRemoved = keepRemoved;
         this.threshold = threshold;
@@ -136,6 +135,37 @@ export class RepeatActivityRule extends Rule {
         }
     }
 
+    getActivityIdentifier(activity: (Submission | Comment), length = 200, transform = true) {
+        let identifier: string;
+        if (asSubmission(activity)) {
+            if (activity.is_self) {
+                identifier = `${activity.title}${activity.selftext.slice(0, length)}`;
+            } else if(isRedditMedia(activity)) {
+                identifier = activity.title;
+            } else {
+                identifier = parseUsableLinkIdentifier(activity.url) as string;
+            }
+        } else {
+            identifier = activity.body.slice(0, length);
+        }
+
+        if(!transform) {
+            return identifier;
+        }
+
+        // apply any transforms
+        if (this.transformations.length > 0) {
+            identifier = searchAndReplace(identifier, this.transformations);
+        }
+
+        // perform after transformations so as not to mess up regex's depending on case
+        if(!this.caseSensitive) {
+            identifier = identifier.toLowerCase();
+        }
+
+        return identifier;
+    }
+
     async process(item: Submission|Comment): Promise<[boolean, RuleResult]> {
         let referenceUrl;
         if(asSubmission(item) && this.useSubmissionAsReference) {
@@ -162,9 +192,10 @@ export class RepeatActivityRule extends Rule {
             const acc = await accProm;
             const {openSets = [], allSets = []} = acc;
 
-            let identifier = getActivityIdentifier(activity);
+            let identifier = this.getActivityIdentifier(activity);
+
             const isUrl = isExternalUrlSubmission(activity);
-            let fu = new Fuse([identifier], !isUrl ? fuzzyOptions : {...fuzzyOptions, distance: 5});
+            //let fu = new Fuse([identifier], !isUrl ? fuzzyOptions : {...fuzzyOptions, distance: 5});
             const validSub = await this.activityFilterFunc(activity);
             let minMet = identifier.length >= this.minWordCount;
 
@@ -174,12 +205,15 @@ export class RepeatActivityRule extends Rule {
             let currIdentifierInOpen = false;
             const bufferedActivities = this.gapAllowance === undefined || this.gapAllowance === 0 ? [] : activities.slice(Math.max(0, index - this.gapAllowance), Math.max(0, index));
             for (const o of openSets) {
-                const res = fu.search(o.identifier);
-                const match = res.length > 0;
-                if (match && validSub && minMet) {
+                const strMatchResults = stringSameness(o.identifier, identifier);
+                if (strMatchResults.highScoreWeighted >= this.matchScore && minMet) {
                     updatedOpenSets.push({...o, sets: [...o.sets, activity]});
                     currIdentifierInOpen = true;
-                } else if (bufferedActivities.some(x => fu.search(getActivityIdentifier(x)).length > 0) && validSub && minMet) {
+                } else if (bufferedActivities.some(x => {
+                    let buffIdentifier = this.getActivityIdentifier(x);
+                    const buffMatch = stringSameness(identifier, buffIdentifier);
+                    return buffMatch.highScoreWeighted >= this.matchScore;
+                }) && validSub && minMet) {
                     updatedOpenSets.push(o);
                 } else if(!currIdentifierInOpen && !isUrl) {
                     updatedAllSets.push(o);
@@ -193,15 +227,18 @@ export class RepeatActivityRule extends Rule {
                     // could be that a spammer is using different URLs for each submission but similar submission titles so search by title as well
                     const sub = activity as Submission;
                     identifier = sub.title;
-                    fu = new Fuse([identifier], !isUrl ? fuzzyOptions : {...fuzzyOptions, distance: 5});
+                    //fu = new Fuse([identifier], !isUrl ? fuzzyOptions : {...fuzzyOptions, distance: 5});
                     minMet = identifier.length >= this.minWordCount;
                     for (const o of openSets) {
-                        const res = fu.search(o.identifier);
-                        const match = res.length > 0;
-                        if (match && validSub && minMet) {
+                        const strMatchResults = stringSameness(o.identifier, identifier);
+                        if (strMatchResults.highScoreWeighted >= this.matchScore && minMet) {
                             updatedOpenSets.push({...o, sets: [...o.sets, activity]});
                             currIdentifierInOpen = true;
-                        } else if (bufferedActivities.some(x => fu.search(getActivityIdentifier(x)).length > 0) && validSub && minMet && !updatedOpenSets.includes(o)) {
+                        } else if (bufferedActivities.some(x => {
+                            let buffIdentifier = this.getActivityIdentifier(x);
+                            const buffMatch = stringSameness(identifier, buffIdentifier);
+                            return buffMatch.highScoreWeighted >= this.matchScore;
+                        }) && validSub && minMet && !updatedOpenSets.includes(o)) {
                             updatedOpenSets.push(o);
                         } else if(!updatedAllSets.includes(o)) {
                             updatedAllSets.push(o);
@@ -232,7 +269,7 @@ export class RepeatActivityRule extends Rule {
         let applicableGroupedActivities = identifierGroupedActivities;
         if (this.useSubmissionAsReference) {
             applicableGroupedActivities = new Map();
-            let identifier = getActivityIdentifier(item);
+            let identifier = this.getActivityIdentifier(item);
             let referenceSubmissions = identifierGroupedActivities.get(identifier);
             if(referenceSubmissions === undefined && isExternalUrlSubmission(item)) {
                 // if external url sub then try by title
@@ -240,7 +277,7 @@ export class RepeatActivityRule extends Rule {
                 referenceSubmissions = identifierGroupedActivities.get(identifier);
                 if(referenceSubmissions === undefined) {
                     // didn't get by title so go back to url since that's the default
-                    identifier = getActivityIdentifier(item);
+                    identifier = this.getActivityIdentifier(item);
                 }
             }
 
@@ -265,7 +302,7 @@ export class RepeatActivityRule extends Rule {
             };
             for (let set of value) {
                 const test = comparisonTextOp(set.length, operator, thresholdValue);
-                const md = set.map((x: (Comment | Submission)) => `[${asSubmission(x) ? x.title : getActivityIdentifier(x, 50)}](https://reddit.com${x.permalink}) in ${x.subreddit_name_prefixed} on ${dayjs(x.created_utc * 1000).utc().format()}`);
+                const md = set.map((x: (Comment | Submission)) => `[${asSubmission(x) ? x.title : this.getActivityIdentifier(x, 50)}](https://reddit.com${x.permalink}) in ${x.subreddit_name_prefixed} on ${dayjs(x.created_utc * 1000).utc().format()}`);
 
                 summaryData.sets.push(set);
                 summaryData.largestTrigger = Math.max(summaryData.largestTrigger, set.length);
@@ -325,7 +362,7 @@ interface SummaryData {
     triggeringSetsMarkdown: string[]
 }
 
-interface RepeatActivityConfig extends ActivityWindow, ReferenceSubmission {
+interface RepeatActivityConfig extends ActivityWindow, ReferenceSubmission, TextMatchOptions {
     /**
      * The number of repeat submissions that will trigger the rule
      * @default ">= 5"
@@ -383,18 +420,9 @@ interface RepeatActivityConfig extends ActivityWindow, ReferenceSubmission {
     keepRemoved?: boolean
 
     /**
-     * For activities that are text-based this is the minimum number of words required for the activity to be considered for a repeat
-     *
-     * EX if `minimumWordCount=5` and a comment is `what about you` then it is ignored because `3 is less than 5`
-     *
-     * **For self-text submissions** -- title + body text
-     *
-     * **For comments* -- body text
-     *
-     * @default 1
-     * @example [1]
+     * A set of search-and-replace operations to perform on text values before performing a match. Transformations are performed in the order they are defined.
      * */
-    minWordCount?: number,
+    transformations?: SearchAndReplaceRegExp[]
 }
 
 export interface RepeatActivityOptions extends RepeatActivityConfig, RuleOptions {

src/Rule/RepostRule.ts

@@ -18,7 +18,7 @@ import {
     RepostItem,
     RepostItemResult,
     SearchAndReplaceRegExp,
-    SearchFacetType,
+    SearchFacetType, TextMatchOptions, TextTransformOptions,
 } from "../Common/interfaces";
 import objectHash from "object-hash";
 import {getActivities, getAttributionIdentifier} from "../Utils/SnoowrapUtils";
@@ -30,59 +30,6 @@ import {rest} from "lodash";
 
 const parseYtIdentifier = parseUsableLinkIdentifier();
 
-export interface TextMatchOptions {
-    /**
-     * The percentage, as a whole number, of a repost title/comment that must match the title/comment being checked in order to consider both a match
-     *
-     * Note: Setting to 0 will make every candidate considered a match -- useful if you want to match if the URL has been reposted anywhere
-     *
-     * Defaults to `85` (85%)
-     *
-     * @default 85
-     * @example [85]
-     * */
-    matchScore?: number
-
-    /**
-     * The minimum number of words in the activity being checked for which this rule will run on
-     *
-     * If the word count is below the minimum the rule fails
-     *
-     * Defaults to 2
-     *
-     * @default 2
-     * @example [2]
-     * */
-    minWordCount?: number
-
-    /**
-     * Should text matching be case sensitive?
-     *
-     * Defaults to false
-     *
-     * @default false
-     * @example [false]
-     **/
-    caseSensitive?: boolean
-}
-
-export interface TextTransformOptions {
-    /**
-     * A set of search-and-replace operations to perform on text values before performing a match. Transformations are performed in the order they are defined.
-     *
-     * * If `transformationsActivity` IS NOT defined then these transformations will be performed on BOTH the activity text (submission title or comment) AND the repost candidate text
-     * * If `transformationsActivity` IS defined then these transformations are only performed on repost candidate text
-     * */
-    transformations?: SearchAndReplaceRegExp[]
-
-    /**
-     * Specify a separate set of transformations for the activity text (submission title or comment)
-     *
-     * To perform no transformations when `transformations` is defined set this to an empty array (`[]`)
-     * */
-    transformationsActivity?: SearchAndReplaceRegExp[]
-}
-
 export interface SearchFacetJSONConfig extends TextMatchOptions, TextTransformOptions, ActivityWindow {
     kind: SearchFacetType | SearchFacetType[]
 }

src/Schema/Action.json

@@ -290,6 +290,10 @@
                         }
                     ]
                 },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                 "is_self": {
                     "type": "boolean"
                 },

src/Schema/App.json

@@ -2413,6 +2413,11 @@
                     "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s+.*)*$",
                     "type": "string"
                 },
+                "mustMatchCurrent": {
+                    "default": false,
+                    "description": "When `true` the Activity being checked MUST pass the `matchThreshold` before the Rule considers any history\n\nFor use with `activityMatchThreshold`/`totalMatchThreshold` -- useful to conserve API calls",
+                    "type": "boolean"
+                },
                 "name": {
                     "description": "A descriptive name that will be used in logging and be available for templating",
                     "examples": [
@@ -2667,6 +2672,11 @@
                         }
                     ]
                 },
+                "caseSensitive": {
+                    "default": false,
+                    "description": "Should text matching be case sensitive?\n\nDefaults to false",
+                    "type": "boolean"
+                },
                 "exclude": {
                     "description": "If present, activities will be counted only if they are **NOT** found in this list of Subreddits\n\nEach value in the list can be either:\n\n * string (name of subreddit)\n * regular expression to run on the subreddit name\n * `SubredditState`\n\nEX `[\"mealtimevideos\",\"askscience\", \"/onlyfans*\\/i\", {\"over18\": true}]`",
                     "examples": [
@@ -2757,9 +2767,14 @@
                     ],
                     "type": "string"
                 },
+                "matchScore": {
+                    "default": 85,
+                    "description": "The percentage, as a whole number, of a repost title/comment that must match the title/comment being checked in order to consider both a match\n\nNote: Setting to 0 will make every candidate considered a match -- useful if you want to match if the URL has been reposted anywhere\n\nDefaults to `85` (85%)",
+                    "type": "number"
+                },
                 "minWordCount": {
-                    "default": 1,
-                    "description": "For activities that are text-based this is the minimum number of words required for the activity to be considered for a repeat\n\nEX if `minimumWordCount=5` and a comment is `what about you` then it is ignored because `3 is less than 5`\n\n**For self-text submissions** -- title + body text\n\n**For comments* -- body text",
+                    "default": 2,
+                    "description": "The minimum number of words in the activity being checked for which this rule will run on\n\nIf the word count is below the minimum the rule fails\n\nDefaults to 2",
                     "type": "number"
                 },
                 "name": {
@@ -2775,6 +2790,13 @@
                     "description": "The number of repeat submissions that will trigger the rule",
                     "type": "string"
                 },
+                "transformations": {
+                    "description": "A set of search-and-replace operations to perform on text values before performing a match. Transformations are performed in the order they are defined.",
+                    "items": {
+                        "$ref": "#/definitions/SearchAndReplaceRegExp"
+                    },
+                    "type": "array"
+                },
                 "useSubmissionAsReference": {
                     "default": true,
                     "description": "If activity is a Submission and is a link (not self-post) then only look at Submissions that contain this link, otherwise consider all activities.",
@@ -3491,6 +3513,10 @@
                         }
                     ]
                 },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                 "is_self": {
                     "type": "boolean"
                 },

src/Schema/OperatorConfig.json

@@ -633,6 +633,9 @@
                 },
                 "file": {
                     "allOf": [
+                        {
+                            "$ref": "#/definitions/Omit<DailyRotateFileTransportOptions,\"stream\"|\"dirname\"|\"options\"|\"handleRejections\"|\"format\"|\"handleExceptions\"|\"log\"|\"logv\"|\"close\">"
+                        },
                         {
                             "properties": {
                                 "dirname": {
@@ -658,9 +661,6 @@
                                 }
                             },
                             "type": "object"
-                        },
-                        {
-                            "$ref": "#/definitions/Omit<DailyRotateFileTransportOptions,\"stream\"|\"dirname\"|\"options\"|\"handleRejections\"|\"format\"|\"handleExceptions\"|\"log\"|\"logv\"|\"close\">"
                         }
                     ],
                     "description": "Options for Rotating File logging"
@@ -1131,6 +1131,10 @@
                         }
                     ]
                 },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                 "is_self": {
                     "type": "boolean"
                 },
@@ -1380,6 +1384,10 @@
             "$ref": "#/definitions/SnoowrapOptions",
             "description": "Set global snoowrap options as well as default snoowrap config for all bots that don't specify their own"
         },
+        "userAgent": {
+            "description": "Added to the User-Agent information sent to reddit\n\nThis string will be added BETWEEN version and your bot name.\n\nEX: `myBranch` => `web:contextMod:v1.0.0-myBranch:BOT-/u/MyBotUser`\n\n* ENV => `USER_AGENT`",
+            "type": "string"
+        },
         "web": {
             "description": "Settings for the web interface",
             "properties": {

src/Schema/Rule.json

@@ -1286,6 +1286,11 @@
                     "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s+.*)*$",
                     "type": "string"
                 },
+                "mustMatchCurrent": {
+                    "default": false,
+                    "description": "When `true` the Activity being checked MUST pass the `matchThreshold` before the Rule considers any history\n\nFor use with `activityMatchThreshold`/`totalMatchThreshold` -- useful to conserve API calls",
+                    "type": "boolean"
+                },
                 "name": {
                     "description": "A descriptive name that will be used in logging and be available for templating",
                     "examples": [
@@ -1460,6 +1465,11 @@
                         }
                     ]
                 },
+                "caseSensitive": {
+                    "default": false,
+                    "description": "Should text matching be case sensitive?\n\nDefaults to false",
+                    "type": "boolean"
+                },
                 "exclude": {
                     "description": "If present, activities will be counted only if they are **NOT** found in this list of Subreddits\n\nEach value in the list can be either:\n\n * string (name of subreddit)\n * regular expression to run on the subreddit name\n * `SubredditState`\n\nEX `[\"mealtimevideos\",\"askscience\", \"/onlyfans*\\/i\", {\"over18\": true}]`",
                     "examples": [
@@ -1550,9 +1560,14 @@
                     ],
                     "type": "string"
                 },
+                "matchScore": {
+                    "default": 85,
+                    "description": "The percentage, as a whole number, of a repost title/comment that must match the title/comment being checked in order to consider both a match\n\nNote: Setting to 0 will make every candidate considered a match -- useful if you want to match if the URL has been reposted anywhere\n\nDefaults to `85` (85%)",
+                    "type": "number"
+                },
                 "minWordCount": {
-                    "default": 1,
-                    "description": "For activities that are text-based this is the minimum number of words required for the activity to be considered for a repeat\n\nEX if `minimumWordCount=5` and a comment is `what about you` then it is ignored because `3 is less than 5`\n\n**For self-text submissions** -- title + body text\n\n**For comments* -- body text",
+                    "default": 2,
+                    "description": "The minimum number of words in the activity being checked for which this rule will run on\n\nIf the word count is below the minimum the rule fails\n\nDefaults to 2",
                     "type": "number"
                 },
                 "name": {
@@ -1568,6 +1583,13 @@
                     "description": "The number of repeat submissions that will trigger the rule",
                     "type": "string"
                 },
+                "transformations": {
+                    "description": "A set of search-and-replace operations to perform on text values before performing a match. Transformations are performed in the order they are defined.",
+                    "items": {
+                        "$ref": "#/definitions/SearchAndReplaceRegExp"
+                    },
+                    "type": "array"
+                },
                 "useSubmissionAsReference": {
                     "default": true,
                     "description": "If activity is a Submission and is a link (not self-post) then only look at Submissions that contain this link, otherwise consider all activities.",
@@ -1961,6 +1983,10 @@
                         }
                     ]
                 },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                 "is_self": {
                     "type": "boolean"
                 },

src/Schema/RuleSet.json

@@ -1260,6 +1260,11 @@
                     "pattern": "^\\s*(>|>=|<|<=)\\s*(\\d+)(\\s+.*)*$",
                     "type": "string"
                 },
+                "mustMatchCurrent": {
+                    "default": false,
+                    "description": "When `true` the Activity being checked MUST pass the `matchThreshold` before the Rule considers any history\n\nFor use with `activityMatchThreshold`/`totalMatchThreshold` -- useful to conserve API calls",
+                    "type": "boolean"
+                },
                 "name": {
                     "description": "A descriptive name that will be used in logging and be available for templating",
                     "examples": [
@@ -1434,6 +1439,11 @@
                         }
                     ]
                 },
+                "caseSensitive": {
+                    "default": false,
+                    "description": "Should text matching be case sensitive?\n\nDefaults to false",
+                    "type": "boolean"
+                },
                 "exclude": {
                     "description": "If present, activities will be counted only if they are **NOT** found in this list of Subreddits\n\nEach value in the list can be either:\n\n * string (name of subreddit)\n * regular expression to run on the subreddit name\n * `SubredditState`\n\nEX `[\"mealtimevideos\",\"askscience\", \"/onlyfans*\\/i\", {\"over18\": true}]`",
                     "examples": [
@@ -1524,9 +1534,14 @@
                     ],
                     "type": "string"
                 },
+                "matchScore": {
+                    "default": 85,
+                    "description": "The percentage, as a whole number, of a repost title/comment that must match the title/comment being checked in order to consider both a match\n\nNote: Setting to 0 will make every candidate considered a match -- useful if you want to match if the URL has been reposted anywhere\n\nDefaults to `85` (85%)",
+                    "type": "number"
+                },
                 "minWordCount": {
-                    "default": 1,
-                    "description": "For activities that are text-based this is the minimum number of words required for the activity to be considered for a repeat\n\nEX if `minimumWordCount=5` and a comment is `what about you` then it is ignored because `3 is less than 5`\n\n**For self-text submissions** -- title + body text\n\n**For comments* -- body text",
+                    "default": 2,
+                    "description": "The minimum number of words in the activity being checked for which this rule will run on\n\nIf the word count is below the minimum the rule fails\n\nDefaults to 2",
                     "type": "number"
                 },
                 "name": {
@@ -1542,6 +1557,13 @@
                     "description": "The number of repeat submissions that will trigger the rule",
                     "type": "string"
                 },
+                "transformations": {
+                    "description": "A set of search-and-replace operations to perform on text values before performing a match. Transformations are performed in the order they are defined.",
+                    "items": {
+                        "$ref": "#/definitions/SearchAndReplaceRegExp"
+                    },
+                    "type": "array"
+                },
                 "useSubmissionAsReference": {
                     "default": true,
                     "description": "If activity is a Submission and is a link (not self-post) then only look at Submissions that contain this link, otherwise consider all activities.",
@@ -1935,6 +1957,10 @@
                         }
                     ]
                 },
+                "isRedditMediaDomain": {
+                    "description": "Is the submission a reddit-hosted image or video?",
+                    "type": "boolean"
+                },
                 "is_self": {
                     "type": "boolean"
                 },

src/Subreddit/Manager.ts

@@ -24,7 +24,7 @@ import {
     ActionedEvent,
     ActionResult,
     DEFAULT_POLLING_INTERVAL,
-    DEFAULT_POLLING_LIMIT, FilterCriteriaDefaults, Invokee,
+    DEFAULT_POLLING_LIMIT, FilterCriteriaDefaults, Invokee, LogInfo,
     ManagerOptions, ManagerStateChangeOption, ManagerStats, PAUSED,
     PollingOptionsStrong, PollOn, RUNNING, RunState, STOPPED, SYSTEM, USER
 } from "../Common/interfaces";
@@ -87,6 +87,7 @@ export class Manager extends EventEmitter {
     subreddit: Subreddit;
     client: ExtendedSnoowrap;
     logger: Logger;
+    logs: LogInfo[] = [];
     botName: string;
     pollOptions: PollingOptionsStrong[] = [];
     submissionChecks!: SubmissionCheck[];
@@ -211,6 +212,11 @@ export class Manager extends EventEmitter {
                 return getDisplay()
             }
         }, mergeArr);
+        this.logger.stream().on('log', (log: LogInfo) => {
+            if(log.subreddit !== undefined && log.subreddit === this.getDisplay()) {
+                this.logs = [log, ...this.logs].slice(0, 301);
+            }
+        });
         this.globalDryRun = dryRun;
         this.wikiLocation = wikiLocation;
         this.filterCriteriaDefaults = filterCriteriaDefaults;
@@ -270,15 +276,21 @@ export class Manager extends EventEmitter {
         })(this), 10000);
     }
 
-    protected async getModPermissions(): Promise<string[]> {
+    public async getModPermissions(): Promise<string[]> {
         if(this.modPermissions !== undefined) {
             return this.modPermissions as string[];
         }
         this.logger.debug('Retrieving mod permissions for bot');
-        const userInfo = parseRedditEntity(this.botName, 'user');
-        const mods = this.subreddit.getModerators({name: userInfo.name});
-        // @ts-ignore
-        this.modPermissions = mods[0].mod_permissions;
+        try {
+            const userInfo = parseRedditEntity(this.botName, 'user');
+            const mods = this.subreddit.getModerators({name: userInfo.name});
+            // @ts-ignore
+            this.modPermissions = mods[0].mod_permissions;
+        } catch (e) {
+            const err = new ErrorWithCause('Unable to retrieve moderator permissions', {cause: e});
+            this.logger.error(err);
+            return [];
+        }
         return this.modPermissions as string[];
     }
 
@@ -735,7 +747,7 @@ export class Manager extends EventEmitter {
                     actionsRun = runActions.length;
 
                     if(check.notifyOnTrigger) {
-                        const ar = runActions.map(x => x.name).join(', ');
+                        const ar = runActions.filter(x => x.success).map(x => x.name).join(', ');
                         this.notificationManager.handle('eventActioned', 'Check Triggered', `Check "${check.name}" was triggered on Event: \n\n ${ePeek} \n\n with the following actions run: ${ar}`);
                     }
                     break;

src/Subreddit/SubredditResources.ts

@@ -12,6 +12,7 @@ import winston, {Logger} from "winston";
 import as from 'async';
 import fetch from 'node-fetch';
 import {
+    asActivity,
     asSubmission,
     buildCacheOptionsFromProvider,
     buildCachePrefix,
@@ -19,18 +20,18 @@ import {
     compareDurationValue,
     comparisonTextOp,
     createCacheManager,
-    createHistoricalStatsDisplay, FAIL,
+    createHistoricalStatsDisplay, escapeRegex, FAIL,
     fetchExternalUrl, filterCriteriaSummary,
     formatNumber,
     getActivityAuthorName,
     getActivitySubredditName,
-    isStrongSubredditState, isSubmission,
+    isStrongSubredditState, isSubmission, isUser,
     mergeArr,
     parseDurationComparison,
     parseExternalUrl,
     parseGenericValueComparison,
-    parseRedditEntity,
-    parseWikiContext, PASS,
+    parseRedditEntity, parseStringToRegex,
+    parseWikiContext, PASS, redisScanIterator,
     shouldCacheSubredditStateCriteriaResult,
     subredditStateIsNameOnly,
     toStrongSubredditState
@@ -69,6 +70,7 @@ import {check} from "tcp-port-used";
 import {ExtendedSnoowrap} from "../Utils/SnoowrapClients";
 import dayjs from "dayjs";
 import ImageData from "../Common/ImageData";
+import globrex from 'globrex';
 
 export const DEFAULT_FOOTER = '\r\n*****\r\nThis action was performed by [a bot.]({{botLink}}) Mention a moderator or [send a modmail]({{modmailLink}}) if you any ideas, questions, or concerns about this action.';
 
@@ -205,24 +207,23 @@ export class SubredditResources {
          const at = await this.cache.wrap(`${this.name}-historical-allTime`, () => createHistoricalDefaults(), {ttl: 0}) as object;
          const rehydratedAt: any = {};
          for(const [k, v] of Object.entries(at)) {
-            if(Array.isArray(v)) {
+             const t = typeof v;
+             if(t === 'number') {
+                 // simple number stat like eventsCheckedTotal
+                 rehydratedAt[k] = v;
+             } else if(Array.isArray(v)) {
+                 // a map stat that we have data for is serialized as an array of KV pairs
                 rehydratedAt[k] = new Map(v);
-            } else {
-                rehydratedAt[k] = v;
-            }
+             } else if(v === null || v === undefined || (t === 'object' && Object.keys(v).length === 0)) {
+                 // a map stat that was not serialized (for some reason) or serialized without any data
+                 rehydratedAt[k] = new Map();
+             } else {
+                 // ???? shouldn't get here
+                 this.logger.warn(`Did not recognize rehydrated historical stat "${k}" of type ${t}`);
+                 rehydratedAt[k] = v;
+             }
          }
          this.stats.historical.allTime = rehydratedAt as HistoricalStats;
-
-        // const lr = await this.cache.wrap(`${this.name}-historical-lastReload`, () => createHistoricalDefaults(), {ttl: 0}) as object;
-        // const rehydratedLr: any = {};
-        // for(const [k, v] of Object.entries(lr)) {
-        //     if(Array.isArray(v)) {
-        //         rehydratedLr[k] = new Map(v);
-        //     } else {
-        //         rehydratedLr[k] = v;
-        //     }
-        // }
-        // this.stats.historical.lastReload = rehydratedLr;
     }
 
     updateHistoricalStats(data: HistoricalStatUpdateData) {
@@ -298,6 +299,88 @@ export class SubredditResources {
         return 0;
     }
 
+    async interactWithCacheByKeyPattern(pattern: string | RegExp, action: 'get' | 'delete') {
+        let patternIsReg = pattern instanceof RegExp;
+        let regPattern: RegExp;
+        let globPattern = pattern;
+
+        const cacheDict: Record<string, any> = {};
+
+        if (typeof pattern === 'string') {
+            const possibleRegPattern = parseStringToRegex(pattern, 'ig');
+            if (possibleRegPattern !== undefined) {
+                regPattern = possibleRegPattern;
+                patternIsReg = true;
+            } else {
+                if (this.prefix !== undefined && !pattern.includes(this.prefix)) {
+                    // need to add wildcard to beginning of pattern so that the regex will still match a key with a prefix
+                    globPattern = `${this.prefix}${pattern}`;
+                }
+                // @ts-ignore
+                const result = globrex(globPattern, {flags: 'i'});
+                regPattern = result.regex;
+            }
+        } else {
+            regPattern = pattern;
+        }
+
+        if (this.cacheType === 'redis') {
+            // @ts-ignore
+            const redisClient = this.cache.store.getClient();
+            if (patternIsReg) {
+                // scan all and test key by regex
+                for await (const key of redisClient.scanIterator()) {
+                    if (regPattern.test(key) && (this.prefix === undefined || key.includes(this.prefix))) {
+                        if (action === 'delete') {
+                            await redisClient.del(key)
+                        } else {
+                            cacheDict[key] = await redisClient.get(key);
+                        }
+                    }
+                }
+            } else {
+                // not a regex means we can use glob pattern (more efficient!)
+                for await (const key of redisScanIterator(redisClient, { MATCH: globPattern })) {
+                    if (action === 'delete') {
+                        await redisClient.del(key)
+                    } else {
+                        cacheDict[key] = await redisClient.get(key);
+                    }
+                }
+            }
+        } else if (this.cache.store.keys !== undefined) {
+            for (const key of await this.cache.store.keys()) {
+                if (regPattern.test(key) && (this.prefix === undefined || key.includes(this.prefix))) {
+                    if (action === 'delete') {
+                        await this.cache.del(key)
+                    } else {
+                        cacheDict[key] = await this.cache.get(key);
+                    }
+                }
+            }
+        }
+        return cacheDict;
+    }
+
+    async deleteCacheByKeyPattern(pattern: string | RegExp) {
+        return await this.interactWithCacheByKeyPattern(pattern, 'delete');
+    }
+
+    async getCacheByKeyPattern(pattern: string | RegExp) {
+        return await this.interactWithCacheByKeyPattern(pattern, 'get');
+    }
+
+    async resetCacheForItem(item: Comment | Submission | RedditUser) {
+        if (asActivity(item)) {
+            if (this.filterCriteriaTTL !== false) {
+                await this.deleteCacheByKeyPattern(`itemCrit-${item.name}*`);
+            }
+            await this.setActivity(item, false);
+        } else if (isUser(item) && this.filterCriteriaTTL !== false) {
+            await this.deleteCacheByKeyPattern(`authorCrit-*-${getActivityAuthorName(item)}*`);
+        }
+    }
+
     async getStats() {
         const totals = Object.values(this.stats.cache).reduce((acc, curr) => ({
             miss: acc.miss + curr.miss,
@@ -379,11 +462,8 @@ export class SubredditResources {
                     this.logger.debug(`Cache Hit: Submission ${item.name}`);
                     return cachedSubmission;
                 }
-                // @ts-ignore
-                const submission = await item.fetch();
                 this.stats.cache.submission.miss++;
-                await this.cache.set(hash, submission, {ttl: this.submissionTTL});
-                return submission;
+                return await this.setActivity(item);
             } else if (this.commentTTL !== false) {
                 hash = `comm-${item.name}`;
                 await this.stats.cache.comment.identifierRequestCount.set(hash, (await this.stats.cache.comment.identifierRequestCount.wrap(hash, () => 0) as number) + 1);
@@ -394,11 +474,8 @@ export class SubredditResources {
                     this.logger.debug(`Cache Hit: Comment ${item.name}`);
                     return cachedComment;
                 }
-                // @ts-ignore
-                const comment = await item.fetch();
                 this.stats.cache.comment.miss++;
-                await this.cache.set(hash, comment, {ttl: this.commentTTL});
-                return comment;
+                return this.setActivity(item);
             } else {
                 // @ts-ignore
                 return await item.fetch();
@@ -409,6 +486,37 @@ export class SubredditResources {
         }
     }
 
+    // @ts-ignore
+    public async setActivity(item: Submission | Comment, tryToFetch = true)
+    {
+        let hash = '';
+        if(this.submissionTTL !== false && isSubmission(item)) {
+            hash = `sub-${item.name}`;
+            if(tryToFetch && item instanceof Submission) {
+                // @ts-ignore
+                const itemToCache = await item.fetch();
+                await this.cache.set(hash, itemToCache, {ttl: this.submissionTTL});
+                return itemToCache;
+            } else {
+                // @ts-ignore
+                await this.cache.set(hash, item, {ttl: this.submissionTTL});
+                return item;
+            }
+        } else if(this.commentTTL !== false){
+            hash = `comm-${item.name}`;
+            if(tryToFetch && item instanceof Comment) {
+                // @ts-ignore
+                const itemToCache = await item.fetch();
+                await this.cache.set(hash, itemToCache, {ttl: this.commentTTL});
+                return itemToCache;
+            } else {
+                // @ts-ignore
+                await this.cache.set(hash, item, {ttl: this.commentTTL});
+                return item;
+            }
+        }
+    }
+
     async hasActivity(item: Submission | Comment) {
         const hash = asSubmission(item) ? `sub-${item.name}` : `comm-${item.name}`;
         const res = await this.cache.get(hash);
@@ -978,6 +1086,20 @@ export class SubredditResources {
                                     return false
                                 }
                                 break;
+                            case 'isRedditMediaDomain':
+                                if((item instanceof Comment)) {
+                                    log.warn('`isRedditMediaDomain` is not allowed in `itemIs` criteria when the main Activity is a Comment');
+                                    continue;
+                                }
+                                // @ts-ignore
+                                const isRedditDomain = crit[k] as boolean;
+                                // @ts-ignore
+                                if (item.is_reddit_media_domain !== isRedditDomain) {
+                                    // @ts-ignore
+                                    log.debug(`Failed: Expected => ${k}:${crit[k]} | Found => ${k}:${item.is_reddit_media_domain}`)
+                                    return false
+                                }
+                                break;
                             case 'approved':
                             case 'spam':
                                 if(!item.can_mod_post) {

src/Subreddit/UserNotes.ts

@@ -14,6 +14,7 @@ import Submission from "snoowrap/dist/objects/Submission";
 import {RichContent} from "../Common/interfaces";
 import {Cache} from 'cache-manager';
 import {isScopeError} from "../Utils/Errors";
+import {ErrorWithCause} from "pony-cause";
 
 interface RawUserNotesPayload {
     ver: number,
@@ -63,6 +64,7 @@ export class UserNotes {
     identifier: string;
     cache: Cache
     cacheCB: Function;
+    mod?: RedditUser;
 
     users: Map<string, UserNote[]> = new Map();
 
@@ -110,14 +112,22 @@ export class UserNotes {
         }
     }
 
+    // @ts-ignore
+    async getMod() {
+        if(this.mod === undefined) {
+            // idgaf
+            // @ts-ignore
+            this.mod = await this.subreddit._r.getMe();
+        }
+        return this.mod as RedditUser;
+    }
+
     async addUserNote(item: (Submission|Comment), type: string | number, text: string = ''): Promise<UserNote>
     {
         const payload = await this.retrieveData();
         const userName = getActivityAuthorName(item.author);
 
-        // idgaf
-        // @ts-ignore
-        const mod = await this.subreddit._r.getMe();
+        const mod = await this.getMod();
         if(!payload.constants.users.includes(mod.name)) {
             this.logger.info(`Mod ${mod.name} does not exist in UserNote constants, adding them`);
             payload.constants.users.push(mod.name);
@@ -134,11 +144,11 @@ export class UserNotes {
         }
         payload.blob[userName].ns.push(newNote.toRaw(payload.constants));
 
+        const existingNotes = await this.getUserNotes(item.author);
         await this.saveData(payload);
         if(this.notesTTL > 0) {
-            const currNotes = this.users.get(userName) || [];
-            currNotes.push(newNote);
-            this.users.set(userName, currNotes);
+            existingNotes.push(newNote);
+            this.users.set(userName, existingNotes);
         }
         return newNote;
     }
@@ -150,7 +160,6 @@ export class UserNotes {
     }
 
     async retrieveData(): Promise<RawUserNotesPayload> {
-        let cacheMiss;
         if (this.notesTTL > 0) {
             const cachedPayload = await this.cache.get(this.identifier);
             if (cachedPayload !== undefined && cachedPayload !== null) {
@@ -158,19 +167,9 @@ export class UserNotes {
                 return cachedPayload as unknown as RawUserNotesPayload;
             }
             this.cacheCB(true);
-            cacheMiss = true;
         }
 
         try {
-            // DISABLED for now because I think its causing issues
-            // if(cacheMiss && this.debounceCB !== undefined) {
-            //     // timeout is still delayed. its our wiki data and we want it now! cm cacheworth 877 cache now
-            //     this.logger.debug(`Detected missed cache on usernotes retrieval while batch (${this.batchCount}) save is in progress, executing save immediately before retrieving new notes...`);
-            //     clearTimeout(this.saveDebounce);
-            //     await this.debounceCB();
-            //     this.debounceCB = undefined;
-            //     this.saveDebounce = undefined;
-            // }
             // @ts-ignore
             const wiki = this.client.getSubreddit(this.subreddit.display_name).getWikiPage('usernotes');
             const wikiContent = await wiki.content_md;
@@ -199,33 +198,6 @@ export class UserNotes {
         try {
             const wiki = this.client.getSubreddit(this.subreddit.display_name).getWikiPage('usernotes');
             if (this.notesTTL !== false) {
-                // DISABLED for now because if it fails throws an uncaught rejection
-                // and need to figured out how to handle this other than just logging (want to interrupt action flow too?)
-                //
-                // debounce usernote save by 5 seconds -- effectively batch usernote saves
-                //
-                // so that if we are processing a ton of checks that write user notes we aren't calling to save the wiki page on every call
-                // since we also have everything in cache (most likely...)
-                //
-                // TODO might want to increase timeout to 10 seconds
-                // if(this.saveDebounce !== undefined) {
-                //     clearTimeout(this.saveDebounce);
-                // }
-                // this.debounceCB = (async function () {
-                //     const p = wikiPayload;
-                //     // @ts-ignore
-                //     const self = this as UserNotes;
-                //     // @ts-ignore
-                //     self.wiki = await self.subreddit.getWikiPage('usernotes').edit(p);
-                //     self.logger.debug(`Batch saved ${self.batchCount} usernotes`);
-                //     self.debounceCB = undefined;
-                //     self.saveDebounce = undefined;
-                //     self.batchCount = 0;
-                // }).bind(this);
-                // this.saveDebounce = setTimeout(this.debounceCB,5000);
-                // this.batchCount++;
-                // this.logger.debug(`Saving Usernotes has been debounced for 5 seconds (${this.batchCount} batched)`)
-
                 // @ts-ignore
                 await wiki.edit(wikiPayload);
                 await this.cache.set(this.identifier, payload, {ttl: this.notesTTL});
@@ -237,15 +209,14 @@ export class UserNotes {
 
             return payload as RawUserNotesPayload;
         } catch (err: any) {
-            let msg = 'Could not edit usernotes.';
+            let msg = 'Could not edit usernotes!';
             // Make sure at least one moderator has used toolbox and usernotes before and that this account has editing permissions`;
             if(isScopeError(err)) {
                 msg = `${msg} The bot account did not have sufficient OAUTH scope to perform this action. You must re-authenticate the bot and ensure it has has 'wikiedit' permissions.`
             } else {
                 msg = `${msg} Make sure at least one moderator has used toolbox, created a usernote, and that this account has editing permissions for the wiki page.`;
             }
-            this.logger.error(msg, err);
-            throw new LoggedError(msg);
+            throw new ErrorWithCause(msg, {cause: err});
         }
     }
 }

src/Utils/SnoowrapUtils.ts

@@ -503,7 +503,7 @@ export const testAuthorCriteria = async (item: (Comment | Submission), authorOpt
                         break;
                     case 'isMod':
                         const mods: RedditUser[] = await item.subreddit.getModerators();
-                        const isModerator = mods.some(x => x.name === authorName);
+                        const isModerator = mods.some(x => x.name === authorName) || authorName.toLowerCase() === 'automoderator';
                         const modMatch = authorOpts.isMod === isModerator;
                         propResultsMap.isMod!.found = isModerator;
                         propResultsMap.isMod!.passed = !((include && !modMatch) || (!include && modMatch));

src/Web/Client/CMInstance.ts

@@ -0,0 +1,148 @@
+import {URL} from "url";
+import {Logger} from "winston";
+import {BotInstance, CMInstanceInterface, CMInstanceInterface as CMInterface} from "../interfaces";
+import dayjs from 'dayjs';
+import {BotConnection, LogInfo} from "../../Common/interfaces";
+import normalizeUrl from "normalize-url";
+import {HeartbeatResponse} from "../Common/interfaces";
+import jwt from "jsonwebtoken";
+import got from "got";
+import {ErrorWithCause} from "pony-cause";
+
+export class CMInstance implements CMInterface {
+    friendly?: string;
+    operators: string[] = [];
+    operatorDisplay: string = '';
+    url: URL;
+    normalUrl: string;
+    lastCheck?: number;
+    online: boolean = false;
+    subreddits: string[] = [];
+    bots: BotInstance[] = [];
+    error?: string | undefined;
+    host: string;
+    secret: string;
+
+    logger: Logger;
+    logs: LogInfo[] = [];
+
+    constructor(options: BotConnection, logger: Logger) {
+        const {
+            host,
+            secret
+        } = options;
+
+        this.host = host;
+        this.secret = secret;
+
+        const normalized = normalizeUrl(options.host);
+        this.normalUrl = normalized;
+        this.url = new URL(normalized);
+
+        const name = this.getName;
+
+        this.logger = logger.child({
+            get instance() {
+                return name();
+            }
+        });
+
+        this.logger.stream().on('log', (log: LogInfo) => {
+            if(log.instance !== undefined && log.instance === this.getName()) {
+                this.logs = [log, ...this.logs].slice(0, 301);
+            }
+        });
+    }
+
+    getData(): CMInterface {
+        return {
+            friendly: this.getName(),
+            operators: this.operators,
+            operatorDisplay: this.operatorDisplay,
+            url: this.url,
+            normalUrl: this.normalUrl,
+            lastCheck: this.lastCheck,
+            online: this.online,
+            subreddits: this.subreddits,
+            bots: this.bots,
+            error: this.error,
+            host: this.host,
+            secret: this.secret
+        }
+    }
+
+    getName = () => {
+        if (this.friendly !== undefined) {
+            return this.friendly
+        }
+        return this.url.host;
+    }
+
+    matchesHost = (val: string) => {
+        return normalizeUrl(val) == this.normalUrl;
+    }
+
+    updateFromHeartbeat = (resp: HeartbeatResponse, otherFriendlies: string[] = []) => {
+        this.operators = resp.operators ?? [];
+        this.operatorDisplay = resp.operatorDisplay ?? '';
+
+        const fr = resp.friendly;
+        if (fr !== undefined) {
+            if (otherFriendlies.includes(fr)) {
+                this.logger.warn(`Client returned a friendly name that is not unique (${fr}), will fallback to host as friendly (${this.url.host})`);
+            } else {
+                this.friendly = fr;
+            }
+        }
+
+        this.subreddits = resp.subreddits;
+        this.bots = resp.bots.map(x => ({...x, instance: this}));
+    }
+
+    checkHeartbeat = async (force = false, otherFriendlies: string[] = []) => {
+        let shouldCheck = force;
+        if (!shouldCheck) {
+            if (this.lastCheck === undefined) {
+                shouldCheck = true;
+            } else {
+                const lastCheck = dayjs().diff(dayjs.unix(this.lastCheck), 's');
+                if (!this.online) {
+                    if (lastCheck > 15) {
+                        shouldCheck = true;
+                    }
+                } else if (lastCheck > 60) {
+                    shouldCheck = true;
+                }
+            }
+        }
+        if (shouldCheck) {
+            this.logger.debug('Starting Heartbeat check');
+            this.lastCheck = dayjs().unix();
+            const machineToken = jwt.sign({
+                data: {
+                    machine: true,
+                },
+            }, this.secret, {
+                expiresIn: '1m'
+            });
+
+            try {
+                const resp = await got.get(`${this.normalUrl}/heartbeat`, {
+                    headers: {
+                        'Authorization': `Bearer ${machineToken}`,
+                    }
+                }).json() as CMInstanceInterface;
+
+                this.online = true;
+                this.updateFromHeartbeat(resp as HeartbeatResponse, otherFriendlies);
+                this.logger.verbose(`Heartbeat detected`);
+            } catch (err: any) {
+                this.online = false;
+                this.error = err.message;
+                const badHeartbeat = new ErrorWithCause('Heartbeat response was not ok', {cause: err});
+                this.logger.error(badHeartbeat);
+            }
+        }
+    }
+
+}

src/Web/Client/index.ts

@@ -9,12 +9,12 @@ import {Strategy as CustomStrategy} from 'passport-custom';
 import {OperatorConfig, BotConnection, LogInfo} from "../../Common/interfaces";
 import {
     buildCachePrefix,
-    createCacheManager, defaultFormat, filterLogBySubreddit,
-    formatLogLineToHtml,
+    createCacheManager, defaultFormat, filterLogBySubreddit, filterLogs,
+    formatLogLineToHtml, getUserAgent,
     intersect, isLogLineMinLevel,
     LogEntry, parseInstanceLogInfoName, parseInstanceLogName, parseRedditEntity,
     parseSubredditLogName, permissions,
-    randomId, sleep, triggeredIndicator
+    randomId, replaceApplicationIdentifier, sleep, triggeredIndicator
 } from "../../util";
 import {Cache} from "cache-manager";
 import session, {Session, SessionData} from "express-session";
@@ -39,7 +39,7 @@ import DelimiterStream from 'delimiter-stream';
 import {pipeline} from 'stream/promises';
 import {defaultBotStatus} from "../Common/defaults";
 import {arrayMiddle, booleanMiddle} from "../Common/middleware";
-import {BotInstance, CMInstance} from "../interfaces";
+import {BotInstance, CMInstanceInterface} from "../interfaces";
 import { URL } from "url";
 import {MESSAGE} from "triple-beam";
 import Autolinker from "autolinker";
@@ -50,6 +50,7 @@ import {BotStatusResponse} from "../Common/interfaces";
 import {TransformableInfo} from "logform";
 import {SimpleError} from "../../Utils/Errors";
 import {ErrorWithCause} from "pony-cause";
+import {CMInstance} from "./CMInstance";
 
 const emitter = new EventEmitter();
 
@@ -106,7 +107,7 @@ interface ConnectUserObj {
     [key: string]: ConnectedUserInfo
 }
 
-const createToken = (bot: CMInstance, user?: Express.User | any, ) => {
+const createToken = (bot: CMInstanceInterface, user?: Express.User | any, ) => {
     const payload = user !== undefined ? {...user, machine: false} : {machine: true};
     return jwt.sign({
         data: payload,
@@ -117,14 +118,13 @@ const createToken = (bot: CMInstance, user?: Express.User | any, ) => {
 
 const availableLevels = ['error', 'warn', 'info', 'verbose', 'debug'];
 
-const instanceLogMap: Map<string, LogEntry[]> = new Map();
-
 const webClient = async (options: OperatorConfig) => {
     const {
         operator: {
             name,
             display,
         },
+        userAgent: uaFragment,
         web: {
             port,
             caching,
@@ -149,22 +149,18 @@ const webClient = async (options: OperatorConfig) => {
         },
     } = options;
 
+    const userAgent = getUserAgent(`web:contextBot:{VERSION}{FRAG}:dashboard`, uaFragment);
+
+    app.use((req, res, next) => {
+        res.locals.applicationIdentifier = replaceApplicationIdentifier('{VERSION}{FRAG}', uaFragment);
+        next();
+    });
+
     const webOps = operators.map(x => x.toLowerCase());
 
     const logger = getLogger({defaultLabel: 'Web', ...options.logging}, 'Web');
 
     logger.stream().on('log', (log: LogInfo) => {
-        const logEntry: LogEntry = [dayjs(log.timestamp).unix(), log];
-        const {instance: instanceLogName} = log;
-        if (instanceLogName !== undefined) {
-            const subLogs = instanceLogMap.get(instanceLogName) || [];
-            subLogs.unshift(logEntry);
-            instanceLogMap.set(instanceLogName, subLogs.slice(0, 200 + 1));
-        } else {
-            const appLogs = instanceLogMap.get('web') || [];
-            appLogs.unshift(logEntry);
-            instanceLogMap.set('web', appLogs.slice(0, 200 + 1));
-        }
         emitter.emit('log', log[MESSAGE]);
     });
 
@@ -330,7 +326,7 @@ const webClient = async (options: OperatorConfig) => {
                 userName,
             };
             if(invite.instance !== undefined) {
-                const bot = cmInstances.find(x => x.friendly === invite.instance);
+                const bot = cmInstances.find(x => x.getName() === invite.instance);
                 if(bot !== undefined) {
                     const botPayload: any = {
                         overwrite: invite.overwrite === true,
@@ -430,7 +426,7 @@ const webClient = async (options: OperatorConfig) => {
             clientId,
             clientSecret,
             token: req.isAuthenticated() && req.user?.clientData?.webOperator ? token : undefined,
-            instances: cmInstances.filter(x => req.user?.isInstanceOperator(x)).map(x => x.friendly),
+            instances: cmInstances.filter(x => req.user?.isInstanceOperator(x)).map(x => x.getName()),
         });
     });
 
@@ -544,7 +540,7 @@ const webClient = async (options: OperatorConfig) => {
                 delimiter: '\r\n',
             });
 
-            const currInstance = cmInstances.find(x => x.friendly === sessionData.botId);
+            const currInstance = cmInstances.find(x => x.getName() === sessionData.botId);
             if(currInstance !== undefined) {
                 const ac = new AbortController();
                 const options = {
@@ -557,7 +553,8 @@ const webClient = async (options: OperatorConfig) => {
                     });
 
                     if(err !== undefined) {
-                        logger.warn(`Log streaming encountered an error, trying to reconnect (retries: ${retryCount}) -- ${err.code !== undefined ? `(${err.code}) ` : ''}${err.message}`, {instance: currInstance.friendly});
+                        // @ts-ignore
+                        currInstance.logger.warn(new ErrorWithCause(`Log streaming encountered an error, trying to reconnect (retries: ${retryCount})`, {cause: err}), {user: user.name});
                     }
                     const gotStream = got.stream.get(`${currInstance.normalUrl}/logs`, {
                         retry: {
@@ -578,7 +575,7 @@ const webClient = async (options: OperatorConfig) => {
 
                     if(err !== undefined) {
                         gotStream.once('data', () => {
-                            logger.info('Streaming resumed', {subreddit: currInstance.friendly});
+                            currInstance.logger.info('Streaming resumed', {instance: currInstance.getName(), user: user.name});
                         });
                     }
 
@@ -592,7 +589,8 @@ const webClient = async (options: OperatorConfig) => {
                     // ECONNRESET
                     s.catch((err) => {
                         if(err.code !== 'ABORT_ERR' && err.code !== 'ERR_STREAM_PREMATURE_CLOSE') {
-                            logger.error(`Unexpected error, or too many retries, occurred while streaming logs -- ${err.code !== undefined ? `(${err.code}) ` : ''}${err.message}`, {instance: currInstance.friendly});
+                            // @ts-ignore
+                            currInstance.logger.error(new ErrorWithCause('Unexpected error, or too many retries, occurred while streaming logs', {cause: err}), {user: user.name});
                         }
                     });
 
@@ -640,7 +638,7 @@ const webClient = async (options: OperatorConfig) => {
         delete req.session.authBotId;
 
         const msg = 'Bot does not exist or you do not have permission to access it';
-        const instance = cmInstances.find(x => x.friendly === req.query.instance);
+        const instance = cmInstances.find(x => x.getName() === req.query.instance);
         if (instance === undefined) {
             return res.status(404).render('error', {error: msg});
         }
@@ -653,15 +651,15 @@ const webClient = async (options: OperatorConfig) => {
             return res.status(404).render('error', {error: msg});
         }
         req.instance = instance;
-        req.session.botId = instance.friendly;
+        req.session.botId = instance.getName();
         if(req.user?.canAccessInstance(instance)) {
-            req.session.authBotId = instance.friendly;
+            req.session.authBotId = instance.getName();
         }
         return next();
     }
 
 
-    const botWithPermissions = async (req: express.Request, res: express.Response, next: Function) => {
+    const botWithPermissions = (required: boolean = false, setDefault: boolean = false) => async (req: express.Request, res: express.Response, next: Function) => {
 
         const instance = req.instance;
         if(instance === undefined) {
@@ -670,28 +668,42 @@ const webClient = async (options: OperatorConfig) => {
 
         const msg = 'Bot does not exist or you do not have permission to access it';
         const botVal = req.query.bot as string;
-        if(botVal === undefined) {
+        if(botVal === undefined && required) {
             return res.status(400).render('error', {error: `"bot" param must be defined`});
         }
 
-        const botInstance = instance.bots.find(x => x.botName === botVal);
-        if(botInstance === undefined) {
-            return res.status(404).render('error', {error: msg});
+        if(botVal !== undefined || setDefault) {
+
+            let botInstance;
+            if(botVal === undefined) {
+                // find a bot they can access
+                botInstance = instance.bots.find(x => req.user?.canAccessBot(x));
+                if(botInstance !== undefined) {
+                    req.query.bot = botInstance.botName;
+                }
+            } else {
+                botInstance = instance.bots.find(x => x.botName === botVal);
+            }
+
+            if(botInstance === undefined) {
+                return res.status(404).render('error', {error: msg});
+            }
+
+            if (!req.user?.clientData?.webOperator && !req.user?.canAccessBot(botInstance)) {
+                return res.status(404).render('error', {error: msg});
+            }
+
+            if (req.params.subreddit !== undefined && !req.user?.isInstanceOperator(instance) && !req.user?.subreddits.includes(req.params.subreddit)) {
+                return res.status(404).render('error', {error: msg});
+            }
+            req.bot = botInstance;
         }
 
-        if (!req.user?.clientData?.webOperator && !req.user?.canAccessBot(botInstance)) {
-            return res.status(404).render('error', {error: msg});
-        }
-
-        if (req.params.subreddit !== undefined && !req.user?.isInstanceOperator(instance) && !req.user?.subreddits.includes(req.params.subreddit)) {
-            return res.status(404).render('error', {error: msg});
-        }
-        req.bot = botInstance;
         next();
     }
 
     const createUserToken = async (req: express.Request, res: express.Response, next: Function) => {
-        req.token = createToken(req.instance as CMInstance, req.user);
+        req.token = createToken(req.instance as CMInstanceInterface, req.user);
         next();
     }
 
@@ -715,10 +727,10 @@ const webClient = async (options: OperatorConfig) => {
     // botUserRouter.use([ensureAuthenticated, defaultSession, botWithPermissions, createUserToken]);
     // app.use(botUserRouter);
 
-    app.useAsync('/api/', [ensureAuthenticated, defaultSession, instanceWithPermissions, botWithPermissions, createUserToken], (req: express.Request, res: express.Response) => {
+    app.useAsync('/api/', [ensureAuthenticated, defaultSession, instanceWithPermissions, botWithPermissions(true), createUserToken], (req: express.Request, res: express.Response) => {
         req.headers.Authorization = `Bearer ${req.token}`
 
-        const instance = req.instance as CMInstance;
+        const instance = req.instance as CMInstanceInterface;
         return proxy.web(req, res, {
             target: {
                 protocol: instance.url.protocol,
@@ -744,16 +756,25 @@ const webClient = async (options: OperatorConfig) => {
             });
 
             if(accessibleInstance === undefined) {
+                logger.warn(`User ${user.name} is not an operator and has no subreddits in common with any *running* bot instances. If you are sure they should have common subreddits then this client may not be able to access all defined CM servers or the bot may be offline.`, {user: user.name});
                 return res.render('noAccess');
             }
 
             return res.redirect(`/?instance=${accessibleInstance.friendly}`);
         }
-        const instance = cmInstances.find(x => x.friendly === req.query.instance);
+        const instance = cmInstances.find(x => x.getName() === req.query.instance);
         req.instance = instance;
         next();
     }
 
+    const defaultSubreddit = async (req: express.Request, res: express.Response, next: Function) => {
+        if(req.bot !== undefined && req.query.subreddit === undefined) {
+            const firstAccessibleSub = req.bot.subreddits.find(x => req.user?.isInstanceOperator(req.instance) || req.user?.subreddits.includes(x));
+            req.query.subreddit = firstAccessibleSub;
+        }
+        next();
+    }
+
     const initHeartbeat = async (req: express.Request, res: express.Response, next: Function) => {
         if(!init) {
             for(const c of clients) {
@@ -773,7 +794,7 @@ const webClient = async (options: OperatorConfig) => {
         next();
     }
 
-    app.getAsync('/', [initHeartbeat, redirectBotsNotAuthed, ensureAuthenticated, defaultSession, defaultInstance, instanceWithPermissions, createUserToken], async (req: express.Request, res: express.Response) => {
+    app.getAsync('/', [initHeartbeat, redirectBotsNotAuthed, ensureAuthenticated, defaultSession, defaultInstance, instanceWithPermissions, botWithPermissions(false, true), createUserToken], async (req: express.Request, res: express.Response) => {
 
         const user = req.user as Express.User;
         const instance = req.instance as CMInstance;
@@ -786,13 +807,13 @@ const webClient = async (options: OperatorConfig) => {
             const isBotOperator = req.user?.isInstanceOperator(curr);
             if(user?.clientData?.webOperator) {
                 // @ts-ignore
-                return acc.concat({...curr, canAccessLocation: true, isOperator: isBotOperator});
+                return acc.concat({...curr.getData(), canAccessLocation: true, isOperator: isBotOperator});
             }
             if(!isBotOperator && !req.user?.canAccessInstance(curr)) {
                 return acc;
             }
             // @ts-ignore
-            return acc.concat({...curr, canAccessLocation: isBotOperator, isOperator: isBotOperator, botId: curr.friendly});
+            return acc.concat({...curr.getData(), canAccessLocation: isBotOperator, isOperator: isBotOperator, botId: curr.getName()});
         },[]);
 
         let resp;
@@ -802,6 +823,8 @@ const webClient = async (options: OperatorConfig) => {
                     'Authorization': `Bearer ${req.token}`,
                 },
                 searchParams: {
+                    bot: req.query.bot as (string | undefined),
+                    subreddit: req.query.sub as (string | undefined) ?? 'all',
                     limit,
                     sort,
                     level,
@@ -810,14 +833,15 @@ const webClient = async (options: OperatorConfig) => {
             }).json() as any;
 
         } catch(err: any) {
-            logger.error(`Error occurred while retrieving bot information. Will update heartbeat -- ${err.message}`, {instance: instance.friendly});
-            refreshClient(clients.find(x => normalizeUrl(x.host) === instance.normalUrl) as BotConnection);
+            instance.logger.error(new ErrorWithCause(`Could not retrieve instance information. Will attempted to update heartbeat.`, {cause: err}));
+            refreshClient({host: instance.host, secret: instance.secret});
+            const isOp = req.user?.isInstanceOperator(instance);
             return res.render('offline', {
                 instances: shownInstances,
-                instanceId: (req.instance as CMInstance).friendly,
-                isOperator: req.user?.isInstanceOperator(instance),
+                instanceId: (req.instance as CMInstance).getName(),
+                isOperator: isOp,
                 // @ts-ignore
-                logs: filterLogBySubreddit(instanceLogMap, [instance.friendly], {limit, sort, level, allLogName: 'web', allLogsParser: parseInstanceLogInfoName }).get(instance.friendly),
+                logs: filterLogs((isOp ? instance.logs : instance.logs.filter(x => x.user === undefined || x.user.includes(req.user.name))), {limit, sort, level}),
                 logSettings: {
                     limitSelect: [10, 20, 50, 100, 200].map(x => `<option ${limit === x ? 'selected' : ''} class="capitalize ${limit === x ? 'font-bold' : ''}" data-value="${x}">${x}</option>`).join(' | '),
                     sortSelect: ['ascending', 'descending'].map(x => `<option ${sort === x ? 'selected' : ''} class="capitalize ${sort === x ? 'font-bold' : ''}" data-value="${x}">${x}</option>`).join(' '),
@@ -870,8 +894,8 @@ const webClient = async (options: OperatorConfig) => {
                 });
                 return {...x, subreddits: subredditsWithSimpleLogs};
             }),
-            botId: (req.instance as CMInstance).friendly,
-            instanceId: (req.instance as CMInstance).friendly,
+            botId: (req.instance as CMInstanceInterface).friendly,
+            instanceId: (req.instance as CMInstanceInterface).friendly,
             isOperator: isOp,
             system: isOp ? {
                 logs: resp.system.logs,
@@ -901,7 +925,7 @@ const webClient = async (options: OperatorConfig) => {
         });
     });
 
-    app.postAsync('/config', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions], async (req: express.Request, res: express.Response) => {
+    app.postAsync('/config', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions(true)], async (req: express.Request, res: express.Response) => {
         const {subreddit} = req.query as any;
         const {location, data, create = false} = req.body as any;
 
@@ -942,9 +966,9 @@ const webClient = async (options: OperatorConfig) => {
         return res.send();
     });
 
-    app.getAsync('/events', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions, createUserToken], async (req: express.Request, res: express.Response) => {
+    app.getAsync('/events', [ensureAuthenticatedApi, defaultSession, instanceWithPermissions, botWithPermissions(true), createUserToken], async (req: express.Request, res: express.Response) => {
         const {subreddit} = req.query as any;
-        const resp = await got.get(`${(req.instance as CMInstance).normalUrl}/events`, {
+        const resp = await got.get(`${(req.instance as CMInstanceInterface).normalUrl}/events`, {
             headers: {
                 'Authorization': `Bearer ${req.token}`,
             },
@@ -1055,6 +1079,9 @@ const webClient = async (options: OperatorConfig) => {
         const session = socket.handshake.session as (Session & Partial<SessionData> | undefined);
         // @ts-ignore
         const user = session !== undefined ? session?.passport?.user as Express.User : undefined;
+
+        let liveInterval: any = undefined;
+
         if (session !== undefined && user !== undefined) {
             clearSockStreams(socket.id);
             socket.join(session.id);
@@ -1069,8 +1096,44 @@ const webClient = async (options: OperatorConfig) => {
             emitter.on('log', webLogListener);
             socketListeners.set(socket.id, [...(socketListeners.get(socket.id) || []), webLogListener]);
 
+            socket.on('viewing', (data) => {
+                if(user !== undefined) {
+                    const {subreddit, bot: botVal} = data;
+                    const currBot = cmInstances.find(x => x.getName() === session.botId);
+                    if(currBot !== undefined) {
+
+                        if(liveInterval !== undefined) {
+                            clearInterval(liveInterval)
+                        }
+
+                        const liveEmit = async () => {
+                            try {
+                                const resp = await got.get(`${currBot.normalUrl}/liveStats`, {
+                                    headers: {
+                                        'Authorization': `Bearer ${createToken(currBot, user)}`,
+                                    },
+                                    searchParams: {
+                                        bot: botVal,
+                                        subreddit
+                                    }
+                                });
+                                const stats = JSON.parse(resp.body);
+                                io.to(session.id).emit('liveStats', stats);
+                            } catch (err: any) {
+                                currBot.logger.error(new ErrorWithCause('Could not retrieve live stats', {cause: err}));
+                            }
+                        }
+
+                        // do an initial get
+                        liveEmit();
+                        // and then every 5 seconds after that
+                        liveInterval = setInterval(async () => await liveEmit(), 5000);
+                    }
+                }
+            });
+
             if(session.botId !== undefined) {
-                const bot = cmInstances.find(x => x.friendly === session.botId);
+                const bot = cmInstances.find(x => x.getName() === session.botId);
                 if(bot !== undefined) {
                     // web log listener for bot specifically
                     const botWebLogListener = (log: string) => {
@@ -1100,7 +1163,7 @@ const webClient = async (options: OperatorConfig) => {
                                 }).json() as object;
                                 io.to(session.id).emit('opStats', resp);
                             } catch (err: any) {
-                                logger.error(`Could not retrieve stats ${err.message}`, {instance: bot.friendly});
+                                bot.logger.error(new ErrorWithCause('Could not retrieve stats', {cause: err}));
                                 clearInterval(interval);
                             }
                         }, 5000);
@@ -1113,12 +1176,12 @@ const webClient = async (options: OperatorConfig) => {
         socket.on('disconnect', (reason) => {
             clearSockStreams(socket.id);
             clearSockListeners(socket.id);
+            clearInterval(liveInterval);
         });
     });
 
     const loopHeartbeat = async () => {
         while(true) {
-            logger.debug('Starting heartbeat check');
             for(const c of clients) {
                 await refreshClient(c);
             }
@@ -1127,7 +1190,7 @@ const webClient = async (options: OperatorConfig) => {
         }
     }
 
-    const addBot = async (bot: CMInstance, userPayload: any, botPayload: any) => {
+    const addBot = async (bot: CMInstanceInterface, userPayload: any, botPayload: any) => {
         try {
             const token = createToken(bot, userPayload);
             const resp = await got.post(`${bot.normalUrl}/bot`, {
@@ -1144,81 +1207,13 @@ const webClient = async (options: OperatorConfig) => {
     }
 
     const refreshClient = async (client: BotConnection, force = false) => {
-        const normalized = normalizeUrl(client.host);
-        const existingClientIndex = cmInstances.findIndex(x => x.normalUrl === normalized);
-        const existingClient = existingClientIndex === -1 ? undefined : cmInstances[existingClientIndex];
+        const existingClientIndex = cmInstances.findIndex(x => x.matchesHost(client.host));
+        const instance = existingClientIndex === -1 ? new CMInstance(client, logger) : cmInstances[existingClientIndex];
 
-        let shouldCheck = false;
-        if(!existingClient) {
-            shouldCheck = true;
-        } else if(force) {
-            shouldCheck = true;
-        } else  {
-            const lastCheck = dayjs().diff(dayjs.unix(existingClient.lastCheck), 's');
-            if(!existingClient.online) {
-                if(lastCheck > 15) {
-                    shouldCheck = true;
-                }
-            } else if(lastCheck > 60) {
-                shouldCheck = true;
-            }
-        }
-        if(shouldCheck)
-        {
-            const machineToken = jwt.sign({
-                data: {
-                    machine: true,
-                },
-            }, client.secret, {
-                expiresIn: '1m'
-            });
-            //let base = `${c.host}${c.port !== undefined ? `:${c.port}` : ''}`;
-            const normalized = normalizeUrl(client.host);
-            const url = new URL(normalized);
-            let botStat: CMInstance = {
-                ...client,
-                subreddits: [] as string[],
-                operators: [] as string[],
-                operatorDisplay: '',
-                online: false,
-                friendly: url.host,
-                lastCheck: dayjs().unix(),
-                normalUrl: normalized,
-                url,
-                bots: [],
-            };
-            try {
-                const resp = await got.get(`${normalized}/heartbeat`, {
-                    headers: {
-                        'Authorization': `Bearer ${machineToken}`,
-                    }
-                }).json() as CMInstance;
+        await instance.checkHeartbeat(force);
 
-                const {bots, ...restResp} = resp;
-
-                botStat = {...botStat, ...restResp, bots: bots.map(x => ({...x, instance: botStat})), online: true};
-                const sameNameIndex = cmInstances.findIndex(x => x.friendly === botStat.friendly);
-                if(sameNameIndex > -1 && sameNameIndex !== existingClientIndex) {
-                    logger.warn(`Client returned a friendly name that is not unique (${botStat.friendly}), will fallback to host as friendly (${botStat.normalUrl})`);
-                    botStat.friendly = botStat.normalUrl;
-                }
-                botStat.online = true;
-                // if(botStat.online) {
-                //     botStat.indicator = botStat.running ? 'green' : 'yellow';
-                // } else {
-                //     botStat.indicator = 'red';
-                // }
-                logger.verbose(`Heartbeat detected`, {instance: botStat.friendly});
-            } catch (err: any) {
-                botStat.error = err.message;
-                logger.error(`Heartbeat response from ${botStat.friendly} was not ok: ${err.message}`, {instance: botStat.friendly});
-            } finally {
-                if(existingClientIndex !== -1) {
-                    cmInstances.splice(existingClientIndex, 1, botStat);
-                } else {
-                    cmInstances.push(botStat);
-                }
-            }
+        if(existingClientIndex === -1) {
+            cmInstances.push(instance);
         }
     }
 }

src/Web/Common/User/ClientUser.ts

@@ -1,14 +1,14 @@
-import {BotInstance, CMInstance} from "../../interfaces";
+import {BotInstance, CMInstanceInterface} from "../../interfaces";
 import CMUser from "./CMUser";
 import {intersect, parseRedditEntity} from "../../../util";
 
-class ClientUser extends CMUser<CMInstance, BotInstance, string> {
+class ClientUser extends CMUser<CMInstanceInterface, BotInstance, string> {
 
-    isInstanceOperator(val: CMInstance): boolean {
+    isInstanceOperator(val: CMInstanceInterface): boolean {
         return val.operators.map(x=> x.toLowerCase()).includes(this.name.toLowerCase());
     }
 
-    canAccessInstance(val: CMInstance): boolean {
+    canAccessInstance(val: CMInstanceInterface): boolean {
         return this.isInstanceOperator(val) || intersect(this.subreddits, val.subreddits.map(x => parseRedditEntity(x).name)).length > 0;
     }
 

src/Web/Common/User/ServerUser.ts

@@ -1,4 +1,4 @@
-import {BotInstance, CMInstance} from "../../interfaces";
+import {BotInstance, CMInstanceInterface} from "../../interfaces";
 import CMUser from "./CMUser";
 import {intersect, parseRedditEntity} from "../../../util";
 import {App} from "../../../App";

src/Web/Common/interfaces.ts

@@ -1,5 +1,6 @@
 import {RunningState} from "../../Subreddit/Manager";
 import {LogInfo, ManagerStats} from "../../Common/interfaces";
+import {BotInstance} from "../interfaces";
 
 export interface BotStats {
     startedAtHuman: string,
@@ -73,3 +74,11 @@ export interface IUser {
     token?: string
     tokenExpiresAt?: number
 }
+
+export interface HeartbeatResponse {
+    subreddits: string[]
+    operators: string[]
+    operatorDisplay?: string
+    friendly?: string
+    bots: BotInstance[]
+}

src/Web/Server/middleware.ts

@@ -46,19 +46,24 @@ export const subredditRoute = (required = true) => async (req: Request, res: Res
     if(subreddit === undefined && !required) {
         next();
     } else {
-        //const {name: userName} = req.user as Express.User;
 
-        const manager = bot.subManagers.find(x => x.displayLabel === subreddit);
-        if (manager === undefined) {
-            return res.status(400).send('Cannot access route for subreddit you do not manage or is not run by the bot')
+        if(subreddit.toLowerCase() === 'all') {
+            next();
+        } else {
+            //const {name: userName} = req.user as Express.User;
+
+            const manager = bot.subManagers.find(x => x.displayLabel === subreddit);
+            if (manager === undefined) {
+                return res.status(400).send('Cannot access route for subreddit you do not manage or is not run by the bot')
+            }
+
+            if (!req.user?.canAccessSubreddit(bot, subreddit)) {
+                return res.status(400).send('Cannot access route for subreddit you do not manage or is not run by the bot')
+            }
+
+            req.manager = manager;
+
+            next();
         }
-
-        if (!req.user?.canAccessSubreddit(bot, subreddit)) {
-            return res.status(400).send('Cannot access route for subreddit you do not manage or is not run by the bot')
-        }
-
-        req.manager = manager;
-
-        next();
     }
 }

src/Web/Server/routes/authenticated/user/action.ts

@@ -19,7 +19,7 @@ const action = async (req: Request, res: Response) => {
 
     for (const manager of subreddits) {
         const mLogger = manager.logger;
-        mLogger.info(`/u/${userName} invoked '${action}' action for ${type} on ${manager.displayLabel}`);
+        mLogger.info(`/u/${userName} invoked '${action}' action for ${type} on ${manager.displayLabel}`, {user: userName});
         try {
             switch (action) {
                 case 'start':

src/Web/Server/routes/authenticated/user/index.ts

@@ -106,7 +106,7 @@ const action = async (req: Request, res: Response) => {
     }
 
     if (a === undefined) {
-        winston.loggers.get('app').error('Could not parse Comment or Submission ID from given URL', {subreddit: `/u/${userName}`});
+        winston.loggers.get('app').error('Could not parse Comment or Submission ID from given URL', {user: userName});
         return res.send('OK');
     } else {
         // @ts-ignore
@@ -118,15 +118,15 @@ const action = async (req: Request, res: Response) => {
         if (manager === undefined || !req.user?.canAccessSubreddit(req.serverBot, manager.subreddit.display_name)) {
             let msg = 'Activity does not belong to a subreddit you moderate or the bot runs on.';
             if (subreddit === 'All') {
-                msg = `${msg} If you want to test an Activity against a Subreddit\'s config it does not belong to then switch to that Subreddit's tab first.`
+                msg = `${msg} If you want to test an Activity against a Subreddit's config it does not belong to then switch to that Subreddit's tab first.`
             }
-            winston.loggers.get('app').error(msg, {subreddit: `/u/${userName}`});
+            winston.loggers.get('app').error(msg, {user: userName});
             return res.send('OK');
         }
 
         // will run dryrun if specified or if running activity on subreddit it does not belong to
         const dr: boolean | undefined = (dryRun || manager.subreddit.display_name !== sub) ? true : undefined;
-        manager.logger.info(`/u/${userName} running${dr === true ? ' DRY RUN ' : ' '}check on${manager.subreddit.display_name !== sub ? ' FOREIGN ACTIVITY ' : ' '}${url}`);
+        manager.logger.info(`/u/${userName} running${dr === true ? ' DRY RUN ' : ' '}check on${manager.subreddit.display_name !== sub ? ' FOREIGN ACTIVITY ' : ' '}${url}`, {user: userName, subreddit});
         await manager.runChecks(activity instanceof Submission ? 'Submission' : 'Comment', activity, {dryRun: dr, force: true})
     }
     res.send('OK');

src/Web/Server/routes/authenticated/user/liveStats.ts

@@ -0,0 +1,261 @@
+import {authUserCheck, botRoute, subredditRoute} from "../../../middleware";
+import {Request, Response} from "express";
+import Bot from "../../../../../Bot";
+import {boolToString, cacheStats, filterLogs, formatNumber, logSortFunc, pollingInfo} from "../../../../../util";
+import dayjs from "dayjs";
+import {LogInfo, ResourceStats, RUNNING, STOPPED, SYSTEM} from "../../../../../Common/interfaces";
+import {Manager} from "../../../../../Subreddit/Manager";
+import winston from "winston";
+import {opStats} from "../../../../Common/util";
+import {BotStatusResponse} from "../../../../Common/interfaces";
+
+const liveStats = () => {
+    const middleware = [
+        authUserCheck(),
+        botRoute(),
+        subredditRoute(false),
+    ]
+
+    const response = async (req: Request, res: Response) =>
+    {
+        const bot = req.serverBot as Bot;
+        const manager = req.manager;
+        
+        if(manager === undefined) {
+            // getting all
+            const subManagerData: any[] = [];
+            for (const m of req.user?.accessibleSubreddits(bot) as Manager[]) {
+                const sd = {
+                    queuedActivities: m.queue.length(),
+                    runningActivities: m.queue.running(),
+                    maxWorkers: m.queue.concurrency,
+                    subMaxWorkers: m.subMaxWorkers || bot.maxWorkers,
+                    globalMaxWorkers: bot.maxWorkers,
+                    checks: {
+                        submissions: m.submissionChecks === undefined ? 0 : m.submissionChecks.length,
+                        comments: m.commentChecks === undefined ? 0 : m.commentChecks.length,
+                    },
+                    stats: await m.getStats(),
+                }
+            }
+
+            const totalStats = subManagerData.reduce((acc, curr) => {
+                return {
+                    checks: {
+                        submissions: acc.checks.submissions + curr.checks.submissions,
+                        comments: acc.checks.comments + curr.checks.comments,
+                    },
+                    historical: {
+                        allTime: {
+                            eventsCheckedTotal: acc.historical.allTime.eventsCheckedTotal + curr.stats.historical.allTime.eventsCheckedTotal,
+                            eventsActionedTotal: acc.historical.allTime.eventsActionedTotal + curr.stats.historical.allTime.eventsActionedTotal,
+                            checksRunTotal: acc.historical.allTime.checksRunTotal + curr.stats.historical.allTime.checksRunTotal,
+                            checksFromCacheTotal: acc.historical.allTime.checksFromCacheTotal + curr.stats.historical.allTime.checksFromCacheTotal,
+                            checksTriggeredTotal: acc.historical.allTime.checksTriggeredTotal + curr.stats.historical.allTime.checksTriggeredTotal,
+                            rulesRunTotal: acc.historical.allTime.rulesRunTotal + curr.stats.historical.allTime.rulesRunTotal,
+                            rulesCachedTotal: acc.historical.allTime.rulesCachedTotal + curr.stats.historical.allTime.rulesCachedTotal,
+                            rulesTriggeredTotal: acc.historical.allTime.rulesTriggeredTotal + curr.stats.historical.allTime.rulesTriggeredTotal,
+                            actionsRunTotal: acc.historical.allTime.actionsRunTotal + curr.stats.historical.allTime.actionsRunTotal,
+                        }
+                    },
+                    maxWorkers: acc.maxWorkers + curr.maxWorkers,
+                    subMaxWorkers: acc.subMaxWorkers + curr.subMaxWorkers,
+                    globalMaxWorkers: acc.globalMaxWorkers + curr.globalMaxWorkers,
+                    runningActivities: acc.runningActivities + curr.runningActivities,
+                    queuedActivities: acc.queuedActivities + curr.queuedActivities,
+                };
+            }, {
+                checks: {
+                    submissions: 0,
+                    comments: 0,
+                },
+                historical: {
+                    allTime: {
+                        eventsCheckedTotal: 0,
+                        eventsActionedTotal: 0,
+                        checksRunTotal: 0,
+                        checksFromCacheTotal: 0,
+                        checksTriggeredTotal: 0,
+                        rulesRunTotal: 0,
+                        rulesCachedTotal: 0,
+                        rulesTriggeredTotal: 0,
+                        actionsRunTotal: 0,
+                    }
+                },
+                maxWorkers: 0,
+                subMaxWorkers: 0,
+                globalMaxWorkers: 0,
+                runningActivities: 0,
+                queuedActivities: 0,
+            });
+            const {
+                checks,
+                maxWorkers,
+                globalMaxWorkers,
+                subMaxWorkers,
+                runningActivities,
+                queuedActivities,
+                ...rest
+            } = totalStats;
+
+            let cumRaw = subManagerData.reduce((acc, curr) => {
+                Object.keys(curr.stats.cache.types as ResourceStats).forEach((k) => {
+                    acc[k].requests += curr.stats.cache.types[k].requests;
+                    acc[k].miss += curr.stats.cache.types[k].miss;
+                    // @ts-ignore
+                    acc[k].identifierAverageHit += (typeof curr.stats.cache.types[k].identifierAverageHit === 'string' ? Number.parseFloat(curr.stats.cache.types[k].identifierAverageHit) : curr.stats.cache.types[k].identifierAverageHit);
+                    acc[k].averageTimeBetweenHits += curr.stats.cache.types[k].averageTimeBetweenHits === 'N/A' ? 0 : Number.parseFloat(curr.stats.cache.types[k].averageTimeBetweenHits)
+                });
+                return acc;
+            }, cacheStats());
+            cumRaw = Object.keys(cumRaw).reduce((acc, curr) => {
+                const per = acc[curr].miss === 0 ? 0 : formatNumber(acc[curr].miss / acc[curr].requests) * 100;
+                // @ts-ignore
+                acc[curr].missPercent = `${formatNumber(per, {toFixed: 0})}%`;
+                acc[curr].identifierAverageHit = formatNumber(acc[curr].identifierAverageHit);
+                acc[curr].averageTimeBetweenHits = formatNumber(acc[curr].averageTimeBetweenHits)
+                return acc;
+            }, cumRaw);
+            const cacheReq = subManagerData.reduce((acc, curr) => acc + curr.stats.cache.totalRequests, 0);
+            const cacheMiss = subManagerData.reduce((acc, curr) => acc + curr.stats.cache.totalMiss, 0);
+            const sharedSub = subManagerData.find(x => x.stats.cache.isShared);
+            const sharedCount = sharedSub !== undefined ? sharedSub.stats.cache.currentKeyCount : 0;
+            const scopes = req.user?.isInstanceOperator(bot) ? bot.client.scope : [];
+            let allManagerData: any = {
+                name: 'All',
+                status: bot.running ? 'RUNNING' : 'NOT RUNNING',
+                indicator: bot.running ? 'green' : 'grey',
+                maxWorkers,
+                globalMaxWorkers,
+                scopes: scopes === null || !Array.isArray(scopes) ? [] : scopes,
+                subMaxWorkers,
+                runningActivities,
+                queuedActivities,
+                botState: {
+                    state: RUNNING,
+                    causedBy: SYSTEM
+                },
+                dryRun: boolToString(bot.dryRun === true),
+                checks: checks,
+                softLimit: bot.softLimit,
+                hardLimit: bot.hardLimit,
+                stats: {
+                    ...rest,
+                    cache: {
+                        currentKeyCount: sharedCount + subManagerData.reduce((acc, curr) => curr.stats.cache.isShared ? acc : acc + curr.stats.cache.currentKeyCount,0),
+                        isShared: false,
+                        totalRequests: cacheReq,
+                        totalMiss: cacheMiss,
+                        missPercent: `${formatNumber(cacheMiss === 0 || cacheReq === 0 ? 0 : (cacheMiss / cacheReq) * 100, {toFixed: 0})}%`,
+                        types: {
+                            ...cumRaw,
+                        }
+                    }
+                },
+            };
+            // if(isOperator) {
+            allManagerData.startedAt = bot.startedAt.local().format('MMMM D, YYYY h:mm A Z');
+            allManagerData.heartbeatHuman = dayjs.duration({seconds: bot.heartbeatInterval}).humanize();
+            allManagerData.heartbeat = bot.heartbeatInterval;
+            allManagerData = {...allManagerData, ...opStats(bot)};
+            //}
+
+            const botDur = dayjs.duration(dayjs().diff(bot.startedAt))
+            if (allManagerData.stats.cache.totalRequests > 0) {
+                const minutes = botDur.asMinutes();
+                if (minutes < 10) {
+                    allManagerData.stats.cache.requestRate = formatNumber((10 / minutes) * allManagerData.stats.cache.totalRequests, {
+                        toFixed: 0,
+                        round: {enable: true, indicate: true}
+                    });
+                } else {
+                    allManagerData.stats.cache.requestRate = formatNumber(allManagerData.stats.cache.totalRequests / (minutes / 10), {
+                        toFixed: 0,
+                        round: {enable: true, indicate: true}
+                    });
+                }
+            } else {
+                allManagerData.stats.cache.requestRate = 0;
+            }
+
+            const data = {
+                bot: bot.getBotName(),
+                system: {
+                    startedAt: bot.startedAt.local().format('MMMM D, YYYY h:mm A Z'),
+                    running: bot.running,
+                    error: bot.error,
+                    ...opStats(bot),
+                },
+            };
+            return res.json(data);
+        } else {
+            // getting specific subreddit stats
+            const sd = {
+                name: manager.displayLabel,
+                botState: manager.botState,
+                eventsState: manager.eventsState,
+                queueState: manager.queueState,
+                indicator: 'gray',
+                permissions: await manager.getModPermissions(),
+                queuedActivities: manager.queue.length(),
+                runningActivities: manager.queue.running(),
+                maxWorkers: manager.queue.concurrency,
+                subMaxWorkers: manager.subMaxWorkers || bot.maxWorkers,
+                globalMaxWorkers: bot.maxWorkers,
+                validConfig: boolToString(manager.validConfigLoaded),
+                configFormat: manager.wikiFormat,
+                dryRun: boolToString(manager.dryRun === true),
+                pollingInfo: manager.pollOptions.length === 0 ? ['nothing :('] : manager.pollOptions.map(pollingInfo),
+                checks: {
+                    submissions: manager.submissionChecks === undefined ? 0 : manager.submissionChecks.length,
+                    comments: manager.commentChecks === undefined ? 0 : manager.commentChecks.length,
+                },
+                wikiRevisionHuman: manager.lastWikiRevision === undefined ? 'N/A' : `${dayjs.duration(dayjs().diff(manager.lastWikiRevision)).humanize()} ago`,
+                wikiRevision: manager.lastWikiRevision === undefined ? 'N/A' : manager.lastWikiRevision.local().format('MMMM D, YYYY h:mm A Z'),
+                wikiLastCheckHuman: `${dayjs.duration(dayjs().diff(manager.lastWikiCheck)).humanize()} ago`,
+                wikiLastCheck: manager.lastWikiCheck.local().format('MMMM D, YYYY h:mm A Z'),
+                stats: await manager.getStats(),
+                startedAt: 'Not Started',
+                startedAtHuman: 'Not Started',
+                delayBy: manager.delayBy === undefined ? 'No' : `Delayed by ${manager.delayBy} sec`,
+            };
+            // TODO replace indicator data with js on client page
+            let indicator;
+            if (manager.botState.state === RUNNING && manager.queueState.state === RUNNING && manager.eventsState.state === RUNNING) {
+                indicator = 'green';
+            } else if (manager.botState.state === STOPPED && manager.queueState.state === STOPPED && manager.eventsState.state === STOPPED) {
+                indicator = 'red';
+            } else {
+                indicator = 'yellow';
+            }
+            sd.indicator = indicator;
+            if (manager.startedAt !== undefined) {
+                const dur = dayjs.duration(dayjs().diff(manager.startedAt));
+                sd.startedAtHuman = `${dur.humanize()} ago`;
+                sd.startedAt = manager.startedAt.local().format('MMMM D, YYYY h:mm A Z');
+
+                if (sd.stats.cache.totalRequests > 0) {
+                    const minutes = dur.asMinutes();
+                    if (minutes < 10) {
+                        sd.stats.cache.requestRate = formatNumber((10 / minutes) * sd.stats.cache.totalRequests, {
+                            toFixed: 0,
+                            round: {enable: true, indicate: true}
+                        });
+                    } else {
+                        sd.stats.cache.requestRate = formatNumber(sd.stats.cache.totalRequests / (minutes / 10), {
+                            toFixed: 0,
+                            round: {enable: true, indicate: true}
+                        });
+                    }
+                } else {
+                    sd.stats.cache.requestRate = 0;
+                }
+            }
+
+            return res.json(sd);
+        }
+    }
+    return [...middleware, response];
+}
+
+export default liveStats;

src/Web/Server/routes/authenticated/user/logs.ts

@@ -1,19 +1,29 @@
 import {Router} from '@awaitjs/express';
 import {Request, Response} from 'express';
-import {filterLogBySubreddit, isLogLineMinLevel, LogEntry, parseSubredditLogName} from "../../../../../util";
+import {
+    filterLogBySubreddit,
+    filterLogs,
+    isLogLineMinLevel,
+    LogEntry,
+    parseSubredditLogName
+} from "../../../../../util";
 import {Transform} from "stream";
 import winston from "winston";
 import pEvent from "p-event";
 import {getLogger} from "../../../../../Utils/loggerFactory";
 import {booleanMiddle} from "../../../../Common/middleware";
-import {authUserCheck, botRoute} from "../../../middleware";
+import {authUserCheck, botRoute, subredditRoute} from "../../../middleware";
 import {LogInfo} from "../../../../../Common/interfaces";
 import {MESSAGE} from "triple-beam";
+import {Manager} from "../../../../../Subreddit/Manager";
+import Bot from "../../../../../Bot";
 
 // TODO update logs api
-const logs = (subLogMap: Map<string, LogEntry[]>) => {
+const logs = () => {
     const middleware = [
         authUserCheck(),
+        botRoute(false),
+        subredditRoute(false),
         booleanMiddle([{
             name: 'stream',
             defaultVal: false
@@ -33,8 +43,8 @@ const logs = (subLogMap: Map<string, LogEntry[]>) => {
             try {
                 logger.stream().on('log', (log: LogInfo) => {
                     if (isLogLineMinLevel(log, level as string)) {
-                        const {subreddit: subName} = log;
-                        if (isOperator || (subName !== undefined && (realManagers.includes(subName) || subName.includes(userName)))) {
+                        const {subreddit: subName, user} = log;
+                        if (isOperator || (subName !== undefined && (realManagers.includes(subName) || (user !== undefined && user.includes(userName))))) {
                             if(streamObjects) {
                                 let obj: any = log;
                                 if(!formatted) {
@@ -52,7 +62,7 @@ const logs = (subLogMap: Map<string, LogEntry[]>) => {
                 });
                 logger.info(`${userName} from ${origin} => CONNECTED`);
                 await pEvent(req, 'close');
-                console.log('Request closed detected with "close" listener');
+                //logger.debug('Request closed detected with "close" listener');
                 res.destroy();
                 return;
             } catch (e: any) {
@@ -64,24 +74,61 @@ const logs = (subLogMap: Map<string, LogEntry[]>) => {
                 res.destroy();
             }
         } else {
-            const logs = filterLogBySubreddit(subLogMap, realManagers, {
-                level: (level as string),
-                operator: isOperator,
-                user: userName,
-                sort: sort as 'descending' | 'ascending',
-                limit: Number.parseInt((limit as string)),
-                returnType: 'object',
-            });
-            const subArr: any = [];
-            logs.forEach((v: (string|LogInfo)[], k: string) => {
-                let logs = v as LogInfo[];
-                let output: any[] = formatted ? logs : logs.map((x) => {
-                    const {[MESSAGE]: fMessage, ...rest} = x;
-                    return rest;
-                })
-                subArr.push({name: k, logs: output});
-            });
-            return res.json(subArr);
+            let bots: Bot[] = [];
+            if(req.serverBot !== undefined) {
+                bots = [req.serverBot];
+            } else {
+                bots = req.user?.accessibleBots(req.botApp.bots) as Bot[];
+            }
+
+            const allReq = req.query.subreddit !== undefined && (req.query.subreddit as string).toLowerCase() === 'all';
+
+            const botArr: any = [];
+            for(const b of bots) {
+                const managerLogs = new Map<string, LogInfo[]>();
+                const managers = req.manager !== undefined ? [req.manager] : req.user?.accessibleSubreddits(b) as Manager[];
+                for (const m of managers) {
+                    const logs = filterLogs(m.logs, {
+                        level: (level as string),
+                        // @ts-ignore
+                        sort,
+                        limit: Number.parseInt((limit as string)),
+                        returnType: 'object'
+                    }) as LogInfo[];
+                    managerLogs.set(m.getDisplay(), logs);
+                }
+                const allLogs = filterLogs([...[...managerLogs.values()].flat(), ...(req.user?.isInstanceOperator(req.botApp) ? b.logs : b.logs.filter(x => x.user === req.user?.name))], {
+                    level: (level as string),
+                    // @ts-ignore
+                    sort,
+                    limit: limit as string,
+                    returnType: 'object'
+                }) as LogInfo[];
+                const systemLogs = filterLogs(req.user?.isInstanceOperator(req.botApp) ? b.logs : b.logs.filter(x => x.user === req.user?.name), {
+                    level: (level as string),
+                    // @ts-ignore
+                    sort,
+                    limit: limit as string,
+                    returnType: 'object'
+                }) as LogInfo[];
+                botArr.push({
+                    name: b.getBotName(),
+                    system: systemLogs,
+                    all: formatted ? allLogs.map(x => {
+                        const {[MESSAGE]: fMessage, ...rest} = x;
+                        return {...rest, formatted: fMessage};
+                    }) : allLogs,
+                    subreddits: allReq ? [] : [...managerLogs.entries()].reduce((acc: any[], curr) => {
+                        const l = formatted ? curr[1].map(x => {
+                            const {[MESSAGE]: fMessage, ...rest} = x;
+                            return {...rest, formatted: fMessage};
+                            }) : curr[1];
+                        acc.push({name: curr[0], logs: l});
+                        return acc;
+                    }, [])
+                });
+            }
+            return res.json(botArr);
         }
     };
 

src/Web/Server/routes/authenticated/user/status.ts

@@ -2,26 +2,27 @@ import {Request, Response} from 'express';
 import {
     boolToString,
     cacheStats,
-    filterLogBySubreddit,
+    filterLogBySubreddit, filterLogs,
     formatNumber,
     intersect,
-    LogEntry,
+    LogEntry, logSortFunc,
     pollingInfo
 } from "../../../../../util";
 import {Manager} from "../../../../../Subreddit/Manager";
 import dayjs from "dayjs";
-import {ResourceStats, RUNNING, STOPPED, SYSTEM} from "../../../../../Common/interfaces";
+import {LogInfo, ResourceStats, RUNNING, STOPPED, SYSTEM} from "../../../../../Common/interfaces";
 import {BotStatusResponse} from "../../../../Common/interfaces";
 import winston from "winston";
 import {opStats} from "../../../../Common/util";
-import {authUserCheck, botRoute} from "../../../middleware";
+import {authUserCheck, botRoute, subredditRoute} from "../../../middleware";
 import Bot from "../../../../../Bot";
 
 const status = () => {
 
     const middleware = [
         authUserCheck(),
-        //botRoute(),
+        botRoute(false),
+        subredditRoute(false)
     ];
 
     const response = async (req: Request, res: Response) => {
@@ -32,28 +33,17 @@ const status = () => {
             sort = 'descending',
         } = req.query;
 
-        // @ts-ignore
-        const botLogMap = req.botLogs as Map<string, Map<string, LogEntry[]>>;
-        // @ts-ignore
-        const systemLogs = req.systemLogs as LogEntry[];
+        bots = req.user?.accessibleBots(req.botApp.bots) as Bot[];
 
-
-        if(req.serverBot !== undefined) {
-            bots = [req.serverBot];
-        } else {
-            bots = req.user?.accessibleBots(req.botApp.bots) as Bot[];
-        }
         const botResponses: BotStatusResponse[] = [];
         let index = 1;
         for(const b of bots) {
-            botResponses.push(await botStatResponse(b, req, botLogMap, index));
+            botResponses.push(await botStatResponse(b, req, index));
             index++;
         }
         const system: any = {};
-        if(req.user?.isInstanceOperator(req.botApp)) {
-            // @ts-ignore
-            system.logs = filterLogBySubreddit(new Map([['app', systemLogs]]), [], {level, sort, limit, operator: true}).get('all');
-        }
+        // @ts-ignore
+        system.logs = filterLogs(req.sysLogs, {level, sort, limit, user: req.user?.isInstanceOperator(req.botApp) ? undefined : req.user?.name, returnType: 'object' }) as LogInfo[];
         const response = {
             bots: botResponses,
             system: system,
@@ -61,7 +51,7 @@ const status = () => {
         return res.json(response);
     }
 
-    const botStatResponse = async (bot: Bot, req: Request, botLogMap: Map<string, Map<string, LogEntry[]>>, index: number) => {
+    const botStatResponse = async (bot: Bot, req: Request, index: number) => {
         const {
             //subreddits = [],
             //user: userVal,
@@ -71,28 +61,26 @@ const status = () => {
             lastCheck
         } = req.query;
 
-        const user = req.user?.name as string;
-
-        const logs = filterLogBySubreddit(botLogMap.get(bot.botName as string) || new Map(), req.user?.accessibleSubreddits(bot).map(x => x.displayLabel) as string[], {
-            level: (level as string),
-            operator: req.user?.isInstanceOperator(req.botApp),
-            user,
-            // @ts-ignore
-            sort,
-            limit: Number.parseInt((limit as string)),
-            returnType: 'object'
-        });
+        const allReq = req.query.subreddit !== undefined && (req.query.subreddit as string).toLowerCase() === 'all';
 
         const subManagerData = [];
         for (const m of req.user?.accessibleSubreddits(bot) as Manager[]) {
+            const logs = req.manager === undefined || allReq || req.manager.getDisplay() === m.getDisplay() ? filterLogs(m.logs, {
+                    level: (level as string),
+                    // @ts-ignore
+                    sort,
+                    limit: limit as string,
+                    returnType: 'object'
+                }) as LogInfo[]: [];
             const sd = {
                 name: m.displayLabel,
                 //linkName: s.replace(/\W/g, ''),
-                logs: logs.get(m.displayLabel) || [], // provide a default empty value in case we truly have not logged anything for this subreddit yet
+                logs: logs || [], // provide a default empty value in case we truly have not logged anything for this subreddit yet
                 botState: m.botState,
                 eventsState: m.eventsState,
                 queueState: m.queueState,
                 indicator: 'gray',
+                permissions: [],
                 queuedActivities: m.queue.length(),
                 runningActivities: m.queue.running(),
                 maxWorkers: m.queue.concurrency,
@@ -232,12 +220,22 @@ const status = () => {
         const cacheMiss = subManagerData.reduce((acc, curr) => acc + curr.stats.cache.totalMiss, 0);
         const sharedSub = subManagerData.find(x => x.stats.cache.isShared);
         const sharedCount = sharedSub !== undefined ? sharedSub.stats.cache.currentKeyCount : 0;
+        const scopes = req.user?.isInstanceOperator(bot) ? bot.client.scope : [];
+        const allSubLogs = subManagerData.map(x => x.logs).flat().sort(logSortFunc(sort as string)).slice(0, (limit as number) + 1);
+        const allLogs = filterLogs([...allSubLogs, ...(req.user?.isInstanceOperator(req.botApp) ? bot.logs : bot.logs.filter(x => x.user === req.user?.name))], {
+            level: (level as string),
+            // @ts-ignore
+            sort,
+            limit: limit as string,
+            returnType: 'object'
+        }) as LogInfo[];
         let allManagerData: any = {
             name: 'All',
             status: bot.running ? 'RUNNING' : 'NOT RUNNING',
             indicator: bot.running ? 'green' : 'grey',
             maxWorkers,
             globalMaxWorkers,
+            scopes: scopes === null || !Array.isArray(scopes) ? [] : scopes,
             subMaxWorkers,
             runningActivities,
             queuedActivities,
@@ -246,7 +244,7 @@ const status = () => {
                 causedBy: SYSTEM
             },
             dryRun: boolToString(bot.dryRun === true),
-            logs: logs.get('all'),
+            logs: allLogs,
             checks: checks,
             softLimit: bot.softLimit,
             hardLimit: bot.hardLimit,
@@ -302,7 +300,7 @@ const status = () => {
                 name: (bot.botName as string) ?? `Bot ${index}`,
                 ...opStats(bot),
             },
-            subreddits: [allManagerData, ...subManagerData],
+            subreddits: [allManagerData, ...(allReq ? subManagerData.map(({logs, ...x}) => ({...x, logs: []})) : subManagerData)],
 
         };
 

src/Web/Server/server.ts

@@ -9,11 +9,6 @@ import {Strategy as JwtStrategy, ExtractJwt} from 'passport-jwt';
 import passport from 'passport';
 import tcpUsed from 'tcp-port-used';
 
-import {
-    intersect,
-    LogEntry, parseBotLogName,
-    parseSubredditLogName
-} from "../../util";
 import {getLogger} from "../../Utils/loggerFactory";
 import LoggedError from "../../Utils/LoggedError";
 import {Invokee, LogInfo, OperatorConfigWithFileContext} from "../../Common/interfaces";
@@ -21,13 +16,13 @@ import http from "http";
 import {heartbeat} from "./routes/authenticated/applicationRoutes";
 import logs from "./routes/authenticated/user/logs";
 import status from './routes/authenticated/user/status';
+import liveStats from './routes/authenticated/user/liveStats';
 import {actionedEventsRoute, actionRoute, configRoute, configLocationRoute, deleteInviteRoute, addInviteRoute, getInvitesRoute} from "./routes/authenticated/user";
 import action from "./routes/authenticated/user/action";
 import {authUserCheck, botRoute} from "./middleware";
 import {opStats} from "../Common/util";
 import Bot from "../../Bot";
 import addBot from "./routes/authenticated/user/addBot";
-import dayjs from "dayjs";
 import ServerUser from "../Common/User/ServerUser";
 import {SimpleError} from "../../Utils/Errors";
 import {ErrorWithCause} from "pony-cause";
@@ -47,11 +42,7 @@ declare module 'express-session' {
     }
 }
 
-const subLogMap: Map<string, LogEntry[]> = new Map();
-const systemLogs: LogEntry[] = [];
-const botLogMap: Map<string, Map<string, LogEntry[]>> = new Map();
-
-const botSubreddits: Map<string, string[]> = new Map();
+let sysLogs: LogInfo[] = [];
 
 const rcbServer = async function (options: OperatorConfigWithFileContext) {
 
@@ -73,36 +64,12 @@ const rcbServer = async function (options: OperatorConfigWithFileContext) {
     const logger = getLogger({...options.logging});
 
     logger.stream().on('log', (log: LogInfo) => {
-        const logEntry: LogEntry = [dayjs(log.timestamp).unix(), log];
 
         const {bot: botName, subreddit: subName} = log;
 
-        if(botName === undefined) {
-            systemLogs.unshift(logEntry);
-            systemLogs.slice(0, 201);
-        } else {
-            const botLog = botLogMap.get(botName) || new Map();
-
-            if(subName === undefined) {
-                const appLogs = botLog.get('app') || [];
-                appLogs.unshift(logEntry);
-                botLog.set('app', appLogs.slice(0, 200 + 1));
-            } else {
-                let botSubs = botSubreddits.get(botName) || [];
-                if(app !== undefined && (botSubs.length === 0 || !botSubs.includes(subName))) {
-                    const b = app.bots.find(x => x.botName === botName);
-                    if(b !== undefined) {
-                        botSubs = b.subManagers.map(x => x.displayLabel);
-                        botSubreddits.set(botName, botSubs);
-                    }
-                }
-                if(botSubs.length === 0 || botSubs.includes(subName)) {
-                    const subLogs = botLog.get(subName) || [];
-                    subLogs.unshift(logEntry);
-                    botLog.set(subName, subLogs.slice(0, 200 + 1));
-                }
-            }
-            botLogMap.set(botName, botLog);
+        if(botName === undefined && subName === undefined) {
+            sysLogs.unshift(log);
+            sysLogs = sysLogs.slice(0, 201);
         }
     })
 
@@ -167,7 +134,7 @@ const rcbServer = async function (options: OperatorConfigWithFileContext) {
 
     server.getAsync('/heartbeat', ...heartbeat({name, display, friendly}));
 
-    server.getAsync('/logs', ...logs(subLogMap));
+    server.getAsync('/logs', ...logs());
 
     server.getAsync('/stats', [authUserCheck(), botRoute(false)], async (req: Request, res: Response) => {
         let bots: Bot[] = [];
@@ -186,13 +153,13 @@ const rcbServer = async function (options: OperatorConfigWithFileContext) {
     });
     const passLogs = async (req: Request, res: Response, next: Function) => {
         // @ts-ignore
-        req.botLogs = botLogMap;
-        // @ts-ignore
-        req.systemLogs = systemLogs;
+        req.sysLogs = sysLogs;
         next();
     }
     server.getAsync('/status', passLogs, ...status())
 
+    server.getAsync('/liveStats', ...liveStats())
+
     server.getAsync('/config', ...configRoute);
 
     server.getAsync('/config/location', ...configLocationRoute);

src/Web/assets/public/app.css

@@ -152,3 +152,7 @@ a {
 #saveTip .tooltip:hover {
     transition-delay: 1s;
 }
+
+#redditStatus .iconify-inline {
+    display: inline;
+}

src/Web/assets/views/partials/footer.ejs

@@ -1,5 +1,29 @@
 <div class="py-3 flex items-center justify-around font-semibold">
     <div>
-        <a href="https://github.com/FoxxMD/context-mod">ContextMod Web</a> created by /u/FoxxMD
+        <a href="https://github.com/FoxxMD/context-mod">ContextMod <%= locals.applicationIdentifier%></a> created by /u/FoxxMD
     </div>
 </div>
+<script type="text/javascript" src="https://cdn.statuspage.io/se-v2.js"></script>
+<script>
+    // https://www.redditstatus.com/api#status
+    var sp = new StatusPage.page({ page : '2kbc0d48tv3j' });
+    sp.status({
+        success : function(data) {
+            console.log(data.status.indicator);
+            switch(data.status.indicator){
+                case 'minor':
+                    document.querySelector('#redditStatus').innerHTML = '<span class="iconify-inline yellow" data-icon="ep:warning-filled"></span>';
+                    break;
+                case 'none':
+                    document.querySelector('#redditStatus').innerHTML = '<span class="iconify-inline green" data-icon="ep:circle-check-filled"></span>';
+                    break;
+                default:
+                    document.querySelector('#redditStatus').innerHTML = '<span class="iconify-inline red" data-icon="ep:warning-filled"></span>';
+                    break;
+            }
+            // data.page.updated_at
+            // data.status.indicator => none, minor, major, or critical
+            // data.status.description
+        }
+    });
+</script>

src/Web/assets/views/partials/head.ejs

@@ -1,6 +1,6 @@
 <head>
     <link rel="stylesheet" href="/public/tailwind.min.css"/>
-    <script src="https://code.iconify.design/2/2.1.0/iconify.min.js"></script>
+    <script src="https://code.iconify.design/2/2.1.2/iconify.min.js"></script>
     <link rel="stylesheet" href="/public/themeToggle.css">
     <link rel="stylesheet" href="/public/app.css">
     <title><%= locals.title !== undefined ? title : `${locals.botName !== undefined ? `CM for ${botName}` : 'ContextMod'}`%></title>

src/Web/assets/views/partials/header.ejs

@@ -33,7 +33,12 @@
                 </ul>
                 <% } %>
             </div>
-            <div class="flex items-center flex-end text-sm">
+            <div class="flex items-center mr-8 text-sm">
+                <a href="https://redditstatus.com" target="_blank">
+                    <span>Reddit Status: <span id="redditStatus" class="ml-2"><span class="iconify-inline" data-icon="ep:question-filled"></span></span></span>
+                </a>
+            </div>
+            <div class="flex items-center text-sm">
                 <a href="logout">Logout</a>
             </div>
         </div>

src/Web/assets/views/status.ejs

@@ -160,6 +160,19 @@
                                                         </div>
                                                     </div>
                                                 </div>
+                                                <label>Mod Perms</label>
+                                                <span class="has-tooltip">
+                                                    <span class='tooltip rounded shadow-lg p-1 bg-gray-100 text-black -mt-2'>
+                                                        <span>
+                                                            <ul class="list-inside list-disc modPermissionsList">
+                                                                <% data.permissions.forEach(function (i){ %>
+                                                                    <li class="font-mono"><%= i %></li>
+                                                                <% }) %>
+                                                            </ul>
+                                                        </span>
+                                                    </span>
+                                                    <span class="cursor-help underline modPermissionsCount" style="text-decoration-style: dotted"><%= data.permissions.length %></span>
+                                                </span>
                                                 <label>Slow Mode</label>
                                                 <span><%= data.delayBy %></span>
                                             <% } %>
@@ -223,6 +236,19 @@
                                             <% if (data.name === 'All' && isOperator) { %>
                                                 <label>Operators</label>
                                                 <span><%= operators %></span>
+                                                <label>Oauth Scopes</label>
+                                                    <span class="has-tooltip">
+                                                    <span class='tooltip rounded shadow-lg p-1 bg-gray-100 text-black -mt-2'>
+                                                        <span>
+                                                            <ul class="list-inside list-disc">
+                                                                <% data.scopes.forEach(function (i){ %>
+                                                                    <li class="font-mono"><%= i %></li>
+                                                                <% }) %>
+                                                            </ul>
+                                                        </span>
+                                                    </span>
+                                                    <span class="cursor-help underline" style="text-decoration-style: dotted"><%= data.scopes.length %></span>
+                                                </span>
                                             <% } else %>
                                         </div>
                                         <% if (data.name !== 'All') { %>
@@ -651,6 +677,48 @@
 </div>
 <%- include('partials/instanceTabJs') %>
 <%- include('partials/logSettingsJs') %>
+<script src="https://unpkg.com/autolinker@3.14.3/dist/Autolinker.min.js" crossorigin="anonymous"></script>
+<script src="https://unpkg.com/dayjs@1.10.7/dayjs.min.js" crossorigin="anonymous"></script>
+<script src="https://unpkg.com/dayjs@1.10.7/plugin/advancedFormat.js"></script>
+<script src="https://unpkg.com/dayjs@1.10.7/plugin/timezone.js"></script>
+<script>
+    dayjs.extend(window.dayjs_plugin_timezone)
+    dayjs.extend(window.dayjs_plugin_advancedFormat)
+    window.formattedTime = (short, full) => `<span class="has-tooltip"><span style="margin-top:35px" class='tooltip rounded shadow-lg p-1 bg-gray-100 text-black space-y-3 p-2 text-left'>${full}</span><span>${short}</span></span>`;
+    window.formatLogLineToHtml = (log, timestamp = undefined) => {
+        const val = typeof log === 'string' ? log : log['MESSAGE'];
+        const logContent = Autolinker.link(val, {
+            email: false,
+            phone: false,
+            mention: false,
+            hashtag: false,
+            stripPrefix: false,
+            sanitizeHtml: true,
+        })
+            .replace(/(\s*debug\s*):/i, '<span class="debug blue">$1</span>:')
+            .replace(/(\s*warn\s*):/i, '<span class="warn yellow">$1</span>:')
+            .replace(/(\s*info\s*):/i, '<span class="info green">$1</span>:')
+            .replace(/(\s*error\s*):/i, '<span class="error red">$1</span>:')
+            .replace(/(\s*verbose\s*):/i, '<span class="error purple">$1</span>:')
+            .replaceAll('\n', '<br />');
+        //.replace(HYPERLINK_REGEX, '<a target="_blank" href="$&">$&</a>');
+        let line = '';
+
+        let timestampString = timestamp;
+        if(timestamp === undefined && typeof log !== 'string') {
+            timestampString = log.timestamp;
+        }
+
+        if(timestampString !== undefined) {
+            const timeStampReplacement = formattedTime(dayjs(timestampString).format('HH:mm:ss z'), timestampString);
+            const splitLine = logContent.split(timestampString);
+            line = `<div class="logLine">${splitLine[0]}${timeStampReplacement}<span style="white-space: pre-wrap">${splitLine[1]}</span></div>`;
+        } else {
+            line = `<div style="white-space: pre-wrap" class="logLine">${logContent}</div>`
+        }
+        return line;
+    }
+</script>
 <script>
     window.sort = 'desc';
 
@@ -811,6 +879,44 @@
                 var newRelativePathQuery = window.location.pathname + '?' + searchParams.toString();
                 history.pushState(null, '', newRelativePathQuery);
             }
+            const activeSub = document.querySelector(`[data-subreddit="${subreddit}"][data-bot="${bot}"].sub`);
+            if(!activeSub.classList.contains('seen')) {
+                //firstSub.classList.add('seen');
+
+                //subreddit = firstSub.dataset.subreddit;
+                //bot = subSection.dataset.bot;
+                level = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="level"]`).value;
+                sort = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="sort"]`).value;
+                limitSel = document.querySelector(`[data-subreddit="${subreddit}"] [data-type="limit"]`).value;
+
+                fetch(`/api/logs?instance=<%= instanceId %>&bot=${bot}&subreddit=${subreddit}&level=${level}&sort=${sort}&limit=${limitSel}&stream=false&formatted=true`).then((resp) => {
+                    if (!resp.ok) {
+                        console.error('Response was not OK from logs GET');
+                    } else {
+                        resp.json().then((data) => {
+                            const logContainer = document.querySelector(`[data-subreddit="${subreddit}"] .logs`);
+                            const logLines = (subreddit.toLowerCase() === 'all' ? data[0].all : data[0].subreddits[0].logs).map(x => {
+                                let fString = x.formatted;
+                                if(x.bot !== undefined) {
+                                    fString = fString.replace(`~${x.bot}~ `, '');
+                                }
+                                if(x.subreddit !== undefined && subreddit !== 'All') {
+                                    fString = fString.replace(`{${x.subreddit}} `, '');
+                                }
+                                return window.formatLogLineToHtml(fString, x.timestamp)
+                            }).join('');
+                            logContainer.insertAdjacentHTML('afterbegin', logLines);
+                            activeSub.classList.add('seen');
+                        });
+                    }
+                }).catch((err) => {
+                    console.log(err);
+                });
+
+            }
+            if(window.socket !== undefined) {
+                window.socket.emit('viewing', {bot, subreddit});
+            }
         });
     });
 
@@ -838,7 +944,7 @@
         tabSelect.classList.add('font-bold', 'no-underline', 'pointer-events-none');
     }
     document.querySelectorAll('.tabSelectWrapper').forEach(el => el.classList.add('border'));
-    document.querySelector(`[data-bot="${shownBot}"][data-subreddit="${shownSub}"].sub`).classList.add('active');
+    document.querySelector(`[data-bot="${shownBot}"][data-subreddit="${shownSub}"].sub`).classList.add('active', 'seen');
     const subWrapper = document.querySelector(`ul[data-bot="${shownBot}"] [data-subreddit="${shownSub}"].tabSelectWrapper`);
     if(subWrapper !== null) {
         subWrapper.classList.remove('border');
@@ -865,6 +971,7 @@
     let socket = io({
         reconnectionAttempts: 5, // bail after 5 attempts
     });
+    window.socket = socket;
 
     // get all bots
     let bots = [];
@@ -882,6 +989,11 @@
 
     socket.on("connect", () => {
         document.body.classList.add('connected')
+
+        const shownSub = searchParams.get('sub') || 'All'
+        let shownBot = searchParams.get('bot');
+        window.socket.emit('viewing', {bot: shownBot, subreddit: shownSub});
+
         socket.on("log", data => {
             const {
                 subreddit,
@@ -916,12 +1028,12 @@
                     }
                     subLogs.forEach((logs, subKey) => {
                         // check sub exists -- may be a web log
-                        const el = document.querySelector(`[data-subreddit="${subKey}"][data-bot="${botName}"].sub`);
+                        const el = document.querySelector(`[data-subreddit="${subKey}"][data-bot="${botName}"].sub.seen`);
                         if(null !== el) {
                             const limit = Number.parseInt(document.querySelector(`[data-subreddit="${subKey}"] [data-type="limit"]`).value);
                             const logContainer = el.querySelector(`.logs`);
                             let existingLogs;
-                            if(window.sort === 'desc') {
+                            if(window.sort === 'desc' || window.sort === 'descending') {
                                 logs.forEach((l) => {
                                     logContainer.insertAdjacentHTML('afterbegin', l);
                                 })
@@ -973,6 +1085,29 @@
             }
 
         });
+        socket.on('liveStats', (resp) => {
+            let el;
+            let isAll = resp.system !== undefined;
+            if(isAll) {
+                // got all
+                el = document.querySelector(`[data-subreddit="All"][data-bot="${resp.bot}"].sub`);
+            } else {
+                // got subreddit
+                el = document.querySelector(`[data-subreddit="${resp.name}"].sub`);
+            }
+
+            if(isAll) {
+
+            } else {
+                if(el.querySelector('.modPermissionsCount').innerHTML != resp.permissions.length) {
+                    el.querySelector('.modPermissionsCount').innerHTML = resp.permissions.length;
+                    el.querySelector('.modPermissionsList').innerHTML = '';
+                    el.querySelector('.modPermissionsList').insertAdjacentHTML('afterbegin', resp.permissions.map(x => `<li class="font-mono">${x}</li>`).join(''));
+                }
+            }
+
+            console.log(resp);
+        });
     });
 
     socket.on('disconnect', () => {

src/Web/interfaces.ts

@@ -1,5 +1,6 @@
 import { URL } from "url";
 import {BotConnection} from "../Common/interfaces";
+import {Logger} from "winston";
 
 export interface BotInstance {
     botName: string
@@ -8,16 +9,16 @@ export interface BotInstance {
     subreddits: string[]
     nanny?: string
     running: boolean
-    instance: CMInstance
+    instance: CMInstanceInterface
 }
 
-export interface CMInstance extends BotConnection {
-    friendly: string
+export interface CMInstanceInterface extends BotConnection {
+    friendly?: string
     operators: string[]
     operatorDisplay: string
     url: URL,
     normalUrl: string,
-    lastCheck: number
+    lastCheck?: number
     online: boolean
     subreddits: string[]
     bots: BotInstance[]

src/Web/types/express/index.d.ts

@@ -1,6 +1,6 @@
 import {App} from "../../../App";
 import Bot from "../../../Bot";
-import {BotInstance, CMInstance} from "../../interfaces";
+import {BotInstance, CMInstanceInterface} from "../../interfaces";
 import {Manager} from "../../../Subreddit/Manager";
 import CMUser from "../../Common/User/CMUser";
 
@@ -9,7 +9,7 @@ declare global {
         interface Request {
             botApp: App;
             token?: string,
-            instance?: CMInstance,
+            instance?: CMInstanceInterface,
             bot?: BotInstance,
             serverBot: Bot,
             manager?: Manager,

src/util.ts

@@ -6,8 +6,6 @@ import deepEqual from "fast-deep-equal";
 import {Duration} from 'dayjs/plugin/duration.js';
 import Ajv from "ajv";
 import {InvalidOptionArgumentError} from "commander";
-import Submission from "snoowrap/dist/objects/Submission";
-import {Comment} from "snoowrap";
 import {inflateSync, deflateSync} from "zlib";
 import pixelmatch from 'pixelmatch';
 import os from 'os';
@@ -37,14 +35,14 @@ import {
 import { Document as YamlDocument } from 'yaml'
 import InvalidRegexError from "./Utils/InvalidRegexError";
 import {constants, promises} from "fs";
-import {cacheOptDefaults} from "./Common/defaults";
+import {cacheOptDefaults, VERSION} from "./Common/defaults";
 import cacheManager, {Cache} from "cache-manager";
 import redisStore from "cache-manager-redis-store";
 import crypto from "crypto";
 import Autolinker from 'autolinker';
 import {create as createMemoryStore} from './Utils/memoryStore';
 import {MESSAGE, LEVEL} from "triple-beam";
-import {RedditUser} from "snoowrap/dist/objects";
+import {RedditUser,Comment,Submission} from "snoowrap/dist/objects";
 import reRegExp from '@stdlib/regexp-regexp';
 import fetch, {Response} from "node-fetch";
 import { URL } from "url";
@@ -136,7 +134,7 @@ const errorAwareFormat = {
                 for(const k in err) {
                     if(dummyErr.hasOwnProperty(k) || k === 'cause') {
                         // @ts-ignore
-                        dummyErr[k] = tinfo[k];
+                        dummyErr[k] = info[k];
                     }
                 }
                 // @ts-ignore
@@ -160,7 +158,7 @@ const isProbablyError = (val: any, errName = 'error') => {
     return typeof val === 'object' && val.name !== undefined && val.name.toLowerCase().includes(errName);
 }
 
-export const PASS = '✔';
+export const PASS = '✓';
 export const FAIL = '✘';
 
 export const truncateStringToLength = (length: number, truncStr = '...') => (str: string) => str.length > length ? `${str.slice(0, length - truncStr.length - 1)}${truncStr}` : str;
@@ -1161,9 +1159,14 @@ export const formatLogLineToHtml = (log: string | LogInfo, timestamp?: string) =
         //.replace(HYPERLINK_REGEX, '<a target="_blank" href="$&">$&</a>');
     let line = '';
 
-    if(timestamp !== undefined) {
-        const timeStampReplacement = formattedTime(dayjs(timestamp).format('HH:mm:ss z'), timestamp);
-        const splitLine = logContent.split(timestamp);
+    let timestampString = timestamp;
+    if(timestamp === undefined && typeof log !== 'string') {
+        timestampString = (log as LogInfo).timestamp;
+    }
+
+    if(timestampString !== undefined) {
+        const timeStampReplacement = formattedTime(dayjs(timestampString).format('HH:mm:ss z'), timestampString);
+        const splitLine = logContent.split(timestampString);
         line = `<div class="logLine">${splitLine[0]}${timeStampReplacement}<span style="white-space: pre-wrap">${splitLine[1]}</span></div>`;
     } else {
         line = `<div style="white-space: pre-wrap" class="logLine">${logContent}</div>`
@@ -1173,7 +1176,7 @@ export const formatLogLineToHtml = (log: string | LogInfo, timestamp?: string) =
 
 export type LogEntry = [number, LogInfo];
 export interface LogOptions {
-    limit: number,
+    limit: number | string,
     level: string,
     sort: 'ascending' | 'descending',
     operator?: boolean,
@@ -1185,7 +1188,7 @@ export interface LogOptions {
 
 export const filterLogBySubreddit = (logs: Map<string, LogEntry[]>, validLogCategories: string[] = [], options: LogOptions): Map<string, (string|LogInfo)[]> => {
     const {
-        limit,
+        limit: limitVal,
         level,
         sort,
         operator = false,
@@ -1195,6 +1198,7 @@ export const filterLogBySubreddit = (logs: Map<string, LogEntry[]>, validLogCate
         returnType = 'string',
     } = options;
 
+    let limit = typeof limitVal === 'number' ? limitVal : Number.parseInt(limitVal);
     // get map of valid logs categories
     const validSubMap: Map<string, LogEntry[]> = new Map();
     for(const [k, v] of logs) {
@@ -1244,6 +1248,35 @@ export const filterLogBySubreddit = (logs: Map<string, LogEntry[]>, validLogCate
     return preparedMap;
 }
 
+export const logSortFunc = (sort: string = 'ascending') => sort === 'ascending' ? (a: LogInfo, b: LogInfo) => (dayjs(a.timestamp).isSameOrAfter(b.timestamp) ? 1 : -1) : (a: LogInfo, b: LogInfo) => (dayjs(a.timestamp).isSameOrBefore(b.timestamp) ? 1 : -1);
+
+export const filterLogs= (logs: LogInfo[], options: LogOptions): LogInfo[] | string[] => {
+    const {
+        limit: limitVal,
+        level,
+        sort,
+        operator = false,
+        user,
+        allLogsParser = parseSubredditLogInfoName,
+        allLogName = 'app',
+        returnType = 'string',
+    } = options;
+
+    let limit = typeof limitVal === 'number' ? limitVal : Number.parseInt(limitVal);
+    let leveledLogs = logs.filter(x => isLogLineMinLevel(x, level));
+    if(user !== undefined) {
+        leveledLogs = logs.filter(x => x.user !== undefined && x.user === user);
+    }
+    leveledLogs.sort(logSortFunc(sort));
+    leveledLogs = leveledLogs.slice(0, limit + 1);
+
+    if(returnType === 'string') {
+        return leveledLogs.map(x => formatLogLineToHtml(x));
+    } else {
+        return leveledLogs;
+    }
+}
+
 export const logLevels = {
     error: 0,
     warn: 1,
@@ -1577,13 +1610,33 @@ export const snooLogWrapper = (logger: Logger) => {
  * Cached activities lose type information when deserialized so need to check properties as well to see if the object is the shape of a Submission
  * */
 export const isSubmission = (value: any) => {
-    return value instanceof Submission || value.domain !== undefined;
+    return value instanceof Submission || value.name.includes('t3_');
 }
 
 export const asSubmission = (value: any): value is Submission => {
     return isSubmission(value);
 }
 
+export const isComment = (value: any) => {
+    return value instanceof Comment || value.name.includes('t1_');
+}
+
+export const asComment = (value: any): value is Comment => {
+    return isComment(value);
+}
+
+export const asActivity = (value: any): value is (Submission | Comment) => {
+    return asComment(value) || asSubmission(value);
+}
+
+export const isUser = (value: any) => {
+    return value instanceof RedditUser || value.name.includes('t2_');
+}
+
+export const asUser = (value: any): value is RedditUser => {
+    return isUser(value);
+}
+
 export const isUserNoteCriteria = (value: any) => {
     return value !== null && typeof value === 'object' && value.type !== undefined;
 }
@@ -2020,3 +2073,46 @@ export const likelyJson5 = (str: string): boolean => {
     }
     return validStart;
 }
+
+const defaultScanOptions = {
+    COUNT: '100',
+    MATCH: '*'
+}
+/**
+ * Frankenstein redis scan generator
+ *
+ * Cannot use the built-in scan iterator because it is only available in > v4 of redis client but node-cache-manager-redis is using v3.x --
+ * So combining the async iterator defined in v4 from here https://github.com/redis/node-redis/blob/master/packages/client/lib/client/index.ts#L587
+ * with the scan example from v3 https://github.com/redis/node-redis/blob/8a43dea9bee11e41d33502850f6989943163020a/examples/scan.js
+ *
+ * */
+export async function* redisScanIterator(client: any, options: any = {}): AsyncIterable<string> {
+    let cursor: string = '0';
+    const scanOpts = {...defaultScanOptions, ...options};
+    do {
+        const iterScan = new Promise((resolve, reject) => {
+            client.scan(cursor, 'MATCH', scanOpts.MATCH, 'COUNT', scanOpts.COUNT, (err: any, res: any) => {
+                if(err) {
+                    return reject(err);
+                } else {
+                    const newCursor = res[0];
+                    let keys = res[1];
+                    resolve([newCursor, keys]);
+                }
+            });
+        }) as Promise<[any, string[]]>;
+        const [newCursor, keys] = await iterScan;
+        cursor = newCursor;
+        for (const key of keys) {
+            yield key;
+        }
+    } while (cursor !== '0');
+}
+
+export const getUserAgent = (val: string, fragment?: string) => {
+    return `${replaceApplicationIdentifier(val, fragment)} (developed by /u/FoxxMD)`;
+}
+
+export const replaceApplicationIdentifier = (val: string, fragment?: string) => {
+    return val.replace('{VERSION}', `v${VERSION}`).replace('{FRAG}', (fragment !== undefined ? `-${fragment}` : ''));
+}