Commit Graph

472 Commits

Author SHA1 Message Date
FoxxMD
d96a1f677c refactor(image): Profile image comparison analysis time to debug statement 2021-10-06 14:14:36 -04:00
FoxxMD
b14689791c refactor(image): Document resemble compare options for performance
* Disable color comparison is slower than nothing
* Restrict comparison to a max of 800x800 pixels is slower than nothing
2021-10-06 14:13:50 -04:00
FoxxMD
b70c877e44 refactor(polling): Improve processed list configuration/usage
* Change 'after' type to string duration for friendlier configuration
* Decrease list size trigger === limit instead of 2x (not necessary to have a list that big for polling new)
* Increase initial shared mod polling to max limit (100)
2021-10-06 12:54:23 -04:00
FoxxMD
041655376a feat(filter): Implement age test for comment/submission
Closes #27
2021-10-06 12:54:23 -04:00
FoxxMD
e1eab7696b feat(polling): Implement memory bloat mitigation for long-running SnooStorm polling
The list of processed activities SnooStorm uses to ensure only new activities are emitted when polling is never cleared. MayorMonty/Snoostorm#35

To mitigate the memory bloat this creates when RCB runs for a long time on high-volume subreddits implement user-configurable (with defaults) behavior for clearing the processed activity list. Default values ensure clearing the list does not interfere with checking for new activities.
2021-10-05 14:45:10 -04:00
FoxxMD
65d1d36d53 feat(cache): Implement batching for subreddit info
When using full criteria for subreddit state we can save a ton of api calls by get info for all uncached subreddits at the same time rather than individually
2021-09-29 16:20:02 -04:00
FoxxMD
120d776fc2 feat(ui): Display subreddit and subreddit crit statistics in cache call breakdown popup 2021-09-29 10:47:33 -04:00
FoxxMD
425e16295b refactor(cache): Better cache usage for some known state conditions
* Don't store subreddit state cache results for now since nothing computationally expensive or requires api requests
* Return early on item state check if there is nothing to check so we don't store an empty result in cache
2021-09-29 10:47:05 -04:00
FoxxMD
dd7e9d72cc fix(subreddit): Fix subreddit state testing edge case and undefined criteria
* allow both over_18 and over18 criteria in case user accidentally used name from sub/comm state
* correctly determine if subreddit property exists when testing
* fix cache hit subreddit name logging
2021-09-29 10:21:16 -04:00
FoxxMD
55535ddd62 fix(regex): Fix regex generation for simple subreddit strings
* Trim value before parsing
* If not a valid regex string then when generating regex from simple string add qualifiers for beginning/end of string so any matches must be exact
2021-09-29 09:22:40 -04:00
FoxxMD
be6fa4dd50 fix(cache): Fix accidental re-use of maps 2021-09-24 16:13:58 -04:00
FoxxMD
0d7a82836f refactor(cache): Move bot usage stats into cache
* Moving into cache means stats will persist after restart (yay!)
* Refactored stats structure to be simpler
2021-09-24 15:24:19 -04:00
FoxxMD
d9a59b6824 feat(recent): Print log statement when image processing is causing rule to take a long time 2021-09-23 13:23:58 -04:00
FoxxMD
ddbf8c3189 fix(recent): Actually use filtered activities when using submission as reference 2021-09-23 12:58:43 -04:00
FoxxMD
8393c471b2 fix(image): Dynamically import resemblejs for better compatibility on systems not supporting node-canvas
* By dynamically importing the module any user not using image comparison will not be affected by a lack of node-canvas dependency
* try-catch on import and provide a helpful error message about node-canvas dep
2021-09-23 10:38:34 -04:00
FoxxMD
fe66a2e8f7 fix(docker): Update build to build node-canvas from source 2021-09-23 10:08:26 -04:00
FoxxMD
4b0284102d fix: Improve image comparison threshold and results for typescript 2021-09-22 22:15:00 -04:00
FoxxMD
95529f14a8 feat(recent): Implement pixel-level image comparison when using a reference (image) submission 2021-09-22 16:52:56 -04:00
FoxxMD
26af2c4e4d fix(recent): don't include submission being checked when filtering by reference 2021-09-22 10:29:06 -04:00
FoxxMD
044c293f34 fix(attribution): Update aggregateOn defaults to align with expected behavior
Majority of mods that have used this rule assume it does not aggregate on reddit domains by default (only external links), which is reasonable.
So update the default to follow this assumption.
2021-09-22 10:11:25 -04:00
FoxxMD
a082c9e593 doc(attribution): Remove unused useSubmissionAsReference property 2021-09-22 09:36:44 -04:00
FoxxMD
e242c36c09 fix(tooling): Fix tag pattern for git cliff 2021-09-21 15:18:26 -04:00
FoxxMD
c5018183e0 fix(attribution): Improve parsing of domain type to fix bug with galleries
* Add `redditMedia` as distinct domain type from `self` for more granular aggregation
* Use `redditMedia` to fix bug where video and galleries were being counted as `media`
2021-09-20 16:34:29 -04:00
FoxxMD
c5358f196d feat(author): Handle shadowbanned users
* Allow checking if user is shadowbanned via authorIs (AuthorCriteria)
* try-catch on history get or author criteria to try to detect shadowbanned user for a more descriptive error
2021-09-20 13:49:35 -04:00
FoxxMD
1d9f8245f9 feat(tooling): scope-based sorting with BC note for git cliff generation 2021-09-20 11:51:25 -04:00
FoxxMD
20b37f3a40 Initial git cliff config 2021-09-20 11:03:37 -04:00
FoxxMD
641892cd3e fix: Fix activity push to manager
Should only be using firehose
2021-09-20 09:37:32 -04:00
FoxxMD
1dfb9779e7 feat(attribution): Allow specifying aggregateOn filter when using domain blacklist
May not make sense all the time but a properly configured config could take advantage of this
2021-09-17 15:14:36 -04:00
FoxxMD
40111c54a2 feat(message): Add a markdown formatted 'ruleSummary' property to content template data 2021-09-17 14:38:39 -04:00
FoxxMD
b4745e3b45 feat(message): Implement arbitrary message recipient to enable modmail
* Can send message to any entity (user/subreddit) using 'to' property, or leave unspecified to send to author of activity
* Parse entity type (user or subreddit) from to value and ensure its in a valid format we can understand with regex
2021-09-17 13:36:28 -04:00
FoxxMD
838da497ce feat: Add more detail to actioned events and logging for action results 2021-09-17 12:46:00 -04:00
FoxxMD
01755eada5 feat: De-dup activities from different polling sources
Previously CM would process the same activity multiple times if it was ingested from two different polling sources (modqueue and unmoderated/newSub). Introduce queue control flow to ensure activity is de-duped or refreshed before processing if this scenario occurs.

* Use a queue (firehose) to bottleneck all activities from different sources before pushing to worker queues
* Keep track of items currently ingested but not completely processed and use firehose to de-dupe queued items (flag to refresh) or re-queue if currently processing (and flag to refresh)
2021-09-17 11:50:49 -04:00
FoxxMD
1ff59ad6e8 feat: Add report count comparison to comment/submission state 2021-09-17 10:21:46 -04:00
FoxxMD
d8fd8e6140 feat: Add score (karma) comparison to comment/submission state 2021-09-17 10:13:21 -04:00
FoxxMD
255ffdb417 fix(recent): Deduplicate present subreddits 2021-09-16 16:48:00 -04:00
FoxxMD
f0199366a0 feat(history)!: Implement subreddit state and subreddit name parsing
* Implement total threshold to compare filtered activities against window activities

BREAKING CHANGE: include/exclude now filters POST activity window and all comparisons are done on those filtered activities against window activities
2021-09-16 15:36:06 -04:00
FoxxMD
20c724cab5 fix: Fix bug where non-media domains were not counted for attribution rule 2021-09-16 15:33:59 -04:00
FoxxMD
a670975f14 feat(repeat activity): Implement subreddit state and regex parsing 2021-09-16 14:12:16 -04:00
FoxxMD
ee13feaf57 feat(recent activity): Implement subreddit state and regex parsing for recent activity
* SubredditState can be used to check some subreddit attributes alongside, or in place of, a subreddit name
* Regex parsing for subreddit name string in recent activity
2021-09-16 13:34:19 -04:00
FoxxMD
23a24b4448 feat(regex)!: Simplify regex parsing from config
Reduce regex complexity in config by parsing a normal regex straight from config string value (including flags)

BREAKING CHANGE: regex must now be enclosed in forward slashes, flags must be on regex value, and regexFlags property has been removed
2021-09-16 10:53:33 -04:00
FoxxMD
269b1620b9 fix(regex): fix when to add match sample to summary
Only include summary if there were non-zero matches
2021-09-07 15:04:30 -04:00
FoxxMD
6dee734440 fix(recent activity): fix subreddit summary in result
* Only include if any subreddits with activity found
* Use correct subreddit array
2021-09-07 15:03:56 -04:00
FoxxMD
3aea422eff fix(cache): check manager has cache (valid config) before trying to get actioned events 2021-09-07 14:56:03 -04:00
FoxxMD
e707e5a9a8 fix(manager): revert commented notification line from debugging 2021-09-07 14:07:00 -04:00
FoxxMD
2a24eea3a5 fix(rule): fix regex rule matching default behavior and improve log message
* Default to global flag if none specified so that all matches per activity are found
* Improve result message section ordering and display a sample of up to 4 found matches
2021-09-07 14:06:30 -04:00
FoxxMD
8ad8297c0e feat(rule): improve recent activity result (log message) by listing on subreddits with found activity 2021-09-07 14:04:26 -04:00
FoxxMD
0b94a14ac1 feat(ui): improve actioned events interactions
* Refactor api to get all accessible events, sorted by time, when subreddit is not specified
* Add subreddit name to actioned event data to differentiate between events
* Show actioned events link in "All" subreddit view
* Remove user-select css style (left over from config template)
* Format timestamp to be more human friendly
* Remove success/triggered text and just use checkmarks (same as log)
2021-09-07 13:33:35 -04:00
FoxxMD
a04e0d2a9b fix(cache): Set actioned events not to expire in cache 2021-09-07 13:26:30 -04:00
FoxxMD
3a1348c370 feat(ui): move actioned events link to "Actions Run" statistic
More intuitive location
2021-09-07 12:59:51 -04:00
FoxxMD
507818037f feat(cache): refactor actioned events into cache for persistence and make number stored configurable
* refactor actioned events into bot-configured cache so they can be persisted between restarts
* add config params for actionedEventsMax and actionedEventsDefault to allow defining defaults at operator/bot/subreddit level
2021-09-07 12:55:19 -04:00