521 Commits

Author SHA1 Message Date
FoxxMD
39daa11f2d Merge branch 'edge' 0.10.1 2021-11-15 12:53:28 -05:00
FoxxMD
93de38a845 fix(filter): Missing 'op' property test on itemIs for comment 2021-11-15 12:47:34 -05:00
FoxxMD
43caaca1f2 fix(config): Fix wrong provider data used when bot has no caching config
Use previously built defaultProvider, instead of hardcoded memory config, so any cache config from operator waterfalls to bot/subreddit

Closes #46
2021-11-02 15:55:03 -04:00
FoxxMD
7bcc0195fe fix(author): change include/exclude on RULE to be optional
One of these properties must be present for the rule to be valid. This is enforced at runtime. Change the schema so that both are optional from the config-side though.
2021-11-02 09:41:55 -04:00
FoxxMD
dac6541e28 Merge branch 'edge' 0.10.0 2021-11-01 16:12:43 -04:00
FoxxMD
2504a34a34 refactor(docker): Remove old steps from node-canvas dep
Not using node-canvas anymore so dockerfile can be simplified #48
2021-11-01 15:55:12 -04:00
FoxxMD
e19639ad0d fix(ui): Forgot to add tailwin css to git
Oops!
2021-11-01 15:06:50 -04:00
FoxxMD
b8084e02b5 Merge branch 'colorImprovements' into edge 2021-11-01 14:58:01 -04:00
FoxxMD
97906281e6 Merge branch 'edge' 0.9.4 2021-11-01 14:55:10 -04:00
FoxxMD
2cea119657 feat(ui): Add tooltip descriptions for depleted/limit reset stats
Closes #36
2021-11-01 14:52:35 -04:00
FoxxMD
6f16d289dd feat(ui): Add lang attribute for html
Closes #35
2021-11-01 14:22:20 -04:00
FoxxMD
a96575c6b3 refactor(ui): Use local tailwind css asset
Closes #34
2021-11-01 14:19:47 -04:00
FoxxMD
0a82e83352 refactor(ui)!: Implement colorbind color changes for rest of UI and remove dark/light mode
* Apply changes made in ac409dce to the rest of the application
* Remove dark/light mode -- now always dark mode (easier to maintain this way)
* Remove dependency on tailwind dark css
2021-11-01 14:14:49 -04:00
FoxxMD
d5e1cdec61 fix(criteria): Improve criteria filtering for removed/deleted activities and logging
* Use different logging messages when criteria is not available due to mod permissions (property not available to non-mods)
* Change logging level for missing/unavailable criteria to reduce logging noise. On unavailable use debug, on missing use warn
* Improve activity removed/deleted detection based on whether activity is moddable by current user
2021-11-01 13:25:54 -04:00
FoxxMD
ef40c25b09 feat(attribution): Add additional subreddit and activity filtering functionality to criteria
* Refactor subreddit filtering with include/exclude to use subreddit state
* Add submissionState and commentState filters
2021-11-01 11:26:55 -04:00
FoxxMD
6370a2976a Merge branch 'edge' into colorImprovements 2021-11-01 10:23:40 -04:00
FoxxMD
d8180299ea fix(author): Fix missing true return statement for author flair check 2021-11-01 10:23:22 -04:00
FoxxMD
ac409dce3d POC color improvements for color blindness
Needs another pass and consolidate/clean up for pages other than status
2021-10-20 21:12:31 -04:00
FoxxMD
56c007c20d feat(author): Implement author profile description regex/string testing
May test "description" in authorIs as a regular express of string literal -- or as an array of the aforementioned values
2021-10-20 19:52:46 -04:00
FoxxMD
487f13f704 Merge branch 'edge' 0.9.3 2021-10-12 11:56:51 -04:00
FoxxMD
00b9d87cdc Remove unused types package 2021-10-12 11:56:29 -04:00
FoxxMD
2c797e0b9b docs(image): Add documentation for image comparison
#26
2021-10-12 11:10:14 -04:00
FoxxMD
4a2b27bfbf fix(recent): Revert debug change for concurrency on activity comparisons 2021-10-12 11:08:53 -04:00
FoxxMD
463a4dc0eb feat(image): Implement perceptual hashing image comparison
* Generate perceptual hashes using blockhash-js of images that can be cache/stored
* Take advantage of reddit thumbnail code ImageData to hash lower-res to begin with (but represent full url)
* Refactor imageDetection config so hash and pixel approaches have different configs
* Cache phash results to reduce reddit traffic and speed up performance

Addresses cpu/memory issues with pixel comparison. Allow pixel for finer comparisons if needed using tiered thresholds. Closes #26
2021-10-11 15:28:48 -04:00
FoxxMD
4b3bea661d Merge branch 'imageComparisonOptimization' into edge
# Conflicts:
#	src/Rule/RecentActivityRule.ts
2021-10-08 13:50:49 -04:00
FoxxMD
976f310f51 refactor(image): Remove resemblejs dependency
* Refactor image comparison to use pixelmatch only so resemblejs can be removed (too much memory usage)
* Heavier usage of sharp to get images into same dimensions prior to pixelmatch
* Refactor image conversion into ImageData to clean up utils/recent activity rule
2021-10-08 13:49:46 -04:00
FoxxMD
4d8d3dc266 fix(docker): Add libvips dependency 2021-10-08 13:47:35 -04:00
FoxxMD
ce9e678c4c fix(image): clone sharp instance so converting between outputs doesn't cause issues 2021-10-07 18:06:05 -04:00
FoxxMD
8cf30b6b7d refactor: Introduce staggered startup for bots/polling to decrease load on host/reddit and improve image comparison performance
* Implement staggered startup for bots (reddit accounts, top-level)
* Implement staggered startup for managers (subreddits) and subreddit polling
* Introduce random -1/+1 second to polling interval for every stream to ensure none are synced so there is no instantaneous spike in cpu/traffic/memory on host/reddit
* Add user-configurable stagger interval for shared mod polling
* Implement second image comparison approach with pixelmatch for reduced memory usage when image dimensions are exactly the same
* Use sharp to resize images to 400 width max when using resemblejs to reduce memory usage
2021-10-07 17:13:27 -04:00
FoxxMD
2b6d08f8a5 Dummy commit
To get github actions to run
2021-10-06 18:03:26 -04:00
FoxxMD
f8fc63991f feat(image): Leverage reddit image previews for download/comparison
* Refactor image acquisition/parsing to use on-demand fetching and track different resolutions
* Try to use smaller previews (under 1000px), when possible, for comparing images and downloading
* Do image comparisons in parallel
2021-10-06 17:33:56 -04:00
FoxxMD
d96a1f677c refactor(image): Profile image comparison analysis time to debug statement 2021-10-06 14:14:36 -04:00
FoxxMD
b14689791c refactor(image): Document resemble compare options for performance
* Disable color comparison is slower than nothing
* Restrict comparison to a max of 800x800 pixels is slower than nothing
2021-10-06 14:13:50 -04:00
FoxxMD
b70c877e44 refactor(polling): Improve processed list configuration/usage
* Change 'after' type to string duration for friendlier configuration
* Decrease list size trigger === limit instead of 2x (not necessary to have a list that big for polling new)
* Increase initial shared mod polling to max limit (100)
2021-10-06 12:54:23 -04:00
FoxxMD
041655376a feat(filter): Implement age test for comment/submission
Closes #27
2021-10-06 12:54:23 -04:00
FoxxMD
e1eab7696b feat(polling): Implement memory bloat mitigation for long-running SnooStorm polling
The list of processed activities SnooStorm uses to ensure only new activities are emitted when polling is never cleared. MayorMonty/Snoostorm#35

To mitigate the memory bloat this creates when RCB runs for a long time on high-volume subreddits implement user-configurable (with defaults) behavior for clearing the processed activity list. Default values ensure clearing the list does not interfere with checking for new activities.
2021-10-05 14:45:10 -04:00
FoxxMD
65d1d36d53 feat(cache): Implement batching for subreddit info
When using full criteria for subreddit state we can save a ton of api calls by get info for all uncached subreddits at the same time rather than individually
2021-09-29 16:20:02 -04:00
FoxxMD
120d776fc2 feat(ui): Display subreddit and subreddit crit statistics in cache call breakdown popup 2021-09-29 10:47:33 -04:00
FoxxMD
425e16295b refactor(cache): Better cache usage for some known state conditions
* Don't store subreddit state cache results for now since nothing computationally expensive or requires api requests
* Return early on item state check if there is nothing to check so we don't store an empty result in cache
2021-09-29 10:47:05 -04:00
FoxxMD
dd7e9d72cc fix(subreddit): Fix subreddit state testing edge case and undefined criteria
* allow both over_18 and over18 criteria in case user accidentally used name from sub/comm state
* correctly determine if subreddit property exists when testing
* fix cache hit subreddit name logging
2021-09-29 10:21:16 -04:00
FoxxMD
55535ddd62 fix(regex): Fix regex generation for simple subreddit strings
* Trim value before parsing
* If not a valid regex string then when generating regex from simple string add qualifiers for beginning/end of string so any matches must be exact
2021-09-29 09:22:40 -04:00
FoxxMD
631e21452c Merge branch 'edge' 0.9.2 2021-09-28 16:36:13 -04:00
FoxxMD
be6fa4dd50 fix(cache): Fix accidental re-use of maps 2021-09-24 16:13:58 -04:00
FoxxMD
0d7a82836f refactor(cache): Move bot usage stats into cache
* Moving into cache means stats will persist after restart (yay!)
* Refactored stats structure to be simpler
2021-09-24 15:24:19 -04:00
FoxxMD
d9a59b6824 feat(recent): Print log statement when image processing is causing rule to take a long time 2021-09-23 13:23:58 -04:00
FoxxMD
ddbf8c3189 fix(recent): Actually use filtered activities when using submission as reference 2021-09-23 12:58:43 -04:00
FoxxMD
8393c471b2 fix(image): Dynamically import resemblejs for better compatibility on systems not supporting node-canvas
* By dynamically importing the module any user not using image comparison will not be affected by a lack of node-canvas dependency
* try-catch on import and provide a helpful error message about node-canvas dep
2021-09-23 10:38:34 -04:00
FoxxMD
fe66a2e8f7 fix(docker): Update build to build node-canvas from source 2021-09-23 10:08:26 -04:00
FoxxMD
4b0284102d fix: Improve image comparison threshold and results for typescript 2021-09-22 22:15:00 -04:00
FoxxMD
95529f14a8 feat(recent): Implement pixel-level image comparison when using a reference (image) submission 2021-09-22 16:52:56 -04:00