Commit Graph

67 Commits

Author SHA1 Message Date
FoxxMD
207907881f feat: Add initial setup wizard
* If web client does not have credentials (or operator) then redirect login to form to fill them in
  * Write to config location on form completion
2022-08-15 17:17:23 -04:00
FoxxMD
52ae16be1c refactor: Reduce circular dependencies 2022-05-09 20:12:08 -04:00
FoxxMD
3ad5b26be6 fix(ts): Fix TS issue with typings in same folder as regular files preventing schema generation
* Rename interface/types folder
* Move typigns back into folder with same name
2022-05-09 11:09:33 -04:00
FoxxMD
d20c2becc8 feat(window): Refactor window to support subreddit and item filtering, everywhere
* Move some interfaces and types into own files to breakup huge interfaces file
* Refactor window shape for config and "full" usage in app to support subreddit/item filtering
* Refactor author activities into resources class so we can take advantage of caching on subreddit/item filter results
2022-05-09 11:04:08 -04:00
FoxxMD
fd173b6a2f feat(database): Implement user-configurable time series statistics
* Collect same stats as all time but on at a specified frequency
* Frequency is configurable at operator, bot, and subreddit level
* Operator and bot level frequency can have an enforced minimum
2022-04-20 13:44:35 -04:00
FoxxMD
ccee0ef6c0 refactor: Move author interfaces and utility functions into common files 2022-04-08 10:42:16 -04:00
FoxxMD
2e8f90c1a8 feat(database): More database and config improvements for startup
* Improve errors when checking if file is readable or directory/file is writable
* Implement custom winston-typeorm logging mappings so migrations are output at INFO level
* Add typeorm logging options to database config and default to displaying error, warning, and migrations
* Ensure directory is writeable before using log rotation transport for winston
* Move init logger into index so its always available
* Simplify printing stack for SimpleError when it's a cause
* Better handling for access permissions when reading operator config file
* Add hint to errors relating to permissions access if app detects its running in docker
2022-03-31 13:05:07 -04:00
FoxxMD
1a4606ea74 refactor(dispatch): Simplify source data and dispatch/delay data
* Use typeorm to transform delay from int to duration
* Also fix run state init for managers
2022-03-29 16:49:04 -04:00
FoxxMD
9b5111aabf feat(database): Implement activity source data structure
* Implement a generic activity source to track where an event was retrieved from and some initial parameters
* Consolidate dispatch action data into generic source
* Refactor delayUtil logic for event handling to prevent blocking worker for a non-trivial amount of time by dispatching event
2022-03-24 11:20:52 -04:00
FoxxMD
8c33e44e3c fix(database): Fix duplicated filter criteria
* Refactor/remove Author class and replace with function (was unnecessary class)
* Workaround fix for composite key on filter criteria entity
2022-03-23 11:20:25 -04:00
FoxxMD
b62f7f7bb5 refactor: Move RuleResult interface and associated structures into interfaces common file 2022-03-22 09:01:06 -04:00
FoxxMD
b561fa6b3c refactor(database): Partial conflict resolution with edge
Still need to refactor interfaces/code related to storing actioned events
2022-03-17 10:00:21 -04:00
FoxxMD
1f3d0b50a7 feat: Implement re-run
* Implement rerun configuration that satisfies requirements from #72
  * rerun as action
  * optional, user-defined identifier
  * cancel rerun as action
  * cancel based on re-queued sources
  * on existing behavior
  * can specify initial goto
* filter item by source (where item was retrieved from for non-cached items)
* filter item by rerun state/identifier
* Add rerun label to event logging
* Add rerun data to actioned event data
2022-03-04 15:52:05 -05:00
FoxxMD
7f9adcef36 refactor: Refactor manager/run/check handling for better single responsibility 2022-02-09 16:47:57 -05:00
FoxxMD
4eae07f831 feat(flow): Implement basic flow control structures #73
* Add Run and postCheckBehavior config structures to schema and interfaces
* Implement parsing from config and initial flow logic for running on activities in manager
2022-02-08 13:39:02 -05:00
FoxxMD
41038b9bcd feat(logging): Implement richer errors everywhere
* Use ErrorWithCause so we can get and print a chain of error causes
* Make reddit error response in stack trace more readable by replacing them with a "translated" parent response and add them as the cause
* Properly handle error formatting for winston by looking at shape of log object for error rather than testing instanceof (see comments in errorAwareFormat)
* Fix formatting in web interface for log lines with white-space pre css and properly splitting timestamp from rest of the message
2022-01-27 16:27:03 -05:00
FoxxMD
5811af0342 feat(config): Refactor config parsing to preserve comments and enable writing
* use node-comment and yaml@next to keep comment information intact
* store ast/source version of parsed config for operator
* implement generic yaml/json operator config classes to keep everything organized and simplify marshalling source to js/string
* refactor file parsing and json/yaml parsing to have better single responsibility
2022-01-17 15:51:43 -05:00
FoxxMD
dcbeb784e8 refactor: Remove set-random-interval
* depended on and always downloaded an entire, older typescript version (even with production install) -- was not necessary for one function
* refactor project to use newer TS version (specify any type for catch blocks to fix compiler errors)
2021-12-17 14:25:48 -05:00
FoxxMD
b2846efd2b fix: Improve falloff behavior when reddit api error are encountered
* Turn off snoowrap request queuing
* Aggregate errors from all managers at bot-level and force all to stop (and clear queue/polling) if a small threshold is met
* Add activity refresh on check into try-catch so delayed activities in queue don't cause a loop if they fail due to api issues
2021-12-06 17:16:04 -05:00
FoxxMD
641892cd3e fix: Fix activity push to manager
Should only be using firehose
2021-09-20 09:37:32 -04:00
FoxxMD
0b94a14ac1 feat(ui): improve actioned events interactions
* Refactor api to get all accessible events, sorted by time, when subreddit is not specified
* Add subreddit name to actioned event data to differentiate between events
* Show actioned events link in "All" subreddit view
* Remove user-select css style (left over from config template)
* Format timestamp to be more human friendly
* Remove success/triggered text and just use checkmarks (same as log)
2021-09-07 13:33:35 -04:00
FoxxMD
b6d638d6c5 Implement easier way to determine if reddit api error is due to insufficient oauth scope 2021-08-23 12:00:49 -04:00
FoxxMD
2fd1ffed19 Rewrite docs to explain server-client architecture 2021-08-20 13:47:55 -04:00
FoxxMD
0f84a7cf6b Lots of log (streaming) improvements
* Use autolinker to deal with url parsing in logs and sanitize html
* Fix missing mergeArr arguments on child loggers
* Implement DuplexTransport to make streaming from winston less verbose and allow access to log object
* Refactor log parsing on server/client side to deal with log objects or strings
2021-08-18 15:47:31 -04:00
FoxxMD
c817716aa1 First pass at refactoring to use new instance/bot/subreddit architecture 2021-08-16 14:14:42 -04:00
FoxxMD
3d79a9217a Refactor the oauth helper flow
* Integrate into the main client
* Do more checks for correct credentials before auth code flow init
* Implement special bypass token to help with authorization flow with account other than operator/logged in
* Add more credential options to helper page (can provide separate id/secret) and permission options
2021-08-12 11:57:54 -04:00
FoxxMD
5c76f9ab1c Continue to simplify stream handling
* Cleanup stream usage for winston and simplify adding to logger
* Use pipeline with delimiter stream for parsing logs from client
* End log streaming with promise abort (cleaner)
2021-08-10 23:10:06 -04:00
FoxxMD
3b905e6961 General improvements for ui and error handling
* Move Bot webhook listeners for crash/shutdown into Bot code itself
* Simplify running modes
* Parse web operator into user data
* Add basic status tooltip for bots in ui
2021-08-10 16:39:39 -04:00
FoxxMD
2442fc2483 More refactoring for client/server decoupling
Most things working now...
2021-08-09 17:44:55 -04:00
FoxxMD
e762cc29ef Big ol mess for now 2021-08-06 22:04:29 -04:00
FoxxMD
af1ea5543e Implement caching for more components
* Implement caching for specific activities
* Implement/refactor item is criteria to cache activities and results
2021-08-02 15:10:49 -04:00
FoxxMD
4865259ae8 Improve app exit notifications and fix wiki location
* Refactor server so app is passed back to main index.js so we can handle SIGTERM in a central location and determine if exit was based on uncaught error or not
* Fix missing assignment of default wikiLocation to manager
* await discord notifier so on app exit the notification is actually sent before exit
2021-07-28 10:45:12 -04:00
FoxxMD
67ed8ab4ee Add more caching info and tooltips for major components 2021-07-26 14:47:58 -04:00
FoxxMD
7cda47183f Simplify cli options and improve web server checks
* Check for in-use web port before starting so we can display a useful error message
* Remove 'web' command and instead make it an optional argument for `run` (default to web)
* Update dockerfile to use run command
* Remove remaining commander option defaults since they are defined in config builder now
2021-07-26 12:18:51 -04:00
FoxxMD
73c3052c69 Refactor how configuration is parsed
* Implement a weak/strong interface for operator (app) config
* Consolidate config parsing code into ConfigBuilder
* Implement parsing configuration from a json file
* Refactor configuration parsing so there is a clear hierarchy to where and how config is overridden
2021-07-22 15:17:41 -04:00
FoxxMD
c69d66c001 Initial implementation of Regex Rule 2021-07-21 15:47:26 -04:00
FoxxMD
9316019b01 Implement api nanny mode to help with heavy api usage 2021-07-20 20:15:15 -04:00
FoxxMD
4642f67104 Implement loading operator arguments (main program) from file 2021-07-20 16:38:35 -04:00
FoxxMD
c9bdae66dd Implement configurable shared modqueue for operator convenience
If operator is running subreddits for lop-sided traffic burdens then /r/mod may be over the 100 activity limit, all from subreddit X while sub Y and Z have few activities. In this scenario Y and Z would never run since X would take up all results. Therefore default to individual modqueues and make shared modqueue opt-in by operator
2021-07-19 23:04:47 -04:00
FoxxMD
6ab162888b try-catch all the things 2021-07-19 22:37:50 -04:00
FoxxMD
5df4fd5ccc Separate event polling and activity processing systems to improve robustness
* Use async/queue for handling activity processing on each manager
* Refactor polling to push activities into activity queue
* Refactor manager state to control activity processing queue and event polling independently
* Pause activity processing and wait until no work is being done to do config update
* Add way more logging for new systems
* Add basic ui controls/view for new systems
2021-07-19 16:32:41 -04:00
FoxxMD
6167d45e49 Refactor configuration loading to be independent of manager initialization
* Makes it easier to deal with never-successful subs (from startup) and invalid configs during run
* Paves the way for adding managers in-situ
* Add validConfig property to manager to track this and in UI
* Track if user manually stopped manager so we don't try to restart on heartbeat
2021-07-16 16:57:25 -04:00
FoxxMD
d84e6f1905 Implement oauth helper ui 2021-07-15 19:34:01 -04:00
FoxxMD
f9c7cf433e Switch to web as default run command and provide fallback
* Use 'web' as run command for dockerfile
* If on 'web' there is no redirect uri fallback to cli (doesn't break compatibility this way)
2021-07-15 15:44:47 -04:00
FoxxMD
2917233728 More startup logging and better logging on invalid/missing credentials 2021-07-15 15:31:56 -04:00
FoxxMD
14e7275f64 Implement operator-specific UI components
* Add api limit, reset, and heartbeat interval to All overview when operator is viewing
* Stream op stats on log emit
* Add env/arg for setting "Operated by"
2021-07-15 14:57:05 -04:00
FoxxMD
15e031d448 Implement web ui
* Refactor program options to allow running as web
* Implement authentication using reddit oauth
* Use session in memory to store identification and accessible subreddits
* Implement socket.io with shared session to enable streaming logs
* Implement log streaming with per-subreddit views
2021-07-15 01:18:21 -04:00
FoxxMD
15824e5d0f Implement delay before processing
Using criteria for Activity being N seconds old allows for delay with possibility of immediate processing to avoid api call

Closes #23
2021-07-07 15:23:59 -04:00
FoxxMD
6ca65079b3 Add author and item checks to Actions
So that Actions can be skipped based on item/author criteria
2021-06-21 13:00:45 -04:00
FoxxMD
58c51e56b1 Test Author account age using string comparison (like automod) 2021-06-18 13:52:52 -04:00