* Implement separate language detection functionality
* Clearer/simpler sentiment processing
* Add languageHints to help coerce low confidence language detection
* Add test cases for lang detection, sentiment detection, and sentiment tests
* Fix neutral range -- was not using normalized score range
* Build NLP container ad-hoc so only supported languages are included from npm
* Use vader/wink as heuristics for detecting language when content is very short
* Add languageHint option for sentiment config to make coercing a confident sentiment easier
* Refactor lang processing to fail the sentiment test rather than throwing an error when language is not support/not confident -- provides more insight into outcome
* Use browserify to include logform functions and triple-beam symbols in client js
* Implement default log transform function in client js
* Remove formatted message and transport data from non-streaming log data sent to client
* Implement english-only scoring with wink https://github.com/winkjs/wink-sentiment (AFINN, emojis)
* Implement english-only scoring with NLP.js https://github.com/axa-group/nlp.js/blob/master/docs/v3/sentiment-analysis.md (AFINN, Senticon, Pattern)
* Refactor language processing into standalone functions for future use
* Add limited multi-langauge support
* Can run sentiment with NLP.js on english, german, spanish, and french
* Normalize all scores to range between -1 and +1
* Improve score accuracy by averaging all scores
* Add deprecation warnings to rules when building if properties should be migrated to window
* Add `debug` option to window to increase verbosity of filter logging. Default to false.
* Fix object assigment when building hash key for window filters
* Further cleanup for circular dependencies by moving some filter and logging functions into respective files
* Query for non-hydrated events to get ids then get fully hydrated objects using only ids -- dramatically improves performance
* Remove typeorm-pagination due to non-optimized count/select approach (should use typeorm getManyAndCount)
* Also removes dup typeorm dependency
* Reworked many mappings to fix cascades and relationships
* Add constructors to entities where useful
* Renamed many db entities classes to be different than interface names
* Re-generated initial migration and added convenience npm script for running it
* Added ormconfig for typeorm 3 cli
* Create all run/check/rule/action premises (static info) on manager creation
Makes CM less sensitive to random blips in reddit API by enabling snoowrap to retry some network-issue related error codes alongside status codes
* Add timeoutCodes as an operator configurable array of error codes that is passed to extended snoowrap
* Patch snoowrap to check for timeout codes on request error and use retry logic if found
* Additionally, add retryErrorCodes (status codes) to operator configuration
Using mocha, chai, and nyc
* tests for parsing string for numeric value comparison
* tests for parsing string for durations and duration comparisons
* tests for parsing reddit entity (subreddit/user) from string
* tests for parsing submission/comment id from reddit permalink string
* tests for initial config parsing/merging
Still can't get nyc to get coverage for everything in src using "all" -- causes reporting to show 0 for everything??
* Store migration state in cache instance
* Migrate on default cache init or private cache init
* Implement first migration to deal with run structure in actioned events
* Implement glob pattern or regex as argument
* Implement scan search for redis for efficiency otherwise iterate keys using generic function
* Implement cache reset based on passed item from action -- reset item crit for activities, author crit for users, and overwrite any cached activity
* Use ErrorWithCause so we can get and print a chain of error causes
* Make reddit error response in stack trace more readable by replacing them with a "translated" parent response and add them as the cause
* Properly handle error formatting for winston by looking at shape of log object for error rather than testing instanceof (see comments in errorAwareFormat)
* Fix formatting in web interface for log lines with white-space pre css and properly splitting timestamp from rest of the message
* Use interface for comparison results at both criteria property level and criteria level
* Implement summary functions to build string results of comparisons
* Output all comparisons to debug and provide summaries to verbose (when applicable)
* use node-comment and yaml@next to keep comment information intact
* store ast/source version of parsed config for operator
* implement generic yaml/json operator config classes to keep everything organized and simplify marshalling source to js/string
* refactor file parsing and json/yaml parsing to have better single responsibility
* Add typeorm dependency with backend drivers for sqljs and, optionally, postgres/mongodb/mysql/mariadb
* Add operator configuration structure for global database connection
* Implement config parsing and defaults for sqljs db location or in-memory fallback
#66
* Use 3 different matching algorithms using the highest score out of the three
* Weight score based on length of the sentence
* Increase minimum number of words to 3
* Enforce min word count on external (youtube) comments
* depended on and always downloaded an entire, older typescript version (even with production install) -- was not necessary for one function
* refactor project to use newer TS version (specify any type for catch blocks to fix compiler errors)
* Generate perceptual hashes using blockhash-js of images that can be cache/stored
* Take advantage of reddit thumbnail code ImageData to hash lower-res to begin with (but represent full url)
* Refactor imageDetection config so hash and pixel approaches have different configs
* Cache phash results to reduce reddit traffic and speed up performance
Addresses cpu/memory issues with pixel comparison. Allow pixel for finer comparisons if needed using tiered thresholds. Closes#26
* Refactor image comparison to use pixelmatch only so resemblejs can be removed (too much memory usage)
* Heavier usage of sharp to get images into same dimensions prior to pixelmatch
* Refactor image conversion into ImageData to clean up utils/recent activity rule
* Implement staggered startup for bots (reddit accounts, top-level)
* Implement staggered startup for managers (subreddits) and subreddit polling
* Introduce random -1/+1 second to polling interval for every stream to ensure none are synced so there is no instantaneous spike in cpu/traffic/memory on host/reddit
* Add user-configurable stagger interval for shared mod polling
* Implement second image comparison approach with pixelmatch for reduced memory usage when image dimensions are exactly the same
* Use sharp to resize images to 400 width max when using resemblejs to reduce memory usage
Reduce regex complexity in config by parsing a normal regex straight from config string value (including flags)
BREAKING CHANGE: regex must now be enclosed in forward slashes, flags must be on regex value, and regexFlags property has been removed
* Replace json view with monaco-editor with schema loaded based on url param
* Add route for unauthenticated config editing
* Auto-load subreddit config when "view" is clicked from subreddit view
* Use autolinker to deal with url parsing in logs and sanitize html
* Fix missing mergeArr arguments on child loggers
* Implement DuplexTransport to make streaming from winston less verbose and allow access to log object
* Refactor log parsing on server/client side to deal with log objects or strings
* Cleanup stream usage for winston and simplify adding to logger
* Use pipeline with delimiter stream for parsing logs from client
* End log streaming with promise abort (cleaner)
