957 Commits

Author SHA1 Message Date
Matt W
41c918ee87 Don't add messages when accumulated bytes exceeds threshold (#932)
* Don't add messages when accumulated bytes exceeds threshold

* Add a leniency factor

* lint
2022-11-07 12:24:49 -05:00
Matt W
1adb6d2726 Update spool to flush on size thresholds instead of batch counts (#930) 2022-11-03 14:55:51 -04:00
Matt W
8c531a256b metrics and logging cleanup (#928)
* Metrics and ambiguous log cleanup

* Fix test
2022-11-01 14:47:49 +00:00
Russell Hancox
5829363733 GUI: Fix EnableSilentMode key (#927) 2022-11-01 10:11:21 -04:00
Matt W
2082345c02 Change order that ES clients are enabled (#923) 2022-10-29 00:15:26 +00:00
Matt W
dd8f81a60e Fix issue in test that would crash on some platforms (#922) 2022-10-28 20:14:53 -04:00
Matt W
8ccb0813f1 More import fixes (#921)
* More import fixes

* lint
2022-10-28 15:57:01 -04:00
Matt W
b24e7e42bf Event metrics (#918)
* WIP. Record event count and processing time metrics. Tests don't currently build.

* Updated tests

* Fix field names

* Remove unused target

* formatting

* Cleanup from PR comments
2022-10-28 14:25:07 -04:00
Pete Markowsky
4821ebebd5 Fix: duplicates bug in SNTMetricSet when using multiple fields (#920)
Fix duplicates bug in SNTMetricSet when using multiple fields names.

This also fixes the santactl metric command and golden files for tests.
2022-10-28 13:50:08 -04:00
Matt W
efeaa82618 Fix issue with transposed remount/banned block messages (#917) 2022-10-26 20:54:17 -04:00
videlanicolas
3f3de02644 USB: usbBlockMessage is not being used. (#915) 2022-10-26 17:42:49 -04:00
Matt W
f6c9456ea7 Fix some more includes (#914) 2022-10-25 16:52:19 -04:00
Matt W
2aaff051c8 Various changes to fix import (#913) 2022-10-25 16:16:44 -04:00
Matt W
2df7e91c87 Change include to import (#912) 2022-10-24 11:56:02 -04:00
Matt W
899ca89e23 Proto minimization (#909)
* Create Light variants of File and ProcessInfo messages to reduce disk/wire byte counts

* Updated golden test data
2022-10-21 19:48:37 -04:00
Matt W
e7281f1c55 Spool writer (#908)
* Spool writer and santactl command to print proto file

* Make valid JSON for multiple paths. Can now create proto/spool logger. Updated logger tests.

* Make fsspool writer and fsspool log batch writer injectable

* Add spool writer tests

* Updated help text for santactl printlog

* Include file cleanup

* Fix dispatch source destruction

* Change config keys for the new Spool writer

* Spool settings now configurable

* Fix param order

* Remove some test sleeps related to control flow
2022-10-21 16:43:12 -04:00
Matt W
bf0ca24ae7 Machine id proto (#907)
* Add MachineID to all BasicString serialized log messages

* machine_id now a top level proto field

* Remove commented code
2022-10-19 10:51:38 -04:00
np5
4fe8b7908f sync: Fix USB blocking config sync (#890) 2022-10-18 10:01:20 -04:00
Matt W
a8dd332402 Update include paths and add include guard (#905) 2022-10-14 17:58:36 -04:00
Matt W
6631b0a8e3 More import fixes (#904)
* Layering check disable

* workaround for layering issue
2022-10-14 17:20:20 -04:00
Matt W
07e09db608 Import fixes (#902)
* Apply clang-format to cc files

* Modify binaryproto namespace

* Add more required includes

* Add proto includes

* Assert message parsing succeeds in test

* Add optional keyword to proto fields to track presence. TESTS BROKEN.

* Update golden test data
2022-10-14 15:51:53 -04:00
Matt W
d041a48c97 Fsspool adopt (#900)
* Added fsspool library, tests

* Cleanup

* Remove extra visibility from BUILD file

* Import foundation so the linter doesn't complain
2022-10-13 20:47:52 -04:00
Matt W
1683e09cc8 Proto serializer (#897)
* Initial proto serializer with close event

* Define move ctors for enriched types, delete copy ctors

* More event proto serialization. Commonized proto test code.

* Started work serializing exec event. Added serializer utilities.

* More progress serializing exec event

* Add mroe test data. Test restructure to permit fine grained mocking.

* Env/FD ES types now wrapped in EndpointSecurityAPI. Added calls to proto serializer.

* Add fd type names to proto

* Version compat. Script and Working Dir encoding.

* Add process start time

* Serialize Link event

* Add null check, mainly to fix tests

* Handle versioned expectations

* Each test now build msg in callbacks to set better expectations

* Serialize rename event and tests

* Serialize unlink event and tests

* Serialize allowlist and bundle events. Add utilities tests.

* Formatting

* Disk event proto serialization and tests

* Fix test only issues

* Rename santa_new.proto to santa.proto

* Change fd type int and string to an enum

* Proto namespace now versioned

* Added comments to proto schema

* Add proto support to indicate if fd list truncated
2022-10-13 13:52:41 -04:00
Ivan Tadeu Ferreira Antunes Filho
d6c73e0c6c common: Make SNTCommonEnums a textual header (#896)
This change fixes -wunused-variable warnings. The header is not valid by itself and should be declared as a textual header rather than as a header.
2022-10-03 13:15:33 -04:00
Matt W
72969a3c92 Fix crash flushing cache on unmount events (#895) 2022-09-27 21:54:35 -04:00
Matt W
d2dbed78dd Return a value from the test block (#894) 2022-09-27 15:07:20 -04:00
Matt W
8fa91e4ff0 Build deps (#893)
* Too bad we can't require explicit build deps...

* More deps
2022-09-23 13:55:48 -04:00
Matt W
551763146d Linter and BUILD deps fixups (#892)
* Minor changes to address lint issues

* Add more BUILD deps

* Include cleanup

* Even more BUILD deps

* Still more BUILD deps
2022-09-23 11:18:58 -04:00
Matt W
7a7f0cd5a8 Ingestion fixups (#891) 2022-09-22 12:30:34 -04:00
Matt W
fcb49701b3 ES and Logging Interfaces Redesign (#888)
* Initial structure for ES wrappers, enriched types, logging

* Basic working ES and logging functionality

* Add in oneTBB and thread-safe-lru deps

* Added a bunch of enriched types

* Auto-mute self when establishing ES client

* Basic auth, tamper client. Syslog of all events. Basic compiler tracking.

* Update copyright header blobs, convert some tabs to spaces

* Auth result cache. Fix getting translocation path.

* Added remaining cache methods

* Add AuthResultCache to Recorder client. Cache now operates on es_file_t.

* Hooked up SNTPrefixTree

* Fix CompilerController for RENAME. Fix AllowList logging missing path.

* Block loading Santa kext

* Added device manager client

* Properly log DiskAppear events

* Fix build to adopt new adhoc build

* Handle clearing cache on UNMOUNT events

* Ignore other ES clients if configured

* Remove SNTAllowlistInfo. Rename AllowList to Allowlist. Minor cleanup.

* Recorder now logs asynchronously. Enricher now returns shared_ptrs.

* Added File writer. Added timestamps to BasicStream serializer.

* Skip calling stat in SNTFileInfo when path given by ES.

* Fix build issue

* Address draft PR feedback

* santactl integrated, XPC works, fix file writer bug

* Integrate syncservice. Start observing some config changes.

* Add metrics service wrapper

* Add metrics config observers and metrics interval reset.

* Start better dependency control. Add Null logger support.

* Added more deps

* Added more deps

* Fix issue where metric service wasn't starting

* Add missing variant include

* Fix missing parent proc name

* Added googletest and new unit test macro

* Started expanding AuthResultCacheTest

* Properly mock EndpointSecurityAPI

* Finished AuthResultCacheTest

* bazelrc now builds all C++ as C++17. Added LoggerTest.

* Add FileTest. Abstract some File constants to Logger.

* Added Empty serializer test

* Started work on BasicStringTest. Fixed some BasicString serialization bugs.

* Added Unlink BasicString serialization test

* Added some more tests. Commonized some test code

* Finished BasicStringTest. Converted to XCTest.

* Standardize esapi variable naming

* Bubble up gTest expect failures to XCTest failures

* AuthResultCacheTest now uses XCTest. Added common TestUtils.h

* EmptyTest now uses XCTest.

* FileTest now uses XCTest

* LoggerTest now uses XCTest. Removed santa_unit_gtest bazel macro.

* Added ClientTest

* Add basic Enricher tests

* Add MessageTest. Make more TestUtils.

* Rename metrics to Metrics

* Add MetricsTest.

* Apply template pattern to Serializer

* Add SNTDecisionCacheTest.

* Add SNTCachedDecisionTest.

* Testing with coveralls debug mode

* Allow manual CI runs

* Remove unused property

* Started work on SNTEndpointSecurityClientTest.

* WIP SNTEndpointSecurityClientTest, fix test run issue

* Added more base ES client tests

* Add more base ES client tests

* Base ES client tests done. Added serializer utils/tests. Expanded basic string tests.

* Add utils test to test suite

* Add copy ctor. Add test output to bazel coverage.

* Single thread bazel coverage

* Updaload coverage file

* Updaload coverage file

* Old gen cov test

* Restructure message handlers to enable better testability

* Added enable tests for all ES clients

* Made a single MockEndpointSecurityAPI class to share everywhere

* Added most of SNTCompilerControllerTest

* Cleanup SNTCompilerControllerTest

* Started expanding Auth client test

* Finished up the Authorizer tests

* Move to using enum class for notify/auth instead of bool

* WIP for tamper resistance test. ASAN issues.

* Add OCMock patch to fix test issue on ARM Macs

* Changed patches directory name to external_patches

* Update WORKSPACE path

* Finished up Tamper Resistance tests

* Finished up Recorder tests.

* Move SNTExecutionControllerTest to ObjC++

* Initial work to port SNTExecutionControllerTest

* Finished porting SNTExecutionControllerTest.

* Added SNTExecutionControllerTest to list of unit tests

* Ported SNTEndpointSecurityDeviceManager.

* Test cleanup, use MockESAPI expectation helpers

* Verify SNTEndpointSecurityDeviceManager expectations differently

* Test cleanup, omit gTest param list where unused

* Log message cleanup

* Rename SNTApplicationTest to santad_test.mm

* Finished porting santad_test, formerly SNTApplicationTest

* Fix SNTEndpointSecurityDeviceManager issues

* Pulled in missed fixes. Updated tests.

* Renamed lowercase filenames to match rest of codebase

* Fix non-static dispatch_once_t, and noisy watching compiler log message

* WIP Started process of removing components no longer used

* WIP Continued process of removing components no longer used

* BUILD file cleanup. Proto warning. Removed unused global

* Rename SNTEventProvider to SNTEndpointSecurityEventHandler

* Rename SNTEndpointSecurityEventHandler protocol

* Remove EnableSysxCache option. Remove --quick flag used during dev.

* Ran testing/fix.sh

* Addmissing param to fix.sh that was omitting .mm files.

* clang-format

* Fix linter: find cmd missing .mm ext, git grep exclude patch files.

* Use MakeESProcess default params in tests

* Move variables to camelCase in objc classes

* More case changes

* Sanitize strings

* Change dispatch queue priorities and standardize daemon queue naming

* Exclude patch files in markdown check

* Ensure string log messages end with newline

* Fix BasicStringTest

* Disable clang-format in code producing different results in local/remote versions

* Moved to using date ranges in copyright notices as per current guidelines

* Update Source/common/SNTConfigurator.h

Suggestion adding whitespace in comment to fix clang-format mangling

Co-authored-by: Russell Hancox <russellhancox@users.noreply.github.com>

* Removed santa_panic macro used in one place

* Updated comment about ES cachability

* Pin oneTBB to specific commit

* Address outstanding WORKSPACE 'canonical reproducible form' messages

* Use string append instead of ostringstream due to benchmark results

* Remove use of freind classes in EnrichedTypes.h

* Added SNTKVOManager, removed observers from SNTConfigurator.

* Fixed SNTEndpointSecurityRecorderTest class name

* Reduce usage of the auto keyword

* Each SNTKVOManager instance now adds its own observer

* Replaced more auto keywords with real types.

* Remove leftover code coverage debugging from ci.yml

* Updated comment

* Memoize SNTFileInfo sha256. Reduce some cache sizes.

* Fix issue checking for translocated paths

* Use more performant NSURL creation method

* Fix lint issue

* Address PR feedback

* Use an array literal for kvo objects

* Fix some clang tidy and import issues

* Replace third party LRU cache with SantaCache for now

* Fix clang tidy issues

* Address PR feedback

* Fix comment typo

Co-authored-by: Pete Markowsky <pmarkowsky@users.noreply.github.com>

* Added todo for when we adopt macOS 13

Co-authored-by: Russell Hancox <russellhancox@users.noreply.github.com>
Co-authored-by: Pete Markowsky <pmarkowsky@users.noreply.github.com>
2022-09-22 10:18:41 -04:00
Pete Markowsky
dc6732ef04 Refactor the SNTApplicationTest unit tests to function correctly (#885)
* Refactor the SNTApplicationTest unit tests to function correctly.

The tests were originally written in a table style and were impacted by the lack of mocking the configurator. This caused issues with static rules to impact the unit tests.

Additionally added improved logging messages for critical binaries and a todo for macOS 13 unit tests.

Added goodbinary and rules.db test files to allstar's ignored paths.
2022-08-29 13:18:04 -04:00
Pete Markowsky
fd23a5c3b7 Fix up endTimestamp to be Monarch compliant (#879)
Fix up endTimestamp field to be Monarch compliant.
2022-08-16 22:32:29 -04:00
Russell Hancox
ec203e8796 Project: Rename Source/santa -> Source/gui (#877) 2022-08-12 14:19:01 -04:00
Russell Hancox
57ff69208d GUI: Missed a required dependency (#876) 2022-08-12 14:02:22 -04:00
Russell Hancox
f00b7d2ded GUI: Expose SNTNotificationManager.h for the test. (#875) 2022-08-12 13:46:25 -04:00
Russell Hancox
9791fdd53c Project: Add a GH action to prevent trailing whitespace (#873) 2022-08-12 12:46:11 -04:00
Russell Hancox
26e2203f1e GUI: Improve signing chain key reporting in distributed notifications. (#874)
Also add a group for GUI unit_tests and include in the overall project tests group.
2022-08-12 11:03:21 -04:00
Russell Hancox
4a47195d12 Santa: Post distributed notification when showing block UI (#870)
Fixes #869
2022-08-11 12:34:35 -04:00
Russell Hancox
4436e221df GUI: Add silent mode configuration option. (#871)
When enabled, this option disables *all* GUI notifications from Santa. This is intended for kiosk-style machines where it is not expected for users to _ever_ execute unknown binaries.

Fixes #862
2022-08-11 09:17:07 -04:00
Russell Hancox
deccc8a148 GUI: For App Store published apps, include team ID. (#872)
With this change, the publisher field for an App Store published app will be  instead of

Fixes #758
2022-08-11 08:15:42 -04:00
Russell Hancox
6db7fea8ae syncservice: Add tests for NSData+Zlib and Postflight (#864) 2022-07-26 13:05:35 -04:00
bfreezy
8b55ee4da5 santad: only allow root read+write permissions on sync-state.plist (#858) 2022-07-18 13:32:08 -04:00
Russell Hancox
cc3177502c Tests: Fix un-needed expectation in SNTExecutionControllerTest.allEventUpload (#857) 2022-07-15 18:03:34 -04:00
Pete Markowsky
234f81ea7c Ensure KVO works for USB config options (#853)
Ensure KVO works for USB config options.
2022-07-15 15:13:55 -04:00
Russell Hancox
743c567bf8 santad: Log team ID in execution logs, where available (#850) 2022-07-15 12:41:56 -04:00
Russell Hancox
21220f1499 santad: Add DisableUnknownEventUpload option. (#852) 2022-07-15 12:30:20 -04:00
Russell Hancox
39f3ffe8fc santactl/status: Fix printing of static rules (#848) 2022-07-15 11:53:38 -04:00
Russell Hancox
fdb01928a0 santad: Fix re-establishment of syncservice connection (#849)
* santad: Fix re-establishment of syncservice connection

The previous version could lead to santad having lots of threads stuck waiting for connections
2022-07-15 11:53:17 -04:00
Russell Hancox
fbefbc5910 santasyncservice: Keep XSRF token in memory, don't send to daemon (#851) 2022-07-15 11:52:43 -04:00
Russell Hancox
9db00d143d santad: Improve caching of static rules (#847)
In #846 I forgot that  is only a count of the entries so if the config changes but the number of rules remains the same we would never update the cache. This PR moves the processing of the raw config into the KVO handler code so it is not at all in the hot-path.
2022-07-14 10:50:30 -04:00