2 Commits

Author	SHA1	Message	Date
Azri Kahar	89fcb19a16	Expose schema endpoints (#14713) ... * Expose schema endpoints * respond with 204 * fix payload for export flag * allow export as YAML * use SchemaService * move getStringFromStream to utils * clearer exceptions with logs * check empty request body * specs * add bypassCache flag and remove flushCaches calls * Identify integer typed fields in Oracle * Fix uuid foreign keys type for Oracle * Detect autoincrement for CockroachDB * Bypass cache for fields and relations services * Add same DB tests * Test if working for all vendors * Add schema snapshot testing into sequential flow * Fix schema re-apply attempts when it's a different database vendor (#14816) * prevent diffing on certain properties * fix apply-snapshot test * prevent updates to autoincrement fields' schema * sanitize relations * remove `colleciton.schema.sql` parsing in test It is now being sanitized by `sanitizeCollection()` function in get-snapshot-diff.ts * Change concurrency group to not cancel test running on main * remove multipart for apply & add hash checking * check incoming version & add force query param * refine error message & stub for nested hash * add `vendor` top-level property as an additional safety net for cross db vendor applies * sanitize generated schema snapshots * snapshot joi validation * diff joi validation * minor cleanup * extract applyDiff & use deep-diff applyChange * use applyDiff in schema service * Mark vendor as optional * Update tests to apply diffs * move force flag check into service * Patch mssql vendor typo * Set relation no action in cyclic relations * Update mysql timestamp default value * Oracle cannot define no action * Update oracle timestamp type default values * add hash checking for outdated diffs * fix diff service & endpoint * Add hashes when returning after * Fix self referencing relations for Oracle * Add temp fix for CURRENT_TIMESTAMP defaults in Oracle * clean up driver and database client types * only require diff hash if not kind N * update hash comparison logic for create/delete * Set no action constraint for self referencing M2M in MSSQL * Add basic hash check tests * omit default value when hashing auto increments Specifically for CockroachDB with differing sequences * add vendor check * update specs * Validate vendors with type definition * Spread the vendors input array * re-add Export component * re-add js-yaml to root * Propagate mutation options for schema apply * Verify that snapshots differ and clear cache before hash tests * Fix unit test * Revert temp fix for CURRENT_TIMESTAMP defaults in Oracle * Define and reuse type for export format * Define and reuse list of database clients * change `were` to `was` * change `where` to `were` * add some empty lines for readability * tweak exception message * fix test * use object-hash * use boolean to check whether file is included * simplify request content type check * throw error when multiple files were uploaded * use nullish coalesce instead of short circuit * Update api/src/services/schema.ts Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com> * Remove unnecessary `Omit` on `SnapshotWithHash` type * Revert "Remove unnecessary `Omit` on `SnapshotWithHash` type" This reverts commit d22ac771ec. * check empty snapshot earlier * use allow-list logic via pick instead of omit * Update api/src/services/schema.ts Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch> * Move to own /schema endpoint controller * Fix refs to schema endpoint * move streamToString to utils package * move get-versioned-hash and add test * extract kind into an enum * Fix mysql5 timestamp value * Fix test collection existing on rerun * resolve TODO comment in blackbox test * Drop deep level hashes in diff These hashes are used only for more accurate error reporting but are otherwise superfluous, since changes are already detected by the top level hash. Therefore we remove them in favor of a simpler diff format and implementation. * Revert schema "fix" for createItem, add comment * Strict diff schema validation * Revert CrDB auto-increment detection patch in 816c998 * Clear systemCache to expose newly added fields * Use DiffKind constants * Extract diff & snapshot validation into own tested utils * Apply suggestions from @azrikahar * Update knex-schema-inspector to 3.0.1 Includes the fix for CrDB auto-increment detection (knex/knex-schema-inspector#135) * Update knex-schema-inspector in packages * Update lock file * add test for schema service * add test for export service * add relevant tests to modified util functions * fix csv test to account for os end of line * fix files controller test * dedupe test data for schema service * Align schema specs with docs * Update api/src/controllers/schema.ts * Revert testing for all vendors --------- Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com> Co-authored-by: ian <licitdev@gmail.com> Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>	2023-01-31 17:36:49 +00:00
Rijk van Zanten	00865fbd84	File Storage 2.0 (#16825) ... * Refactor storage logic to testable setup * Add tests to get storage config * Add tests for register drivers * Tests for index * Organize imports * Add missing getStorage * Setup boilerplate for cloudinary drive * Add build script * Add cloudinary configuration * WIP tweaks for R&D * Start storage abstraction v2 * Make storage manager single file * Add test coverage * Organize imports * Setup local driver boilerplate * [WIP] Start on local driver 2.0 * Add more methods * Lunchtime * Add put method * Add list method * [WIP] Try using storage in api * Use node16 module-resolution * Lets not mess with apply-query rn * I love CJS. Death to CJS. * Use dynamic imports * Make things work * Add path normalizer * Add azure storage driver * Update lock * Start on tests * Add getBuffer test * Add getStat tests * Add tests for exists * Add tests for move * Add tests for copy * Add tests for put * Add tests for delete * Add test coverage for list * Add removeLeading option to normalizePath * Use removeLeading option * Start on gcs * Add fullpath test * Add getStream * Add getBuffer * Add getStat * Add exists * Add move * Add copy * Add put * Add delete * Finish DriverGCS * Cleanup tests a bit * Start s3 * Add getStream * Add getBuffer * Please Wryn * Add #exists * Use randomize data * No more hardcoded values 🙅‍♀️ * Add tests for copy * Add tests for put * Add put/copy/delete * Add tests for delete * WIP list * Remove duplicate fullPath check * Finish AWS tests * Listen to wryn * Mission critical tweak * Add randomization, cleanup tests * Check src vs dest full path * Start on Cloudinary * Add parameter signature helper * Fix ESM building of shared * Fix ESM building of shared * Improve tests * Update pnpm * Remove old build commands * Generated d.ts files * Fix export naming * Move ESM only utils to new @directus/utils * Update lockfile * Fix type exports * Implement getStream * Cleanup tests * Simplify api * Simplify API * Simplify api * Simplify API * Add read/stat * Cleanup / add exists * Add move * Add write * Move uploadChunk to separate method * Add test for #uploadChunk * Add tests for write * Add copy * Add delete * Add list * Add list error handling * Remove old drive packages * Start updating API usage of storage * Use Readable instead of NodeJS.ReadableStream * Use readable instead of buffer * Restore shared to main * Update lockfile * Use a streaming pipeline for sharp metadata * Add basic e2e test for local upload and delete * Fix integer filesize in SQLite * fixed environment mocking in unit tests * applied the same env mocking to other tests * Update api/src/storage/register-drivers.ts Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch> * Use sha256 by default * Add base read test for /assets * Replace exifr with exif-reader * Fix tests for storage/index * Install faking lib for tests * Add test for register-drivers * Add tests for register-locations * Finish tests * Organize imports * Resolve Azris comments * Fix outdated tests Certainly not the cleanest, but I need to do a bigger pass on all these tests to get them up to date with the latest team requirements. Gonna do that in a separate PR.. * Test for sha256 * Attempt 1 at fixing toString errorr I'm not seeing this issue locally, so we'll spam a couple commits here to get things going * Use node 18 in tests?! * Fix localhost resolution with 127.0.0.1 * Mock getEnv() * Use @directus/tsconfig instead of duplicated conf * Does this fix it? * OK fun detour * Recreate lockfile * Update config files * Use multipart uploads in S3 * Cleanup imports * File Storage 2.0: Make metadata extraction backward-compatible (#16868) * Reinstall packages using pnpm instead of manually removing them (#16871) * Added extra environment setting for sharp processing of invalid images (#16811) * Added extra environment setting for sharp processing of invalid images * renamed environment var to `ASSETS_INVALID_IMAGE_SENSITIVITY_LEVEL` * Remove unused excludes from tsconfig * Remove copy/paste leftover * Update packages/utils/readme.md Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch> * Update packages/utils/package.json Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch> Co-authored-by: ian <licitdev@gmail.com> Co-authored-by: Brainslug <tim@brainslug.nl> Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch> Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>	2022-12-21 10:04:03 -05:00