21 Commits

Author SHA1 Message Date
ian
802dfa0703 Fix inner query sort limit (#16922)
* Add sort within inner query

* Run tests for all vendors

* Apply limit to all queries

* Enable sorting in MSSQL subquery by setting a limit

* Add tests

* Parse function field names for a2o

* Improve order string spacing

* Update unit test

* Fix mysql5 tests

* Revert testing for all vendors

* Extract original collection name

* Add comment for MSSQL applyLimit helper

Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>
2023-01-04 19:43:17 +08:00
ian
f1a8e0446f Fix duplicated results and functions in nested filters (#14798)
* Speed query up by reusing existing aliases which reduces table joins

* Use subquery in top level m2o to remove duplicates

* Fix linting

* Apply distinct on primary key field in subqueries

* Use distinct instead as there are only primary keys

* Apply subquery on top level

* Try remove sub sub query

* Test if working for all vendors

* Add support for _none and _some

* Use subquery only when field depth > 1

* Add tests

* Use original table names for columns with functions (#14690)

* Use original table names for columns with functions

* Extract filter function path parsing as shared util

* Fix filter function path when adding node

* Pass the originalCollectionName into filter functions

* Update unit test

* Replace functions within deep GraphQL

* Fix invalid operator error for _none and _some

* Add filter function tests

* Revert triggering for all vendors

* Simplify aliasMap

* Replace functions in filter within GraphQL aggregate query

* Add API support for filtering of alias field

* Mark schema as optional

* Shift logical operators upwards

* Separate recursive parseFilter

* Rework shifting of logical operators

* Error on invalid usage of _none and _some

* Use inner join to preserve sort order

* Run tests for all vendors

* Reuse aliasMap for sort and filter

* Sort on top level query

* Remove unnecessary limit on wrapper query

* Refactor applyQuery options

* Remove duplicates from nested multi relational sort

* Fix offset in MSSQL requiring OrderBy

* Disable schema cache

* Use inner query only for nested sort or multi relational filter

* Fix MSSQL duplicate order column

* Use inner query only for multi relational

* Additional integration tests

* Order within partition for multi relational sorts

* Rename to directus_row_number

* Fix unit test

* Add base sort and filter tests

* Fix Oracle uppercased rowNumber column

* Fix unit test

* Fix top level query sort with function

* Parse functions in inner query

* Increase clarity with knex.ref()

* Remove sort filter for top level primary key

* Fix unit test

* Bypass queries with groupBy

* Add collection to aliasMap to fix functions in nested sort

* Fix multi relational sort with functions

* Add tests for filter and sort with functions

* Fix accidental deletion of brackets

* Fix top level alias filter node interface

* Update M2M sort tests

* Add M2A tests

* Cast m2a primary key as varchar2 for oracle

* Enable filtering tests for M2A

* Fix prototype polluting assignment in aliasMap

* Remove unnecessary currentKey

* Simplify code to increase readability

Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>

* Fix linting and missing 'this' error

* Revert optional chaining

* Add mysql5 to tests

* Fix mysql5 missing rowNumber()

* Overcome indexing delays in MySQL5

* Verify MySQL5 sorting is in order as the result count varies between runs

* Skip joining when sorting field already exists

* Simplify variable assignment

Co-authored-by: Azri Kahar <42867097+azrikahar@users.noreply.github.com>

* Fix linting

* Reduce duplicate logic with vars

* Transform _func fields in GraphQL only for valid functions

* Fix unit test

* Fix unsupported date_part() in CrDB

Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>
Co-authored-by: Roger Stringer <roger@directus.io>
Co-authored-by: Azri Kahar <42867097+azrikahar@users.noreply.github.com>
Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>
2022-12-21 11:56:18 -05:00
Rijk van Zanten
00865fbd84 File Storage 2.0 (#16825)
* Refactor storage logic to testable setup

* Add tests to get storage config

* Add tests for register drivers

* Tests for index

* Organize imports

* Add missing getStorage

* Setup boilerplate for cloudinary drive

* Add build script

* Add cloudinary configuration

* WIP tweaks for R&D

* Start storage abstraction v2

* Make storage manager single file

* Add test coverage

* Organize imports

* Setup local driver boilerplate

* [WIP] Start on local driver 2.0

* Add more methods

* Lunchtime

* Add put method

* Add list method

* [WIP] Try using storage in api

* Use node16 module-resolution

* Lets not mess with apply-query rn

* I love CJS. Death to CJS.

* Use dynamic imports

* Make things work

* Add path normalizer

* Add azure storage driver

* Update lock

* Start on tests

* Add getBuffer test

* Add getStat tests

* Add tests for exists

* Add tests for move

* Add tests for copy

* Add tests for put

* Add tests for delete

* Add test coverage for list

* Add removeLeading option to normalizePath

* Use removeLeading option

* Start on gcs

* Add fullpath test

* Add getStream

* Add getBuffer

* Add getStat

* Add exists

* Add move

* Add copy

* Add put

* Add delete

* Finish DriverGCS

* Cleanup tests a bit

* Start s3

* Add getStream

* Add getBuffer

* Please Wryn

* Add #exists

* Use randomize data

* No more hardcoded values 🙅‍♀️

* Add tests for copy

* Add tests for put

* Add put/copy/delete

* Add tests for delete

* WIP list

* Remove duplicate fullPath check

* Finish AWS tests

* Listen to wryn

* Mission critical tweak

* Add randomization, cleanup tests

* Check src vs dest full path

* Start on Cloudinary

* Add parameter signature helper

* Fix ESM building of shared

* Fix ESM building of shared

* Improve tests

* Update pnpm

* Remove old build commands

* Generated d.ts files

* Fix export naming

* Move ESM only utils to new @directus/utils

* Update lockfile

* Fix type exports

* Implement getStream

* Cleanup tests

* Simplify api

* Simplify API

* Simplify api

* Simplify API

* Add read/stat

* Cleanup / add exists

* Add move

* Add write

* Move uploadChunk to separate method

* Add test for #uploadChunk

* Add tests for write

* Add copy

* Add delete

* Add list

* Add list error handling

* Remove old drive packages

* Start updating API usage of storage

* Use Readable instead of NodeJS.ReadableStream

* Use readable instead of buffer

* Restore shared to main

* Update lockfile

* Use a streaming pipeline for sharp metadata

* Add basic e2e test for local upload and delete

* Fix integer filesize in SQLite

* fixed environment mocking in unit tests

* applied the same env mocking to other tests

* Update api/src/storage/register-drivers.ts

Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>

* Use sha256 by default

* Add base read test for /assets

* Replace exifr with exif-reader

* Fix tests for storage/index

* Install faking lib for tests

* Add test for register-drivers

* Add tests for register-locations

* Finish tests

* Organize imports

* Resolve Azris comments

* Fix outdated tests

Certainly not the cleanest, but I need to do a bigger pass on all these tests to get them up to date with the latest team requirements. Gonna do that in a separate PR..

* Test for sha256

* Attempt 1 at fixing toString errorr

I'm not seeing this issue locally, so we'll spam a couple commits here to get things going

* Use node 18 in tests?!

* Fix localhost resolution with 127.0.0.1

* Mock getEnv()

* Use @directus/tsconfig instead of duplicated conf

* Does this fix it?

* OK fun detour

* Recreate lockfile

* Update config files

* Use multipart uploads in S3

* Cleanup imports

* File Storage 2.0: Make metadata extraction backward-compatible (#16868)

* Reinstall packages using pnpm instead of manually removing them (#16871)

* Added extra environment setting for sharp processing of invalid images (#16811)

* Added extra environment setting for sharp processing of invalid images

* renamed environment var to `ASSETS_INVALID_IMAGE_SENSITIVITY_LEVEL`

* Remove unused excludes from tsconfig

* Remove copy/paste leftover

* Update packages/utils/readme.md

Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>

* Update packages/utils/package.json

Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>

Co-authored-by: ian <licitdev@gmail.com>
Co-authored-by: Brainslug <tim@brainslug.nl>
Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>
Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>
2022-12-21 10:04:03 -05:00
Pascal Jufer
47db8e5d2f Clean-up dependencies (#16481)
* Clean-up api

- Remove unnecessary deps
- rimraf -> del-cli

* Clean-up app

- bytes -> pretty-bytes
- Remove ms as we already have pretty-ms
- Remove other unnecessary deps
- Add storybook build to gitignore

* Clean-up drive packages

- Simplify cleanup scripts
- rimraf -> del-cli
- Remove unnecessary deps

* Clean-up schema

- Remove unnecessary dep (lodash)
- Replace 'npm-watch' with native watch mode

* Clean-up shared

- Remove unnecessary dep (c8, adding @vitest/coverage-c8 will be addressed in
  other PR)
- rimraf -> del-cli
- npm-run-all -> concurrently

* Clean-up root

- Add missing eslint-plugin-jest dep
- listr -> listr2

* Make build output a bit friendlier

* Remove cleanup scripts

Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>
2022-11-18 17:02:41 +00:00
ian
addb077d48 Emit action events with updated schema (#15576)
* Emit action events with updated schema

* Fix collections service consistency

* Pass action event upstream for createMany item

* Pass action event upstream when processing payload

* Skip unnecessary retrival of updated schema

* Fix folder tests

* Fields service consistency

* Prevent duplicate cache clearing when deleting fields

* Add tests_extensions_log table

* Update collections crud tests

* Add fields crud tests

* Add action verify schema tests

* Bypass system cache purge in relations service

* Bypass system cache purge in nested field deletion

* Update source for consistency when creating from collection
2022-11-15 10:02:45 -05:00
Pascal Jufer
20a06a6c0e Clean-up lint / format set-up (#16380)
* Format json & all yaml files and update ignore files

* Fix resulting formatting issues
2022-11-09 07:26:54 -08:00
ian
adff255d91 Skip checking of virtual alias fields (#16320)
* Default virtual fields as alias type

* Make the condition more verbose instead of defaulting

* Remove unnecessary optional chaining

* Add tests

* Revert to alias defaulting in #d06d62

* Shift condition up
2022-11-08 15:49:49 -08:00
Roger Stringer
53a3194d5c Add support for SAML Part 2 (#16145)
* new saml branch

* put saml info back in

* put saml info back in

* clean up code

* validate saml config

* validate schema

* Add saml auth flow tests

* use RelayState for redirects

* Update tests for RelayState

* Fix linting

* remove validateMeta as samlify does it already

* Fix linting

* change catch on login

* Update api/src/auth/drivers/saml.ts

Co-authored-by: Aiden Foxx <aiden.foxx.mail@gmail.com>

* remove login since not needed here

* clear cookie if set on logout

* empty login method

* invalidate logout in db

* if relayState and login failed, redirect back with a reason

* Cleanup linter warnings

* Remove range from packages

* Opinions opinions opinions opinions

Just a couple personal opinion cleanup pieces

Co-authored-by: ian <licitdev@gmail.com>
Co-authored-by: Aiden Foxx <aiden.foxx.mail@gmail.com>
Co-authored-by: rijkvanzanten <rijkvanzanten@me.com>
2022-11-01 22:09:31 +00:00
Azri Kahar
17c0ef9182 Remove Content-Type header check in Files POST requests (#16176)
* remove request header check in files POST

* basic test

* it's filename_download instead of file_download

* unused vars begone

* make type required when creating file via JSON

* fix blackbox test

Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>
2022-11-01 12:06:09 -04:00
ian
83b6027cc3 Run blackbox tests in parallel (#16048)
* Run blackbox tests in parallel

* Add slight delay for updates in timezone tests

* Revert trigger on PR
2022-10-19 10:04:32 -04:00
ian
0f6ed82953 Optimise data model sorting (#16039)
* Add collections batch update endpoint

* Use collections batch update when sorting in data model

* Add autoPurgeSystemCache option

* Change type to false for consistency

* Refactor to InvalidPayloadException

* Add collection batch update test

* Bump default max payload size to 1mb

Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>
2022-10-18 19:19:36 +00:00
ian
1ef17c2ed2 Disable foreign check outside the trx in SQLite (#15403)
* Disable foreign check outside the trx in SQLite

* Update test

* Test if working for all vendors

* Fix sequential tests flow

* Revert triggering for all vendors
2022-09-05 11:31:36 -04:00
ian
cb914746d2 Add singleton tests (#15402)
* Add singleton tests

* Test if working for all vendors

* Revert triggering for all vendors
2022-09-05 09:58:59 -04:00
ian
a520b66c50 Add M2M base tests (#15093)
* Add M2M base tests

* Test if working for all vendors

* Revert triggering for all vendors
2022-08-16 14:47:58 -04:00
ian
75ef3e94a4 Skip seeding of excluded tests when running "only" in blackbox (#15083) 2022-08-16 11:38:26 -04:00
ian
1514b0c204 Enable schema caching for blackbox tests (#15070)
* Enable schema caching for blackbox tests

* Test if working for all vendors

* Undo triggering of all vendors
2022-08-15 09:56:41 -04:00
ian
3586409a0b Only allow null filter operators for fields with "conceal" special (#15010)
* Only allow null filter operators for fields with "conceal" special

* Add tests

* Use GraphQLHash type for fields with "conceal" special

Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>
2022-08-10 11:19:04 -04:00
ian
3dedf1cb3f Check for allowed filter operators when applying filter (#14829)
* Check for allowed filter operators when applying filter

* Add hash type to GraphQL

* Add tests

* Specify text type
2022-08-03 14:14:19 -04:00
ian
9f9ef45846 Disable foreign check on SQLite when deleting fields (#14512)
* Disable foreign check on SQLite when deleting fields

* Add default on_delete constraint

* Add test

* Rename methods

* Fix test sequence
2022-07-21 14:44:05 -04:00
Rijk van Zanten
27cc3e1bec Move repo to pnpm based workflow (#14350)
* Use pnpm instead of npm

* Setup workspace

* Replace scripts / lerna root stuff

* Add todo file

* Update lock, start updating packages

* Update todo

* [WIP] Keep going on module resolution in pnpm

* Fix final missing deps/types for complete build

* [WIP] Replace npm commands with pnpm alternatives

* Update jest config for v28

* Fix missing package under schema

* Update workflow

* Run CI on fork

* Use local reference

* Remove version from local workflow

* Add build to prepare

* Add Node's max old space size flag to linter

* Idemdito for codeql

* Hyphens? Underscores?

* Bang

* Only build one thing at a time

* Underscores after all?

* Match 7gb of GH

* Set env on prepare

* Jest is driving me nuts

* Downgrade jest

* Add root jest

* We'll get there, eventually.

* 🥳

* Het houdt niet op, niet vanzelf

* attempt to fix e2e

* fix unit test & shared import

* add `debug: true` to CodeQL action

* Fix dev call in shared

* Add missing cross-env

* Add missing geo-json dependency for app

* add flag to prepare action to skip build

* remove debug flag from codeql action

* fix invalid env syntax

* add tslib to app dependencies for tests

* another attempt

* Added missing rimraf dependency for the api prebuild/cleanup scripts

* Added missing ts-node dependency for the api cli script

* update E2E tests readme to use pnpm

* Undo move of extensions sdk

* Fixes nested groups in accordions not rendering fields (#14369)

* initial fix for nested groups in accordions

* removed debug code

* Fix strict relative dates showing "incorrect" (#14390)

* Set rounding method of relative formatted date to floor when strict option is selected

* Add round fn as an option

Co-authored-by: rijkvanzanten <rijkvanzanten@me.com>

* Don't use locales in generated camelCased values from env (#14401)

Ref https://github.com/directus/directus/discussions/14122

* Fix list panel descending sort (#14396)

* fix list panel descending sort

* Handle missing or explicit desc value

Co-authored-by: rijkvanzanten <rijkvanzanten@me.com>

* Fix query primary field for system tables (#14402)

* New Crowdin updates (#14403)

* Update source file en-US.yaml

* New translations en-US.yaml (Spanish, Chile)

* New translations en-US.yaml (Spanish, Latin America)

* New translations en-US.yaml (Greek)

* New translations en-US.yaml (Italian)

* New translations en-US.yaml (French, Canada)

* Fix/wysiwyg context menu (#14404)

* New translations en-US.yaml (Polish) (#14407)

* Encoded the url using encodeURIComponent, so that the url gets sanitzed and so, we did not get 404 error. (#14418)

* Encoded the url using encodeURIComponent, so that the url gets sanitized and we did not get 404 error.

* Added required changes as stated by the reviewer

* Used encodeURI instead of encodeURIComponent to encode the params only

* Solved linting error

* Solved linting errors

* checked with npm run lint, no linting errors found

* fix generateJoi error due to empty permissions when creating new role (#14416)

* fix empty permissions when creating new role

* basic test

* move logic up

* additional test

* Update api/tests/utils/filter-items.test.ts

Co-authored-by: ian <licitdev@gmail.com>

Co-authored-by: ian <licitdev@gmail.com>

* Recreate lockfile

* Fix build of extensions-sdk

* Add missing dependency

* Add some more missing deps

* Update blackbox to use pnpm

* Update workflow to use main

* Update pack to work with pnpm

* Simplify commands

* Remove todo file (completed)

* ADd missing types dep

* Use local test version

* Remove version from local reference

* Call super with context

* Add missing dep

* Simplify workflows

Ref https://github.com/directus/organization/issues/135

* Simplify some more

* Linter is on root

Co-authored-by: Azri Kahar <42867097+azrikahar@users.noreply.github.com>
Co-authored-by: brainslug <tim@brainslug.nl>
Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>
Co-authored-by: Gerard Lamusse <gerardlamo@gmail.com>
Co-authored-by: José Varela <joselcvarela@gmail.com>
Co-authored-by: ian <licitdev@gmail.com>
Co-authored-by: Zeel Pathak <60271095+zeel-pathak@users.noreply.github.com>
2022-07-15 17:41:49 -04:00
ian
8d1966ab04 Blackbox testing (#13200)
* Add black box tests

* Revert docker compose file

* Update workflow

* Try use workflow from dev repo

* Increase seedDB() timeout

* Disable other checks for now

* Change DB sequence

* Update jest moduleNameMapper

* Update workflow's docker-compose.yml path

* Slice array first

* Remove differentiation of status code

* Delete field only after foreign key constraints are removed

* Add checks for different types of primary key

* Test global query filter for all field types

* Increase timeout for m2o seeding

* Add case insensitive string operators

* Update filter check to run on relational fields

* Enable time field checks

* Add seeded random and fix relational seeding

* Add casting for integer and bigInteger

* Minor fixes

* Reduce bigInt values

* Separate seeding of DB structure from values

* Add primaryKey seeding function

* Use automatic IDs except for string pk

* Try fix ci

* Update package-lock.json

* Update common.test for concealed user tokens

* Use dynamic field type for m2o.test relational fields

* Temporary disable missing nicontains for string type

* Add support for alias type filtering

* Fix relational filter operator checks

* Add initial o2m test

* Remove integer pk limit

* Add empty checks for string and uuid null

* Limit generated integer value to 4 bytes

* Patch timezone tests for MSSQL

* Remove sample query filter test

* Fix timezone test for sqlite

* Fix MSSQL uuids

* Fix MSSQL timestamp inaccuracy

* Cast datetime schema to milliseconds for comparison

* Fix MySQL / Maria timestamp inaccuracy

* Fix MySQL / Maria between operator inconsistency for float type

* Fix missing time datatype in Oracle

* Skip filter testing on Oracle

* Enable o2m filter tests for other collections

* Run tests only on SQLite for PRs unless the Full Tests label exists

* Try fix actions

* Refactor github actions

* Update tests flow setup to use getURL()

* Start postgres docker

* Reinstate package-lock

* Fix geometry test

* Remove .gitkeep files

* Add todo.md

* Rename black box to blackbox

Co-authored-by: rijkvanzanten <rijkvanzanten@me.com>
2022-07-15 15:25:32 +00:00