Commit Graph

23 Commits

Author SHA1 Message Date
Nicola Krumschmidt
1aaf3fc218 Add support for filtering using a json object as target to durus (#21455)
* Add support for nested object target to abstract query

Co-authored-by: Jan Arends <jaads@users.noreply.github.com>

* Implement nested object target conversion

* added parameters in tests

* tiny refactoring

* test for json filter

* test and fix

* json value as filter target

* fixed mixed props and column name

* support for deeply nested filtering

* casting for filtering against numbers within a json

* another test

* linter fix

* split up set condition nodes

Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>

* added json target support for geo and set condition

Co-authored-by: Nicola Krumschmidt <nicola.krumschmidt@freenet.de>

* linter fixes

* reverted json target for geo condition

* test for comparing field with primitive

* removed geo condition test

* Fix type issue

* Remove double cast to integer

* Implement stringification of order by with json

* Fix spelling error

* Clean up field condition node stringification

* Cast numbers to numeric type instead of integer

* Implement stringification of json geo condition

* Flip around string condition type check

* Refactor number condition stringification

* Refactor set number condition stringification

* Refactor set string condition stringification

* Rename comparisonValue to compareValue for consistency

* test for geo condition with json target

* utility for target conversion

* Convert json target in order by node as object

* test for json target in order by

---------

Co-authored-by: Jan Arends <jaads@users.noreply.github.com>
Co-authored-by: Jan Arends <jan.arends@mailbox.org>
Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>
2024-02-28 14:40:11 +01:00
Jan Arends
673e516d5b Durus: Selection of properties from a object stored in a JSON field (#21158)
* defined json node

* query json field test

* activated json node in ASQL

* renamed test

* replaced dedicated node with existing 'nested' node

* added abstract sql json node

* test for conversion

* another jest test

* added test for picking an attribute which is an array

* randomized values in test

* converted from tree to path list

Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>

* added a column index per json path

* skip TDD tests for now

* adjusted abstract sql json type

Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>
Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>

* overall field conversion test for json

Co-authored-by: Nicola Krumschmidt <nicola.krumschmidt@freenet.de>

* one node per json path

* removed empty lines

* fixed linter

* integrated json node conversion

* integrated json conversion

* renamed variable

* enhanced test and fixed bug

Co-authored-by: Brainslug <tim@brainslug.nl>

* randomized values for test

* fixed func call

* test for json select in driver

* simple json support in driver

* moved json path into the list of parameters

* used operator in pg driver to return an object by key

* fixed tests

* fixed circular dep

* finalized test in driver

* renamed node

* renamed interface type

* Simplify json path accumulation

Co-authored-by: Jan Arends <jaads@users.noreply.github.com>

* Move object type to its own file

* Rename object type name for consistency

* Improve variable naming in test

* Fix formatting

---------

Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>
Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>
Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>
Co-authored-by: Nicola Krumschmidt <nicola.krumschmidt@freenet.de>
Co-authored-by: Brainslug <tim@brainslug.nl>
Co-authored-by: Jan Arends <jaads@users.noreply.github.com>
2024-02-14 23:30:28 +01:00
Nicola Krumschmidt
1acbe2535f Make relational node names less confusing (#21213)
* Change relational-many node type string to relational-single

* Rename relational-single type to align with type string

* Change relational-any node type string to relational-union

* Rename relational-union type to align with type string
2024-01-29 12:43:45 +01:00
Jan Arends
578cbb0146 Durus refactoring: New directory for node conversions (#21168)
* used existing type for definition

* created directory to convert the actual nodes
2024-01-23 12:01:38 +01:00
Nicola Krumschmidt
b230d0695c Use table and column indices instead of random unique aliases (#20843)
* Make "as" required on abstract SQL select nodes

* Adjust driver to always expect as

* Replace parameter index generator with general purpose one

* Use table and column indices instead of unique aliases

* Fix and refactor data-sql tests

* Adjust postgres driver to use table and column indices

* adopted query unit tests

* adopted index file tests in driver

* adopted utility tests in driver

* deleted unused variables

* moved indexing logic into converter specific utils folder

* tiny reference fix

* removed mock

* Remove mock variable from orderBy test

* Move index to identifier functions to data-sql

* Rename columnIndexToName in tests and parameters

---------

Co-authored-by: Jan Arends <jan.arends@mailbox.org>
2024-01-12 18:30:32 +00:00
Pascal Jufer
81c9156508 Update Vitest to v1 (#20694) 2023-12-10 19:49:45 +01:00
Nicola Krumschmidt
e57312a2f2 Add select node to abstract SQL that combines primitive and fn (#20654) 2023-12-06 17:29:00 +01:00
Nicola Krumschmidt
4f2a90c345 Split up abstract sql in root and sub query (#20638)
* Split up abstract sql in root and sub query

* Finish new root/sub query abstract sql mechanism

Co-authored-by: Jan Arends <jaads@users.noreply.github.com>

* playground

* Fix getMappedQueriesStream by closing controller

Co-authored-by: Jan Arends <jaads@users.noreply.github.com>

* Add double nested o2m example to playground

* adopted test to create o2m sub queries

* adopted more tests

* more test fixes

* removed playground again

* Fix tests

Co-authored-by: Jan Arends <jaads@users.noreply.github.com>

* Remove orm dir and move contents to utils

* Add tests for mapResult

* Remove orm export

---------

Co-authored-by: Jan Arends <jaads@users.noreply.github.com>
Co-authored-by: Jan Arends <jan.arends@mailbox.org>
2023-12-06 15:07:11 +00:00
Jan Arends
08bbe27794 Durus: nested type refactoring (#20553)
* initial thoughts

* optional meta data for a2o with json

* added test

* some docs

* Adjust abstract query to support single column any relations

* fix build

* added relational any option to nested one

* applied type refactoring in converter

* improved a2o relation definition

* no pk column name in any json column

* added place for a2o sub query generator

* docs on type

* deleted test

Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>

* docs

* reverted preps

* renamed meta to nesting

* removed reverted prep

* removed unnecessary check

* docs

* renamed meta

Co-authored-by: Nicola Krumschmidt <nicola.krumschmidt@freenet.de>

---------

Co-authored-by: Nicola Krumschmidt <nicola.krumschmidt@freenet.de>
Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>
2023-12-04 18:18:55 +00:00
Pascal Jufer
ba7fac402e Update Prettier v3.1.0 (#20481) 2023-11-20 16:23:22 +01:00
Nicola Krumschmidt
d9bd5adeb1 Refactor the file structure in data-sql (#20433)
* Refactor data-sql types file structure

* used types dir directly

* target conversion outside of conditions, file for filter conversion result

* type import only via index file

* removed unused index files

* Refactor data-sql types file structure

* used types dir directly

* target conversion outside of conditions, file for filter conversion result

* type import only via index file

* removed unused index files

* fixed tests

* rename

---------

Co-authored-by: Jan Arends <jan.arends@mailbox.org>
2023-11-17 20:41:25 +01:00
Jan Arends
a920cd3978 Durus: Modifiers on o2m (#20411)
* docs

* added todo test

* refactoring

* refactoring

* convert filter

* conversion of all modifiers

* added more modifiers to test

* fixed tests

* test for various things

* Fix joins in nested many generation

---------

Co-authored-by: Nicola Krumschmidt <nicola.krumschmidt@freenet.de>
2023-11-16 10:01:23 +01:00
Nicola Krumschmidt
d5861365e9 Make modifiers and aliases on abstract queries required (#20393)
* Make modifiers on abstract queries required

Since the properties on the modifiers object are already optional, we can explicitly require the modifiers property

* Require at least one element if sort modifier is present

* Make aliases on abstract queries required

* Remove alias from abstract sql

* Always use aliases when generating alias mapping
2023-11-15 12:02:32 +01:00
Jan Arends
5c7c79dfc2 Durus sorting on an m2o field (#20357)
* Changed sort conversion to support nested targets

* Refactoring in sort test

* integrated new sort conversion

* formatting

* removed type errors

* test for sort in modifier conversion

* remove unneeded stuff from driver

* fixed tests

* removed unused type

* support functions for sort

* Remove not needed error handling

* Allow sorting by functions in query converter

---------

Co-authored-by: Nicola Krumschmidt <nicola.krumschmidt@freenet.de>
2023-11-10 19:48:28 +00:00
Jan Arends
6f0205d054 Durus: Filtering on m2o relations (#20222)
* initial ideas

Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>
Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>
Co-authored-by: ian <licitdev@users.noreply.github.com>

* removed function as target again

* converter test like the rest

* TDD test for converter

* TDD test for pg statement creation

* fix

* convert nested condition target

* test for function target conversion

* made condition nodes return join clause for nested filter

* handled the additional join for logical nodes

* adopted changes in filter test

* fixed test

* added join to modifiers

* Allow same target in all conditions

* Fix AbstractSqlQueryOrderNode type

* Fix deeply nested filter conversion

* Fix condition converters with nested target

* Make filter result clauses required

* Fix merging joins

* Fix convertLogical type

* Allow all SQL condition nodes to be functions

* fixed test

* rename

* new test, separated integration tests

* remove duplicated joins

* formatter

* fixed string condition

* adopted tests

* preparation for sort

* removed test complexity drastically

* clean up join duplicate removal

* unit test for modifier conversion

* reverted converter test

* removed join duplication removal

---------

Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>
Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>
Co-authored-by: ian <licitdev@users.noreply.github.com>
Co-authored-by: Nicola Krumschmidt <nicola.krumschmidt@freenet.de>
2023-11-10 18:40:47 +01:00
Jan Arends
b2fe81b596 Durus o2m field selects for PostgreSQL (#20037)
* test to split up root query into sub queries

* moved splitter to driver

* moved splitter to data-sql

* filtered o2m within conversion

* removed splitter

* fix

* fix

* util to get root query

* refactoring, preparation for root and sub queries

* preparations for making separate sub queries

* refactoring

* implemented creation of sub query

* made store in relational metadata mandatory

* fixed export

* abstracted the database request

* get root query definition

* preparation to merge streams

* started converting nested many queries in converter

Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>

* refactoring

* at least two childnodes for logical operator

* refactoring missing module

* fixed tests for fields

* refactoring, mainly rename current to internal

* start integrating second approach

* getting closer but wip

* getting closer

* fixed naming

* merged streams into final stream

Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>

* fixed returning array

Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>

* added generic type

* use existing many join node again

* fixed test

* made use of generic type

* changed wording for relational stuff

* composite keys

* new generator for sub query parameters

Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>

* worked on index test

* more specific return type for driver

* fixed index file tests

* removed approach on abstract query level

* worked on index test but wip

* simple test for nested many node

* continued on index test

* finally finished test

* randomized test values again

* moved common nested many logic up to data-sql

* remove closing controller

* typedocs and error handling

* single test for nested many querying logic

* refactor: added utility function for streams

* Remove unneeded type declaration

* Remove bind call

* Use AtLeastOneElement type alias

* Use unknown instead of any as value of Records

* Rename loadAllResultIntoMemory to readToEnd

* Fix o2m with compound keys

* Fix type error in createJoin

* Simplify AbstractSqlNestedMany type

* catch error of conversion in data-sql

* removed additional error handling in driver

* replaced table alias with actual table name

* Allow join field values to be numbers again

* Clarify property names on AbstractSqlNestedMany some more

* nested many node creation tests

* docs

---------

Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>
Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>
Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>
Co-authored-by: Nicola Krumschmidt <nicola.krumschmidt@freenet.de>
2023-10-27 12:19:46 -04:00
Nicola Krumschmidt
2fed2670c8 Remove root property from AbstractQuery and minor cleanup (#19992)
* Move fields file out of fields folder

* Remove root property from AbstractQuery
2023-10-11 00:00:26 +02:00
Nicola Krumschmidt
6c45914b8a Split up related field node type in data abstraction layer (#19942)
* Fix casing in file names

* Refactor file structure

* Split up related field node

* formatter

* renamed fields-node.ts to fields.ts

* Small grammar fix in comment

Co-authored-by: Jan Arends <jan.arends@mailbox.org>

---------

Co-authored-by: Jan Arends <jan.arends@mailbox.org>
Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>
2023-10-09 10:12:42 -04:00
Jan Arends
d6e6208c26 Data abstraction: Field selection, modifiers and m2o for PostgreSQL driver (#19146)
* Naming typo

* First experiments

* Setup testing playground

* Fetch in client stream

* Add logging

* Remove unneeded stream to string util

* added logical operator to complex test and cleanup

* Improve typing

* started adding function support

* Support nested m2o in data-sql

* Implement join sql string creation

* It's alive!

* Remove unused aliases

* Add todo reminder

* fix build

* small improvements for fn conversion

* function conversion in pg driver

* more expressive typedocs

* toMatchObject in data sql to ignore alias and path

* added type to sort node

* moved and fixes tests for comparison

* moved condition tests

* separate file for functions

* test for function condition

* added proper args to function although not in use

* AT TIME ZONE 'UTC' when needed, proper arg value

* intersects support in pg driver

* convert geo value

* docs for intersects

* reworked column as function input

* support for functions in abstract select

* fixes tests

* count support in select

* refactoring: split up filter conversion

* starting every and some operators

Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>

* extracted variable in test

* in operator with sub query support

* split up conditions type into multiple types

* type for a single query parameter

* condition type

* intersects_bbox in pg driver

* finalized type declarations

* geo condition types

* pnpm lock update

* removed playground

* join

* fixed geo

* fixed types in test

* changeset

* made path prop required again

* geojson for intersects to the driver

* removed sub query from set-condition

* clean up form sub query removal

* added between support

* fixed geo condition test type

* formatting

* moved number operators to utils

* xy-condition to condition-xy

* remove between

* between clean up

* formatter

* refactoring: split up condition generation

* changeset

* split up type for geo condition

* refactoring: split up conditions converter

* fix formatter

* split up types

* split up abstract query into multiple files

* export fix

* split up condition builders in pg driver

* enabled all functions in select and nr condition

* adjusted join conversion and added test

* added test for node conversion

* moved index generator to converter dir

* docs

* format fix

* split up abstract query types further

* split up abstract sql types further

* split ups in converter and fine granular tests

* fix format

* proper index file in converter

* proper type for generator

* more docs on data-sql

* updated data readme

* readme tweaks

* removed circular dependency and added union type

* moved create-identifier function

* added index file in sql utils

* comment on converter usage

* removed circular deps in sql types

* last dependency fixes in data sql

* fixed tests

* cleanup up dependencies in overall abstract query

* removed base type to reduce noise

* reduce some index files in sql types declaration

* formatter

* more dependency tweaks

* formatter

* graphs for data and data-sql

* added dependency cruiser

* added lock file

* pnpm downgrade 8.6.0 to fixed lock file

* try fixing again with 8.6.2

* formatting

* formatting, now with correct formatter

* fixed dependencies

* formatter

* extracted variable for more readable code

Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>

* formatter

* fixed export of condition types for driver

* consistent file naming for conditions

* updated docs

* type fix

* removed todo

* another type fix

* split up expand func and added test

* todo test for driver class

* fixed unit test

* fixed typo

* removed todo, though about it, it's good as it is

* separated path map creation from query-converter

* renamed converter to query-converter

* user specific aliases for primitives

* fixed linter

* user specified alias for m2o

* documentation

* redesigned the function types and added alias support

* formatter

* unit test for pg driver index file

* moved call to source as class member

* moved alias map creation back to converter

* moved unique alias generation to ORM

* tsup update

* renamed nodes to fields

* redefined return type of query converter

* modifier conversion similar to fields conversion

* added response converter dir again

* aligned unit tests

* fixed circular dependency

* moved orm logic one level up again

* formatter

* removed SVGs

* Update pull_request_template.md

* Update pull_request_template.md

* Run formatter

* a bit renaming and restructuring

* formatter

---------

Co-authored-by: rijkvanzanten <rijkvanzanten@me.com>
Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>
Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>
2023-10-06 09:32:04 -04:00
Jan Arends
84bd5d0ef3 Basic filters for PostgreSQL driver (#19005)
* conversion to abstract sql

* greater than for where clause in pg driver

* refactoring

* forward targets again

* more filter options and fixes

* added filters, moved operator mapping into driver

* removed trim

* add changeset

* removed first type approach

* enhanced abstract sql

* value and set comparison in abstract query

* split up where and intersect again

* preps for where comparison to a set

* typedocs

* fix linter

* The random things we did, who knows

* adding logical conditions (WIP)

Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>
Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>

* Push values to parameters directly

* Fix convert test

* Fix convert filter tests

* Implement filter conversion for logical and negate nodes

* Fix where tests

* Add logical where stringification

* Negate comparison operators directly

* Remove some unneeded parentheses

* Improve parameter index generator type

* Fix formatter issues

* Add todo statements

* Remove unused node type

* Removed unneeded check

* One more unnecessary check

---------

Co-authored-by: rijkvanzanten <rijkvanzanten@me.com>
Co-authored-by: Nicola Krumschmidt <nickrum@users.noreply.github.com>
Co-authored-by: Nicola Krumschmidt <nicola.krumschmidt@freenet.de>
2023-07-02 15:10:55 +02:00
Jan Arends
0fea5ab016 converted abstract sort modifier (#18938)
* converted sort

* update comment

* changeset

* change wording

* renamed converter tests

* updated docs

* Change changeset description

Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>

* docs

---------

Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>
2023-06-20 10:36:44 -04:00
Jan Arends
1cbae4952c Limit and offset modifiers for Postgres driver (#18894)
Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>
Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>
2023-06-19 15:51:12 +02:00
Rijk van Zanten
bc7e92e4b0 Add boilerplates for da project (#18642)
* Move data types to data package

* Setup boilerplate for driver

* Add basic data manager class

* Rename "datastore" to "store"

* Rename "fieldNodes" -> "nodes"

Opinions incoming!

* Setup manager basics

* fix pnpm-lock file

* Regenerate pnpm-lock file with pnpm v8.5

* add flowchart diagram

* revert flowchart diagram

* Add a basic data driver for PostgreSQL (#18808)

* add very basic query builder

* Start new data-sql package

* Start noodling

* Results of friday collab session

* add tests and separate function

* add flowchart diagram

* fix tests

* dependency diagram and fix

* add tsdocs

* fix query builder test

* small refactoring

* add underlying error in error message

* add tests for escape function

* add check if pool has a client to query the db

* revert tests, query func handles clients itself

* Update packages/data-sql/package.json

* Resolve comments

* Pass query

* Add tsup

* Remove compileroptions with tsup usage

* Split up converter

* Update graph

---------

Co-authored-by: rijkvanzanten <rijkvanzanten@me.com>

* Improvement: Add albanian language translation (#18810)

* Add albanian to available languages

* Add me as contributor (CLA)

* Create fair-clouds-exercise.md

---------

Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>

* Add extra type check in apply-diff (#18783)

* Fix content filter between (#18821)

* fixed incorrectly typecast value

* Create popular-toes-tap.md

* Created new contribution pages for non-code contributions (#18820)

* Community

* Fixed copy in community page

* Sponsrship and advocacy

* Clarified 80/20 rule in support page

* Reworked contribs intro

* Contributing sidebar

* Remove backing directus page

* Translations updates

* Spell check

---------

Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>

* Reuse S3 client and increase maxSockets (#18799)

* Reuse S3 client and increase maxSockets

* Update test to check for timeouts and non-2xx responses

* Create beige-olives-beam.md

* Fix unit tests

* Update packages/storage-driver-s3/src/index.ts

Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>

* Update .changeset/beige-olives-beam.md

Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>

---------

Co-authored-by: Rijk van Zanten <rijkvanzanten@me.com>
Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>

* Switch to tsup builds

* Don't require connect/disconnect

* Add test for engine

* Fix tests

* Cleanup tests

* Update tests

* Add destroy method

* Call register on register

* Tweak readme to drop connect

* Add changeset

* Wrapper function for random identifiers (#18870)

* add util function in data for random identifiers

* move wrapper into random package

* fix docs typo

Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>

* Rename to identifier and add test

---------

Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>
Co-authored-by: rijkvanzanten <rijkvanzanten@me.com>

* Update packages/data/readme.md

Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>

---------

Co-authored-by: Jan Arends <jan.arends@mailbox.org>
Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>
Co-authored-by: Nino Gjoni <49320058+ninogjoni@users.noreply.github.com>
Co-authored-by: Brainslug <br41nslug@users.noreply.github.com>
Co-authored-by: Kevin Lewis <kvn@lws.io>
Co-authored-by: ian <licitdev@gmail.com>
2023-06-13 11:10:47 -04:00