Commit Graph

458 Commits

Author SHA1 Message Date
Luis Montero
30be8cf4ae fix: fix MacOS test launching
* Update makefile to use appropriate arguments to find executables on
  macos systems.
* Add `set -e` to make sure that the tests crash if something goes wrong
  in the build or test of the macos job of the CI.

closes #783
2022-11-25 09:48:45 +01:00
David Testé
1e8c0df381 chore(ci): change benchmark parser input name
The use of "schema" was incorrect since it's meant to be used as
database name when sending data to Slab.
2022-11-23 16:22:17 +01:00
youben11
2877281aa6 chore(ci): re-enable rust tests in the CI
they were mostly removed by a rebase
2022-11-23 14:01:25 +01:00
rudy
454edbb538 fix(ci): KeySetCache pruning was broken 2022-11-22 17:28:40 +01:00
David Testé
3c2a75186f chore(ci): run ml benchmarks in a matrix with slab
This CI "feature" is meant to circumvent the 6 hours hard-limit
for a job in GitHub Action.
The benchmark is done using a matrix which is handled by Slab.
Here's the workflow:

  1. ML benchmarks are started in a fire and forget fashion via
     start_ml_benchmarks.yml
  2. Slab will read ci/slab.toml to get the AWS EC2 configuration
     and the matrix parameters
  3. Slab will launch at most max_parallel_jobs EC2 instances in
     parallel
  4. Each job will trigger ml_benchmark_subset.yml which will run
     only one of the generated YAML file via make generate-mlbench,
     based on the value of the matrix item they were given.
  5. As soon as a job is completed, the next one in the matrix
     will start promptly.

This is done until all the matrix items are exhausted.
2022-11-21 11:25:40 +01:00
Mayeul@Zama
fa3556e8cc feat(CI): add cmake-format CI job 2022-11-18 15:15:41 +01:00
David Testé
77729514e0 chore(ci): parse benchmark results to send to postgres instance
Previousely, we were sending parsed benchmark results to a
Prometheus instance. Do to its time-series nature, Prometheus would
downsample database content to avoid having to much data points
for a given range of time. While this behavior is good for a
continuous stream of data, like monitoring CPU load, it's not suited
for benchmarks. Indeed benchmarks are discrete events that would
occurr once in a while (i.e once a day). Downsampling would, at
some point, simply omit some of benchmarks results.
Using a regular SQL database like PostgreSQL solves this issue.
2022-11-15 16:18:25 +01:00
dependabot[bot]
31d5915cee chore(deps): bump machulav/ec2-github-runner from 2.3.2 to 2.3.3
Bumps [machulav/ec2-github-runner](https://github.com/machulav/ec2-github-runner) from 2.3.2 to 2.3.3.
- [Release notes](https://github.com/machulav/ec2-github-runner/releases)
- [Commits](c34ba2df33...4e0303de21)

---
updated-dependencies:
- dependency-name: machulav/ec2-github-runner
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-15 11:49:52 +01:00
rudy
1abea50ff8 chore(ci): only cancel continuous-integration.yml on PR update 2022-11-14 11:09:03 +01:00
rudy
5db1b7a27c chore(benchmark): compile then download 2022-11-14 11:06:47 +01:00
rudy
5b92ed151c chore(ci): build before download test cache 2022-11-09 16:14:29 +01:00
rudy
819c9e71ca fix(ci): diff for cpp API change 2022-11-08 11:28:51 +01:00
David Testé
d1db4a5e45 tests(benchmarks): reduce number of test cases to speed-up execution
Bench just one compilation option for automatic benchmarks. Only 'loop'
option is tested to take advantage of hardware with a lot of available
CPUs. Running benchmarks with 'default' option is suboptimal for this
kind of hardware since it uses only one CPU.

This also remove time consuming MNIST test, as it should be in ML benchmarks.

Moreover Makefile is fixed to use provided Python executable instead of
relying on system one to generate MLIR Yaml files.
2022-11-03 19:07:39 +01:00
youben11
078a82c1a4 fix(ci): use specific python binary
setup-python seems to choose a binary that lead to issues while trying
to install the wheeled package
2022-11-03 13:29:11 +01:00
dependabot[bot]
88fb2ee134 chore(deps): bump marocchino/sticky-pull-request-comment
Bumps [marocchino/sticky-pull-request-comment](https://github.com/marocchino/sticky-pull-request-comment) from 2.2.1 to 2.3.0.
- [Release notes](https://github.com/marocchino/sticky-pull-request-comment/releases)
- [Commits](97bddef64d...adca94abca)

---
updated-dependencies:
- dependency-name: marocchino/sticky-pull-request-comment
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-03 12:11:54 +01:00
youben11
e3e1602fe8 fix(ci): set python to 3.10 in macos test runner
default is now 3.11 which we don't support yet, mainly due to our
version of pybind11. 2.10.1 is the first pybind11 version to supports it
2022-11-02 14:46:38 +01:00
youben11
e40c6392cb ci: build and test rust bindings 2022-11-02 14:46:38 +01:00
Mayeul@Zama
c994c7afcc chore(CI): add nightly rust fmt and clippy checks 2022-10-31 14:46:48 +01:00
dependabot[bot]
8a43c24821 chore(deps): bump actions/upload-artifact from 3.1.0 to 3.1.1
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.0 to 3.1.1.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](3cea537223...83fd05a356)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-25 11:53:41 +03:00
dependabot[bot]
57ac87ad60 chore(deps): bump actions/download-artifact from 3.0.0 to 3.0.1
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 3.0.0 to 3.0.1.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](fb598a63ae...9782bd6a98)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-25 11:53:31 +03:00
rudy
9156502a5a fix(ci): bump actions versions
to remove reprecation warning about nodejs 12
2022-10-24 15:08:54 +02:00
rudy
1eb8cd06c7 fix(ci): update action to last version to avoid deprecation warnings 2022-10-21 15:54:23 +02:00
rudy
a29c4949e4 ci: do everything in --release to avoid 2x compilations 2022-10-19 15:14:38 +02:00
rudy
b5a269f0a2 chore(tests): merge back expensive tests to normal tests 2022-10-19 15:13:37 +02:00
Arthur Meyre
a487b03699 chore(ci): fix quoting on cp command 2022-10-18 12:02:57 +02:00
David Testé
d2b20ca551 ci(bench): fix end-to-end benchmarks workflow
This includes several fixes and add some functionalities:
 * EC2 instance type can be selected when workflow is triggered manually
 * benchmarks will be run on each push on main branch
 * docker is not used any more due to building issues
 * 10 repetitions are made during the benchmarks then results are aggregated
 * more tags are used to identify benchmarks configuration
2022-10-18 12:02:35 +02:00
dependabot[bot]
adede72d08 chore(deps): bump docker/login-action from 2.0.0 to 2.1.0
Bumps [docker/login-action](https://github.com/docker/login-action) from 2.0.0 to 2.1.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](49ed152c8e...f4ef78c080)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-17 12:00:15 +03:00
dependabot[bot]
40a898ddd1 chore(deps): bump docker/build-push-action from 3.1.1 to 3.2.0
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 3.1.1 to 3.2.0.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](c84f382811...c56af95754)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-17 11:31:45 +03:00
dependabot[bot]
198092af6c chore(deps): bump marocchino/sticky-pull-request-comment
Bumps [marocchino/sticky-pull-request-comment](https://github.com/marocchino/sticky-pull-request-comment) from 2.2.0 to 2.2.1.
- [Release notes](https://github.com/marocchino/sticky-pull-request-comment/releases)
- [Commits](39c5b5dc77...97bddef64d)

---
updated-dependencies:
- dependency-name: marocchino/sticky-pull-request-comment
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-17 11:31:38 +03:00
dependabot[bot]
71d16efd5f chore(deps): bump actions/setup-python from 4.2.0 to 4.3.0
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4.2.0 to 4.3.0.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](b55428b188...13ae5bb136)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-17 11:31:31 +03:00
dependabot[bot]
d7d723487c chore(deps): bump actions/checkout from 3.0.2 to 3.1.0
Bumps [actions/checkout](https://github.com/actions/checkout) from 3.0.2 to 3.1.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](2541b1294d...93ea575cb5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-13 15:29:20 +03:00
Arthur Meyre
4a3eca10bb chore(ci): fix quoting issue from jq command 2022-10-11 17:09:47 +02:00
youben11
0987fade80 fix: parameterize DATAFLOW_EXECUTION in python wheel 2022-10-07 16:59:34 +01:00
David Testé
89d7f065ae ci(compiler): schedule end-to-end benchmarks
Run end-to-end benchmarks on a weekly basis. Results are parsed and
then sent to Slab CI bot to be later plotted into Grafana.
2022-10-07 14:45:56 +02:00
Arthur Meyre
6ccbb8555c chore(ci): remove now useless workflow, remove ffi compilation 2022-10-07 13:44:20 +02:00
Arthur Meyre
9d3791bf96 chore: fix most actionlint and shellcheck lints 2022-10-07 09:49:00 +02:00
rudy
cb2c9ef6bf feat: accept no evaluation keys 2022-09-26 14:43:25 +02:00
rudy
d5188aa2f2 ci: run on macos 2022-09-21 09:45:00 +02:00
youben11
1d52bee1aa ci: use ccache for Mac release 2022-09-19 13:24:20 +01:00
youben11
de19fdc868 fix: set SYSTEM_VERSION_COMPAT to 0 during macos tests
This is a known issue while using setup-python action
https://github.com/actions/setup-python/issues/279
2022-09-19 13:24:20 +01:00
youben11
628df8849c refactor: unify Makefile target naming style 2022-09-19 13:24:20 +01:00
youben11
8f05f9ea72 ci: change package release process
it should be now faster using the same compilation cache as the test
build.
2022-09-19 13:24:20 +01:00
youben11
698bd28104 ci: refactor docker images and CI jobs
use a single docker image for all testing environments
unify some CI jobs on this single environment
also parameterize the build of the doc, which no longer have a hardcoded
path for the compiler build directory
2022-09-13 15:18:15 +01:00
youben11
f88cd95eaf ci: update df image to use latest hpx
also removes some old copies in dockerfile that were meant to reduce
image size. However, as many entities need to be copied now, we might
need to have a better approach for this (using make install maybe), and
copy everything that is necessary into the right place.
2022-09-13 15:18:15 +01:00
youben11
0cd9a7984d feat: support dataflow execution in python packages
use latest image manylinux_2_28_x86_64 using libc 2.28
2022-09-13 15:18:15 +01:00
Umut
ba2d886295 chore: fix release action 2022-09-08 12:38:43 +03:00
youben11
4b584ca64e ci: create new keysetcache
last commit updated which keys we serialize
2022-09-06 07:18:34 +01:00
Umut
0cfceaaf03 feat: support python 3.10 2022-09-02 13:38:17 +03:00
rudy
4a6b9148d1 feat(github): auto cancel previous continuous-integration.yml run when a PR is updated 2022-09-02 09:08:15 +01:00
Quentin Bourgerie
c08a06ed8e chore: Just rename configuration variable to enable the dataflow runtime 2022-08-24 11:35:05 +02:00