Base set of Azure DevOps pipeline library source (#3021)

* Base set of Azure DevOps pipeline library source

A base set of yaml files to orchestrate the build and testing of ROCm compiler and runtime components in an Azure DevOps project.

* Use hipcc in llvm-project, also build OpenCL runtime.

* Adding llvm-lit tests to llvm-project pipeline.

Added comgr ctest as well.

* rocm-cmake unit testing in pipeline

* Pipeline changes corresponding to 6.1 release

.azuredevops/README.md

@@ -0,0 +1,29 @@
+# ROCm-CI Azure DevOps Pipelines
+
+ROCm-CI Azure DevOps Pipelines contains markup language files that orchestrate build and test pipelines for ROCm components using [Azure DevOps](https://dev.azure.com/ROCm-CI/ROCm-CI/_build).
+
+## Project Organization
+
+- `/.azuredevops/variables-global.yml` - set of global variables accessible across any and all pipelines
+  - protected keywords such as tokens and passwords are kept as secrets within the Azure DevOps project
+- `/.azuredevops/components` - the sequence of templated steps for the job that checks out source, builds, packages, and runs tests for a ROCm repo
+- `/.azuredevops/scheduled` - the sequence of templated steps for jobs that are schedule-based and not tied to a specific ROCm repo
+- `/.azuredevops/templates` - reusable yml files representing the templated steps that form the sequences in the above directories
+
+### Per ROCm Repo
+
+- `/.azuredevops/rocm-ci.yml` - contains the CI and PR trigger definitions associated with that repo, pointing to the corresponding yml file in the components folder in this central repository
+
+## Key Azure Reference Links
+
+- [Pipeline Basics](https://learn.microsoft.com/en-us/azure/devops/pipelines/get-started/key-pipelines-concepts?view=azure-devops)
+- [Templates](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops&pivots=templates-includes)
+- [Use Predefined Variables](https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml)
+- [YAML schema](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/?view=azure-pipelines&viewFallbackFrom=azure-devops)
+- [Azure Pipelines Task Index](https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/?view=azure-pipelines)
+
+## Disclaimer
+
+The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard versionchanges, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated.AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes.THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies.
+
+© 2024 Advanced Micro Devices, Inc. All Rights Reserved.

.azuredevops/components/HIP.yml

@@ -0,0 +1,70 @@
+parameters:
+- name: aptPackages
+  type: string
+  default: 'libnuma-dev mesa-common-dev'
+- name: pipModules
+  type: string
+  default: 'CppHeaderParser'
+
+# hip and clr are tightly-coupled
+# run this same template for both repos
+# any changes for clr should just trigger HIP pipeline
+jobs:
+- job: hip_clr_combined
+  variables:
+  - group: common
+  - template: /.azuredevops/variables-global.yml
+  pool:
+    vmImage: ${{ variables.LOW_END_BUILD_POOL }}
+  workspace:
+    clean: all
+  steps:
+# checkout triggering repo (either HIP or clr)
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/checkout.yml
+# if this is triggered by HIP repo, matching repo is clr
+# if this is triggered by clr repo, matching repo is HIP
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/checkout.yml
+    parameters:
+      checkoutRepo: matching_repo
+# ${{ }} are resolved during compile-time
+# so this next step is skipped completely until
+# we define explicit aptPackages needed to install
+  - ${{ if parameters.aptPackages }}:
+    - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/dependencies.yml
+      parameters:
+        aptPackages: ${{ parameters.aptPackages }}
+        pipModules: ${{ parameters.pipModules }}
+# download tasks has built in 4x retry download count
+# these will download last passing build for that branch
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-download.yml
+    parameters:
+      componentName: llvm-project
+      branchName: amd-staging
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-download.yml
+    parameters:
+      componentName: ROCT-Thunk-Interface
+      branchName: master
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-download.yml
+    parameters:
+      componentName: ROCR-Runtime
+      branchName: master
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-download.yml
+    parameters:
+      componentName: rocprofiler-register
+      branchName: amd-mainline
+  - script: 'ls -1R $(Agent.BuildDirectory)/rocm'
+    displayName: 'Artifact listing'
+# compile clr
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/build-cmake.yml
+    parameters:
+      componentName: clr
+      cmakeBuildDir: 'clr/build'
+      extraBuildFlags: >-
+        -DHIP_COMMON_DIR="$(Build.SourcesDirectory)/HIP"
+        -DHIP_PLATFORM=amd
+        -DCMAKE_PREFIX_PATH="$(Agent.BuildDirectory)/rocm"
+        -DROCM_PATH="$(Agent.BuildDirectory)/rocm"
+        -DHIPCC_BIN_DIR="$(Agent.BuildDirectory)/rocm/bin"
+        -DCLR_BUILD_HIP=ON
+        -DCLR_BUILD_OCL=ON
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-upload.yml

.azuredevops/components/ROCR-Runtime.yml

@@ -0,0 +1,40 @@
+parameters:
+- name: aptPackages
+  type: string
+  default: 'libelf-dev g++ libdrm-dev libnuma-dev'
+
+jobs:
+- job: ROCR_Runtime
+  variables:
+  - group: common
+  - template: /.azuredevops/variables-global.yml
+  pool:
+    vmImage: ${{ variables.LOW_END_BUILD_POOL }}
+  workspace:
+    clean: all
+  steps:
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/preamble.yml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/checkout.yml
+# ${{ }} are resolved during compile-time
+# so this next step is run because
+# we define explicit aptPackages needed to install
+  - ${{ if parameters.aptPackages }}:
+    - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/dependencies.yml
+      parameters:
+        aptPackages: ${{ parameters.aptPackages }}
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-download.yml
+    parameters:
+      componentName: llvm-project
+      branchName: amd-staging
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-download.yml
+    parameters:
+      componentName: ROCT-Thunk-Interface
+      branchName: master
+  - script: 'ls -1R $(Agent.BuildDirectory)/rocm'
+    displayName: 'Artifact listing'
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/build-cmake.yml
+    parameters:
+      extraBuildFlags: >-
+        -DCMAKE_PREFIX_PATH="$(Agent.BuildDirectory)/rocm"
+      cmakeBuildDir: 'src/build'
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-upload.yml

.azuredevops/components/ROCT-Thunk-Interface.yml

@@ -0,0 +1,26 @@
+parameters:
+- name: aptPackages
+  type: string
+  default: 'libnuma-dev libdrm-dev'
+
+jobs:
+- job: ROCT_Thunk_Interface
+  variables:
+  - group: common
+  - template: /.azuredevops/variables-global.yml
+  pool:
+    vmImage: ${{ variables.LOW_END_BUILD_POOL }}
+  workspace:
+    clean: all
+  steps:
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/preamble.yml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/checkout.yml
+# ${{ }} are resolved during compile-time
+# so this next step is skipped completely until
+# we define explicit aptPackages needed to install
+  - ${{ if parameters.aptPackages }}:
+    - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/dependencies.yml
+      parameters:
+        aptPackages: ${{ parameters.aptPackages }}
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/build-cmake.yml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-upload.yml

.azuredevops/components/copyHIP.yml

@@ -0,0 +1,32 @@
+parameters:
+- name: aptPackages
+  type: string
+  default: 'libnuma-dev mesa-common-dev'
+- name: pipModules
+  type: string
+  default: 'CppHeaderParser'
+
+# hip and clr are tightly-coupled
+# run this same template for both repos
+# any changes for clr should just trigger HIP pipeline
+jobs:
+- job: hip_clr_combined
+  variables:
+  - group: common
+  - template: /.azuredevops/variables-global.yml
+  pool:
+    vmImage: ${{ variables.LOW_END_BUILD_POOL }}
+  workspace:
+    clean: all
+  steps:
+# checkout nothing, just copy artifacts from triggering HIP job
+# and then publish for this clr job to maintain latest
+  - checkout: none
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-download.yml
+    parameters:
+      componentName: HIP
+      branchName: develop
+  - template:  ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-prepare-package.yml
+    parameters:
+      sourceDir: $(Agent.BuildDirectory)/rocm
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-upload.yml

.azuredevops/components/llvm-project.yml

@@ -0,0 +1,91 @@
+parameters:
+- name: aptPackages
+  type: string
+  default: 'libnuma-dev ninja-build ccache'
+
+jobs:
+- job: llvm_project
+  variables:
+  - group: common
+  - template: /.azuredevops/variables-global.yml
+  pool: ${{ variables.HIGH_END_BUILD_POOL }}
+  workspace:
+    clean: all
+  steps:
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/preamble.yml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/ccache.yml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/checkout.yml
+# ${{ }} are resolved during compile-time
+# so this next step is skipped completely until
+# we define explicit aptPackages needed to install
+  - ${{ if parameters.aptPackages }}:
+    - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/dependencies.yml
+      parameters:
+        aptPackages: ${{ parameters.aptPackages }}
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/build-cmake.yml
+    parameters:
+      componentName: rocm-llvm
+      extraBuildFlags: >-
+        -DCMAKE_PREFIX_PATH="$(Build.BinariesDirectory)/llvm;$(Build.BinariesDirectory)"
+        -DCMAKE_BUILD_TYPE=Release
+        -DLLVM_ENABLE_PROJECTS="llvm;clang;lld;clang-tools-extra"
+        -DLLVM_ENABLE_RUNTIMES="compiler-rt;libunwind"
+        -DCLANG_ENABLE_AMDCLANG="ON"
+        -DLLVM_TARGETS_TO_BUILD="AMDGPU;X86"
+        -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
+        -GNinja
+      cmakeBuildDir: 'llvm/build'
+      installDir: '$(Build.BinariesDirectory)/llvm'
+# use llvm-lit to run unit tests for llvm, clang, and lld
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/test.yml
+    parameters:
+      componentName: check-llvm
+      testDir: 'llvm/build'
+      testExecutable: './bin/llvm-lit'
+      testParameters: '-q --xunit-xml-output=llvm_test_output.xml ./test'
+      testOutputFile: llvm_test_output.xml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/test.yml
+    parameters:
+      componentName: check-clang
+      testDir: 'llvm/build'
+      testExecutable: './bin/llvm-lit'
+      testParameters: '-q --xunit-xml-output=clang_test_output.xml ./tools/clang/test'
+      testOutputFile: clang_test_output.xml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/test.yml
+    parameters:
+      componentName: check-lld
+      testDir: 'llvm/build'
+      testExecutable: './bin/llvm-lit'
+      testParameters: '-q --xunit-xml-output=lld_test_output.xml ./tools/lld/test'
+      testOutputFile: lld_test_output.xml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/build-cmake.yml
+    parameters:
+      componentName: device-libs
+      extraBuildFlags: >-
+        -DCMAKE_PREFIX_PATH="$(Build.SourcesDirectory)/llvm/build"
+        -DCMAKE_BUILD_TYPE=Release
+        -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
+      cmakeBuildDir: 'amd/device-libs/build'
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/build-cmake.yml
+    parameters:
+      componentName: comgr
+      extraBuildFlags: >-
+        -DCMAKE_PREFIX_PATH="$(Build.SourcesDirectory)/llvm/build;$(Build.SourcesDirectory)/amd/device-libs/build"
+        -DCMAKE_BUILD_TYPE=Release
+        -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
+      cmakeBuildDir: 'amd/comgr/build'
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/test.yml
+    parameters:
+      componentName: comgr
+      testParameters: '--output-on-failure --force-new-ctest-process --output-junit comgr_test_output.xml'
+      testDir: 'amd/comgr/build'
+      testOutputFile: comgr_test_output.xml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/build-cmake.yml
+    parameters:
+      componentName: hipcc
+      extraBuildFlags: >-
+        -DCMAKE_BUILD_TYPE=Release
+        -DHIPCC_BACKWARD_COMPATIBILITY=OFF
+        -GNinja
+      cmakeBuildDir: 'amd/hipcc/build'
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-upload.yml

.azuredevops/components/rocm-cmake.yml

@@ -0,0 +1,36 @@
+parameters:
+- name: aptPackages
+  type: string
+  default: 'doxygen doxygen-doc ninja-build python3-sphinx'
+- name: pipModules
+  type: string
+  default: 'cget ninja'
+
+jobs:
+- job: rocm_cmake
+  variables:
+  - group: common
+  - template: /.azuredevops/variables-global.yml
+  pool:
+    vmImage: ${{ variables.LOW_END_BUILD_POOL }}
+  workspace:
+    clean: all
+  steps:
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/preamble.yml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/checkout.yml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/build-cmake.yml
+# dependencies are only for testing, so done after compilation
+  - ${{ if parameters.aptPackages }}:
+    - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/dependencies.yml
+      parameters:
+        aptPackages: ${{ parameters.aptPackages }}
+        pipModules: ${{ parameters.pipModules }}
+# extra steps for ctest suite
+  - script: |
+      python -m pip install -r $(Build.SourcesDirectory)/docs/requirements.txt
+      python -m pip install -r $(Build.SourcesDirectory)/test/docsphinx/docs/.sphinx/requirements.txt
+      git config --global user.email "you@example.com"
+      git config --global user.name "Your Name"
+    displayName: "ctest setup"
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/test.yml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-upload.yml

.azuredevops/components/rocm-core.yml

@@ -0,0 +1,30 @@
+jobs:
+- job: rocm_core
+  variables:
+  - group: common
+  - template: /.azuredevops/variables-global.yml
+  pool:
+    vmImage: ${{ variables.LOW_END_BUILD_POOL }}
+  workspace:
+    clean: all
+  steps:
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/preamble.yml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/checkout.yml
+# ${{ }} are resolved during compile-time
+# so this next step is skipped completely until
+# we define explicit aptPackages needed to install
+  - ${{ if parameters.aptPackages }}:
+    - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/dependencies.yml
+      parameters:
+        aptPackages: ${{ parameters.aptPackages }}
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/build-cmake.yml
+    parameters:
+      extraBuildFlags: >-
+        -DCMAKE_CURRENT_BINARY_DIR=$PWD
+        -DCMAKE_CURRENT_SOURCE_DIR=$PWD/../
+        -DCMAKE_VERBOSE_MAKEFILE=1
+        -DCPACK_GENERATOR=DEB
+        -DCPACK_DEBIAN_PACKAGE_RELEASE="local.9999~99.99"
+        -DCPACK_RPM_PACKAGE_RELEASE="local.9999"
+        -DROCM_VERSION="$(last-release)"
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-upload.yml

.azuredevops/components/rocminfo.yml

@@ -0,0 +1,28 @@
+jobs:
+- job: rocminfo
+  variables:
+  - group: common
+  - template: /.azuredevops/variables-global.yml
+  pool:
+    vmImage: ${{ variables.LOW_END_BUILD_POOL }}
+  workspace:
+    clean: all
+  steps:
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/preamble.yml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/checkout.yml
+# ${{ }} are resolved during compile-time
+# so this next step is skipped completely until
+# we define explicit aptPackages needed to install
+  - ${{ if parameters.aptPackages }}:
+    - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/dependencies.yml
+      parameters:
+        aptPackages: ${{ parameters.aptPackages }}
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-download.yml
+    parameters:
+      componentName: ROCR-Runtime
+      branchName: master
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/build-cmake.yml
+    parameters:
+      extraBuildFlags: >-
+        -DCMAKE_PREFIX_PATH="$(Agent.BuildDirectory)/rocm"
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-upload.yml

.azuredevops/components/rocprofiler-register.yml

@@ -0,0 +1,21 @@
+jobs:
+- job: rocprofiler_register
+  variables:
+  - group: common
+  - template: /.azuredevops/variables-global.yml
+  pool:
+    vmImage: ${{ variables.LOW_END_BUILD_POOL }}
+  workspace:
+    clean: all
+  steps:
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/preamble.yml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/checkout.yml
+# ${{ }} are resolved during compile-time
+# so this next step is skipped completely until
+# we define explicit aptPackages needed to install
+  - ${{ if parameters.aptPackages }}:
+    - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/dependencies.yml
+      parameters:
+        aptPackages: ${{ parameters.aptPackages }}
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/build-cmake.yml
+  - template: ${{ variables.CI_TEMPLATE_PATH }}/steps/artifact-upload.yml

.azuredevops/templates/steps/artifact-download.yml

@@ -0,0 +1,31 @@
+parameters:
+# assumption componentName and pipeline name the same
+- name: componentName
+  type: string
+  default: ''
+- name: branchName
+  type: string
+  default: ''
+
+steps:
+- task: DownloadBuildArtifacts@1
+  displayName: Download ${{parameters.componentName }}
+  inputs:
+    buildType: 'specific'
+    project: ROCm-CI
+    pipeline: ${{parameters.componentName }}
+    specificBuildWithTriggering: true
+    branchName: ${{parameters.branchName }}
+- task: ExtractFiles@1
+  displayName: Extract ${{parameters.componentName }}
+  inputs:
+    archiveFilePatterns: '$(System.ArtifactsDirectory)/**/*.tar.xz'
+    destinationFolder: '$(Agent.BuildDirectory)/rocm'
+    cleanDestinationFolder: false
+    overwriteExistingFiles: true
+- task: DeleteFiles@1
+  displayName: Cleanup Compressed ${{parameters.componentName }}
+  inputs:
+    SourceFolder: '$(System.ArtifactsDirectory)'
+    Contents: '/**/*.tar.xz'
+    RemoveDotFiles: true

.azuredevops/templates/steps/artifact-prepare-package.yml

@@ -0,0 +1,25 @@
+# optional step if not using cmake --install
+# prepare the next package to upload
+parameters:
+- name: sourceDir # REQUIRED PARAMETER
+  type: string
+  default: ''
+- name: contentsString
+  type: string
+  default: '**'
+- name: targetDir
+  type: string
+  default: '$(Build.BinariesDirectory)'
+- name: clean
+  type: boolean
+  default: true
+
+steps:
+- task: CopyFiles@2
+  displayName: 'Prepare for packaging'
+  inputs:
+    CleanTargetFolder: ${{ parameters.clean }}
+    SourceFolder: '${{ parameters.sourceDir }}'
+    Contents: '${{ parameters.contentsString }}'
+    TargetFolder: '${{ parameters.targetDir }}'
+    retryCount: 3

.azuredevops/templates/steps/artifact-upload.yml

@@ -0,0 +1,31 @@
+# compress build products into tarball
+# delete build products after compression
+# publish can be toggled off for jobs that produce multiple tarballs
+# for those cases, only publish the last call which puts all the tarballs in one container folder
+parameters:
+- name: artifactName # REQUIRED PARAMETER
+  type: string
+  default: 'drop'
+- name: publish
+  type: boolean
+  default: true
+
+steps:
+- task: ArchiveFiles@2
+  displayName: '${{ parameters.artifactName }} Compress'
+  inputs:
+    includeRootFolder: false
+    archiveType: 'tar'
+    tarCompression: 'xz'
+    archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.DefinitionName)_$(Build.SourceBranchName)_$(Build.BuildId)_$(Build.BuildNumber)_ubuntu2204_${{ parameters.artifactName }}.tar.xz'
+- task: DeleteFiles@1
+  displayName: 'Cleanup Staging Area'
+  inputs:
+    SourceFolder: '$(Build.BinariesDirectory)'
+    Contents: '/**/*'
+    RemoveDotFiles: true
+# then publish it
+- ${{ if parameters.publish }}:
+  - task: PublishBuildArtifacts@1
+    displayName: '${{ parameters.artifactName }} Publish'
+    retryCountOnTaskFailure: 3

.azuredevops/templates/steps/build-cmake.yml

@@ -0,0 +1,43 @@
+parameters:
+- name: componentName
+  type: string
+  default: ''
+- name: extraBuildFlags
+  type: string
+  default: ''
+- name: cmakeBuildDir
+  type: string
+  default: 'build'
+- name: cmakeTarget
+  type: string
+  default: 'install'
+- name: cmakeTargetDir
+  type: string
+  default: '.'
+- name: installDir
+  type: string
+  default: '$(Build.BinariesDirectory)'
+
+steps:
+# create workingDirectory if it does not exist and change into it
+# call cmake from within that directory using $cmakeArgs as its parameters
+- task: CMake@1
+  displayName: '${{parameters.componentName }} CMake Flags'
+  inputs:
+    workingDirectory: ${{ parameters.cmakeBuildDir }}
+    cmakeArgs: -DCMAKE_INSTALL_PREFIX=${{ parameters.installDir }} ${{ parameters.extraBuildFlags }} ..
+# equivalent to running make $cmakeTargetDir from $cmakeBuildDir
+# i.e., cd $cmakeBuildDir; make $cmakeTargetDir
+- task: CMake@1
+  displayName: '${{parameters.componentName }} Build'
+  inputs:
+    workingDirectory: ${{ parameters.cmakeBuildDir }}
+    cmakeArgs: '--build ${{ parameters.cmakeTargetDir }}'
+    retryCountOnTaskFailure: 10
+# equivalent to running make $cmakeTarget from $cmakeBuildDir
+# e.g., make install
+- task: CMake@1
+  displayName: '${{parameters.componentName }} Install'
+  inputs:
+    workingDirectory: ${{ parameters.cmakeBuildDir }}
+    cmakeArgs: '--build ${{ parameters.cmakeTargetDir }} --target ${{ parameters.cmakeTarget }}'

.azuredevops/templates/steps/build-script.yml

@@ -0,0 +1,22 @@
+parameters:
+- name: componentName
+  type: string
+  default: ''
+- name: scriptDir
+  type: string
+  default: '.'
+- name: scriptExecutable
+  type: string
+  default: './install.sh'
+- name: scriptParameters
+  type: string
+  default: ''
+
+steps:
+- task: Bash@3
+  displayName: '${{ parameters.componentName }} Test'
+  continueOnError: true
+  inputs:
+    targetType: inline
+    script: ${{ parameters.scriptExecutable }} ${{ parameters.scriptParameters }}
+    workingDirectory: ${{ parameters.scriptDir }}

.azuredevops/templates/steps/ccache.yml

@@ -0,0 +1,11 @@
+# to use ccache, run this template step before build-cmake.yml
+# and add this cmake flag to extraBuildFlags
+# -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
+steps:
+- task: Cache@2
+  displayName: Ccache caching
+  inputs:
+    key: 'ccache | "$(Agent.OS)" | $(Build.SourceVersion)'
+    path: $(CCACHE_DIR)
+    restoreKeys: |
+      ccache | "$(Agent.OS)"

.azuredevops/templates/steps/checkout.yml

@@ -0,0 +1,17 @@
+parameters:
+# name of the repo to checkout
+# for most cases, leave as default 'self'
+- name: checkoutRepo
+  type: string
+  default: 'self'
+# submodule download behaviour
+# change to 'recursive' for repos with submodules
+- name: submoduleBehaviour
+  type: string
+  default: 'true'
+
+steps:
+  - checkout: ${{ parameters.checkoutRepo }}
+    clean: true
+    submodules: ${{ parameters.submoduleBehaviour }}
+    retryCountOnTaskFailure: 3

.azuredevops/templates/steps/dependencies.yml

@@ -0,0 +1,38 @@
+parameters:
+# space-delimited list of packages to install via apt-get install command
+# TODO convert to string array and split with spaces
+- name: aptPackages
+  type: string
+  default: ''
+- name: pipModules
+  type: string
+  default: ''
+
+steps:
+- task: Bash@3
+  displayName: 'sudo apt-get update'
+  inputs:
+    targetType: inline
+    script: sudo apt-get update
+- task: Bash@3
+  displayName: 'sudo apt-get upgrade'
+  inputs:
+    targetType: inline
+    script: sudo apt-get update
+- task: Bash@3
+  displayName: 'sudo apt-get fix'
+  inputs:
+    targetType: inline
+    script: sudo apt --yes --fix-broken install
+- ${{ if parameters.aptPackages }}:
+  - task: Bash@3
+    displayName: 'sudo apt-get install ...'
+    inputs:
+      targetType: inline
+      script: sudo apt-get --yes install ${{ parameters.aptPackages }}
+- ${{ if parameters.pipModules }}:
+  - task: Bash@3
+    displayName: 'pip install  ...'
+    inputs:
+      targetType: inline
+      script: pip install ${{ parameters.pipModules }}

.azuredevops/templates/steps/preamble.yml

@@ -0,0 +1,26 @@
+# build artifacts not automatically cleaned up
+# force cleanup, always
+# also display installed components and packages
+steps:
+- task: DeleteFiles@1
+  inputs:
+    SourceFolder: '$(Agent.BuildDirectory)/s'
+    Contents: '**/*'
+- task: Bash@3
+  displayName: 'apt installed list'
+  inputs:
+    targetType: inline
+    script: apt list --installed
+- task: Bash@3
+  displayName: 'python version'
+  inputs:
+    targetType: inline
+    script: python3 --version
+- script: pip list
+  displayName: 'list python packages'
+- task: DeleteFiles@1
+  displayName: 'Cleanup Staging Area'
+  inputs:
+    SourceFolder: '$(Build.ArtifactStagingDirectory)'
+    Contents: '/**/*'
+    RemoveDotFiles: true

.azuredevops/templates/steps/test.yml

@@ -0,0 +1,36 @@
+parameters:
+- name: componentName
+  type: string
+  default: ''
+- name: testDir
+  type: string
+  default: 'build'
+- name: testExecutable
+  type: string
+  default: 'ctest'
+- name: testParameters
+  type: string
+  default: '--output-on-failure --force-new-ctest-process --output-junit test_output.xml'
+- name: testOutputFile
+  type: string
+  default: test_output.xml
+- name: testOutputFormat
+  type: string
+  default: 'junit'
+
+steps:
+# run test, continue on failure to publish results
+# and to publish build artifacts
+- task: Bash@3
+  displayName: '${{ parameters.componentName }} Test'
+  continueOnError: true
+  inputs:
+    targetType: inline
+    script: ${{ parameters.testExecutable }} ${{ parameters.testParameters }}
+    workingDirectory: ${{ parameters.testDir }}
+- task: PublishTestResults@2
+  displayName: '${{ parameters.componentName }} Publish Results'
+  inputs:
+    testResultsFormat: ${{ parameters.testOutputFormat }}
+    testResultsFiles: '**/${{ parameters.testOutputFile }}'
+  condition: succeededOrFailed()

.azuredevops/variables-global.yml

@@ -0,0 +1,13 @@
+# specify non-secret global variables reused across pipelines here
+
+variables:
+- name: CI_ROOT_PATH
+  value: /.azuredevops
+- name: CI_COMPONENT_PATH
+  value: ${{ variables.CI_ROOT_PATH }}/components
+- name: CI_TEMPLATE_PATH
+  value: ${{ variables.CI_ROOT_PATH }}/templates
+- name: LOW_END_BUILD_POOL
+  value: ubuntu-22.04
+- name: HIGH_END_BUILD_POOL
+  value: rocm-ci_build_pool