github/electron
1
1
Fork 0
mirror of https://github.com/electron/electron.git synced 2026-04-10 03:01:51 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
aa00a5191470ae262c4019f0dcf695d5a6e6aa68
electron/.github/workflows/pipeline-segment-electron-publish.yml
Samuel Attard fef2fd2941 ci: zstd-compress the src cache and drop the doubled win_toolchain (#50702) 
* ci: shrink src cache and fix Windows tar cleanup

- Exclude platform-specific toolchains (llvm-build, rust-toolchain) from
  the src cache; all platforms now fetch them via fix-sync post-restore
- Exclude unused test data and benchmarks: blink/web_tests, jetstream,
  speedometer, catapult/tracing/test_data, swiftshader/tests/regres
- Fix Windows restore leaving the tarball on disk after extraction
  ($src_cache was scoped to the previous PowerShell step)
- Bump src-cache key v1 -> v2

* ci: fetch llvm/rust toolchains in gn-check and clang-tidy

These workflows restore the src cache but don't run fix-sync. Now that
llvm-build and rust-toolchain are excluded from the cache, they need to
download them directly — gn gen read_file()s both, and clang-tidy runs
the binary from llvm-build.

* ci: fetch clang-tidy package explicitly

update.py's default 'clang' package doesn't include the clang-tidy
binary; it ships as a separate package.

* ci: preserve blink/web_tests/BUILD.gn when stripping test data

//BUILD.gn references //third_party/blink/web_tests:wpt_tests as a
target label, so the BUILD.gn must exist for gn gen. The data = [...]
entries it declares are runtime-only and not existence-checked at gen
time, so the actual test directories can still be removed.

* ci: compress src cache with zstd and drop gclient sync -vv

The src cache was an uncompressed tar (~16GB after exclusions). Switch
to zstd -T0 --long=30 for ~4x smaller transfer and multi-threaded
compression. Decompress on restore:
- Linux/macOS: zstd -d -c | tar -xf -
- Windows: zstd -d to an intermediate .tar, then the existing 7z
  -snld20 extraction (preserves symlink handling)

All filename references updated .tar -> .tar.zst. -f added to the two
-o invocations so re-runs overwrite instead of failing.

Also drop -vv from gclient sync; default verbosity is sufficient.

* ci: keep .tar extension for src cache (zstd content inside)

The sas-sidecar that issues Azure SAS tokens validates filenames against
/^v[0-9]+-[a-z\-]+-[a-f0-9]+\.(tar|tgz)$/ and is not easily redeployed,
so keep the .tar extension and decode zstd on restore. Windows
decompresses to a distinct intermediate (src_cache.tar) so input and
output don't collide.

* ci: log NTFS 8.3/lastaccess/Defender state before Windows cache extract

Temporary diagnostics to see whether 8.3 short-name generation is the
cause of the ~20 min tar extraction.

* ci: revert src-cache exclusion additions

The new exclusions (web_tests contents, jetstream, speedometer,
catapult test_data, regres, llvm-build, rust-toolchain) caused siso/RBE
cache misses — even data-only deps are part of action input hashes.
Revert to the original exclusion list and drop the corresponding
toolchain-fetch plumbing. zstd compression, the Windows tar cleanup,
and the -vv removal remain.

* ci: drop win_toolchain from src cache; remove NTFS diagnostics

The Windows src cache includes 14.6GB of depot_tools/win_toolchain —
7.3GB of MSVC/SDK doubled because tar captures both the vs_files.ciopfs
backing store and the live ciopfs mount at vs_files/. Every Windows
cache consumer already re-fetches this via vs_toolchain.py update
--force (fix-sync for build/publish, inline for gn-check/clang-tidy),
so the cached copy is never used.

Diagnostics removed — CI confirmed 8dot3, last-access, and Defender are
all already off on the AKS Windows nodes.

* ci: unmount ciopfs vs_files before removing win_toolchain

vs_files is a live ciopfs mount during the win-targeted checkout; rm -rf
fails with EBUSY until it's unmounted.

* ci: skip win_toolchain download during checkout instead of removing after

fusermount isn't on the checkout container, so the ciopfs mount can't be
torn down before rm. Setting DEPOT_TOOLS_WIN_TOOLCHAIN=0 makes the
win_toolchain hook a no-op (vs_toolchain.py:525-527), so there's no
download and no mount. All Windows consumers re-fetch it post-restore
anyway. The rm -rf stays as a safety net.

* ci: also set ELECTRON_DEPOT_TOOLS_WIN_TOOLCHAIN=0 for checkout sync

build.yml sets ELECTRON_DEPOT_TOOLS_WIN_TOOLCHAIN=1 at the job level for
the Windows checkout, which makes e d inject DEPOT_TOOLS_WIN_TOOLCHAIN=1
and override the inline =0. Need both: the ELECTRON_ var stops e d from
overriding, the plain one stops vs_toolchain.py from defaulting to 1.

* ci: extract Windows src cache with piped tar instead of 7z

7z takes ~20 min to extract the ~1.1M-entry tar regardless of size —
~1ms per entry of header parsing and path handling, single-threaded,
well under the 75k IOPS / 1000 MBps the ephemeral disk can do. Switch
to the same zstd -d | tar -xf - pipe used on Linux/macOS (via Git Bash
tar). No intermediate src_cache.tar, download deleted after extract.

The -snld20 flag was working around 7z's own "dangerous symlink"
refusal; GNU tar extracts symlinks as-is so it shouldn't be needed.

* ci: keep depot_tools/win_toolchain scripts in src cache

The rm -rf removed get_toolchain_if_necessary.py (a depot_tools source
file), breaking vs_toolchain.py update --force on restore.
DEPOT_TOOLS_WIN_TOOLCHAIN=0 on the sync already prevents the vs_files
download, so the rm was only removing scripts.

* ci: split src cache into 4 parallel-extractable shards

Windows tar extraction is ~1ms/entry for ~1.2M entries (~20 min)
regardless of tool, well under the 75k IOPS / 1000 MBps the D16lds_v5
ephemeral disk can do. Tar is a sequential stream so the only way to
parallelize is to split at creation time.

Shards (balanced by entry count, ~220-360k each):
  a: src/third_party/blink
  b: src/third_party/{dawn,electron_node,tflite,devtools-frontend}
  c: src/third_party (rest)
  d: src (excluding third_party)

DEPSHASH is now the raw hash; shard files are
v2-src-cache-shard-{a..d}-${DEPSHASH}.tar (all pass the sas-sidecar
filename regex). sas-token is now a JSON keyed by shard letter. All
restore paths extract the four shards in parallel with per-PID wait so
a failed shard aborts the step.

* Revert "ci: split src cache into 4 parallel-extractable shards"

This reverts commit 970574998b.
2026-04-05 17:56:03 -07:00

244 lines
9.9 KiB
YAML
Raw Blame History

# AUTOGENERATED FILE - DO NOT EDIT MANUALLY
# ONLY EDIT .github/workflows/pipeline-segment-electron-build.yml
name: Pipeline Segment - Electron Build
on:
workflow_call:
inputs:
environment:
description: using the production or testing environment
required: false
type: string
target-platform:
type: string
description: Platform to run on, can be macos, win or linux
required: true
target-arch:
type: string
description: Arch to build for, can be x64, arm64, ia32 or arm
required: true
target-variant:
type: string
description: Variant to build for, no effect on non-macOS target platforms. Can
be darwin, mas or all.
default: all
build-runs-on:
type: string
description: What host to run the build
required: true
build-container:
type: string
description: JSON container information for aks runs-on
required: false
default: '{"image":null}'
is-release:
description: Whether this build job is a release job
required: true
type: boolean
default: false
gn-build-type:
description: The gn build type - testing or release
required: true
type: string
default: testing
generate-symbols:
description: Whether or not to generate symbols
required: true
type: boolean
default: false
upload-to-storage:
description: Whether or not to upload build artifacts to external storage
required: true
type: string
default: "0"
is-asan:
description: Building the Address Sanitizer (ASan) Linux build
required: false
type: boolean
default: false
upload-out-gen-artifacts:
description: Whether to upload the src/gen artifacts
required: false
type: boolean
default: false
enable-ssh:
description: Enable SSH debugging
required: false
type: boolean
default: false
permissions: {}
concurrency:
group: electron-build-${{ inputs.target-platform }}-${{ inputs.target-arch
}}-${{ inputs.target-variant }}-${{ inputs.is-asan }}-${{
github.ref_protected == true && github.run_id || github.ref }}
cancel-in-progress: ${{ github.ref_protected != true }}
env:
CHROMIUM_GIT_COOKIE: ${{ secrets.CHROMIUM_GIT_COOKIE }}
CHROMIUM_GIT_COOKIE_WINDOWS_STRING: ${{ secrets.CHROMIUM_GIT_COOKIE_WINDOWS_STRING }}
DD_API_KEY: ${{ secrets.DD_API_KEY }}
ELECTRON_ARTIFACTS_BLOB_STORAGE: ${{ secrets.ELECTRON_ARTIFACTS_BLOB_STORAGE }}
ELECTRON_RBE_JWT: ${{ secrets.ELECTRON_RBE_JWT }}
SUDOWOODO_EXCHANGE_URL: ${{ secrets.SUDOWOODO_EXCHANGE_URL }}
SUDOWOODO_EXCHANGE_TOKEN: ${{ secrets.SUDOWOODO_EXCHANGE_TOKEN }}
GCLIENT_EXTRA_ARGS: ${{ inputs.target-platform == 'macos' &&
'--custom-var=checkout_mac=True --custom-var=host_os=mac' ||
inputs.target-platform == 'win' && '--custom-var=checkout_win=True' ||
'--custom-var=checkout_arm=True --custom-var=checkout_arm64=True' }}
ELECTRON_OUT_DIR: Default
ACTIONS_STEP_DEBUG: ${{ secrets.ACTIONS_STEP_DEBUG }}
jobs:
build:
defaults:
run:
shell: bash
runs-on: ${{ inputs.build-runs-on }}
permissions:
artifact-metadata: write
attestations: write
contents: read
id-token: write
container: ${{ fromJSON(inputs.build-container) }}
environment: ${{ inputs.environment }}
env:
TARGET_ARCH: ${{ inputs.target-arch }}
TARGET_PLATFORM: ${{ inputs.target-platform }}
steps:
- name: Create src dir
run: |
mkdir src
- name: Checkout Electron
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
with:
path: src/electron
fetch-depth: 0
ref: ${{ github.event.pull_request.head.sha }}
- name: Setup SSH Debugging
if: ${{ inputs.target-platform == 'macos' && (inputs.enable-ssh ||
env.ACTIONS_STEP_DEBUG == 'true') }}
uses: ./src/electron/.github/actions/ssh-debug
with:
tunnel: "true"
env:
CLOUDFLARE_TUNNEL_CERT: ${{ secrets.CLOUDFLARE_TUNNEL_CERT }}
CLOUDFLARE_TUNNEL_HOSTNAME: ${{ vars.CLOUDFLARE_TUNNEL_HOSTNAME }}
CLOUDFLARE_USER_CA_CERT: ${{ secrets.CLOUDFLARE_USER_CA_CERT }}
AUTHORIZED_USERS: ${{ secrets.SSH_DEBUG_AUTHORIZED_USERS }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Free up space (macOS)
if: ${{ inputs.target-platform == 'macos' }}
uses: ./src/electron/.github/actions/free-space-macos
- name: Check disk space after freeing up space
if: ${{ inputs.target-platform == 'macos' }}
run: df -h
- name: Setup Node.js/npm
if: ${{ inputs.target-platform == 'macos' }}
uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f
with:
node-version: 22.21.x
cache: yarn
cache-dependency-path: src/electron/yarn.lock
- name: Install Dependencies
uses: ./src/electron/.github/actions/install-dependencies
- name: Install AZCopy
if: ${{ inputs.target-platform == 'macos' }}
run: brew install azcopy
- name: Set GN_EXTRA_ARGS for Linux
if: ${{ inputs.target-platform == 'linux' }}
run: >
if [ "${{ inputs.target-arch }}" = "arm" ]; then
if [ "${{ inputs.is-release }}" = true ]; then
GN_EXTRA_ARGS='target_cpu="arm" build_tflite_with_xnnpack=false symbol_level=1'
else
GN_EXTRA_ARGS='target_cpu="arm" build_tflite_with_xnnpack=false'
fi
elif [ "${{ inputs.target-arch }}" = "arm64" ]; then
GN_EXTRA_ARGS='target_cpu="arm64" fatal_linker_warnings=false enable_linux_installer=false'
elif [ "${{ inputs.is-asan }}" = true ]; then
GN_EXTRA_ARGS='is_asan=true'
fi
echo "GN_EXTRA_ARGS=$GN_EXTRA_ARGS" >> $GITHUB_ENV
- name: Set Chromium Git Cookie
uses: ./src/electron/.github/actions/set-chromium-cookie
- name: Install Build Tools
uses: ./src/electron/.github/actions/install-build-tools
- name: Generate DEPS Hash
run: |
node src/electron/script/generate-deps-hash.js
DEPSHASH=v2-src-cache-$(cat src/electron/.depshash)
echo "DEPSHASH=$DEPSHASH" >> $GITHUB_ENV
echo "CACHE_PATH=$DEPSHASH.tar" >> $GITHUB_ENV
- name: Restore src cache via AZCopy
if: ${{ inputs.target-platform != 'linux' }}
uses: ./src/electron/.github/actions/restore-cache-azcopy
with:
target-platform: ${{ inputs.target-platform }}
- name: Restore src cache via AKS
if: ${{ inputs.target-platform == 'linux' }}
uses: ./src/electron/.github/actions/restore-cache-aks
- name: Checkout Electron
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
with:
path: src/electron
fetch-depth: 0
ref: ${{ github.event.pull_request.head.sha }}
- name: Fix Sync
if: ${{ inputs.target-platform != 'linux' }}
uses: ./src/electron/.github/actions/fix-sync
with:
target-platform: ${{ inputs.target-platform }}
env:
ELECTRON_DEPOT_TOOLS_DISABLE_LOG: true
- name: Init Build Tools
run: >
e init -f --root=$(pwd) --out=Default ${{ inputs.gn-build-type }}
--import ${{ inputs.gn-build-type }} --target-cpu ${{
inputs.target-arch }} --remote-build siso
- name: Run Electron Only Hooks
run: |
e d gclient runhooks --spec="solutions=[{'name':'src/electron','url':None,'deps_file':'DEPS','custom_vars':{'process_deps':False},'managed':False}]"
- name: Regenerate DEPS Hash
run: >
(cd src/electron && git checkout .) && node
src/electron/script/generate-deps-hash.js
echo "DEPSHASH=$(cat src/electron/.depshash)" >> $GITHUB_ENV
- name: Add CHROMIUM_BUILDTOOLS_PATH to env
run: echo "CHROMIUM_BUILDTOOLS_PATH=$(pwd)/src/buildtools" >> $GITHUB_ENV
- name: Free up space (macOS)
if: ${{ inputs.target-platform == 'macos' }}
uses: ./src/electron/.github/actions/free-space-macos
- name: Build Electron
if: ${{ inputs.target-platform != 'macos' || (inputs.target-variant == 'all' ||
inputs.target-variant == 'darwin') }}
uses: ./src/electron/.github/actions/build-electron
with:
target-arch: ${{ inputs.target-arch }}
target-platform: ${{ inputs.target-platform }}
artifact-platform: ${{ inputs.target-platform == 'macos' && 'darwin' ||
inputs.target-platform }}
is-release: ${{ inputs.is-release }}
generate-symbols: ${{ inputs.generate-symbols }}
upload-to-storage: ${{ inputs.upload-to-storage }}
is-asan: ${{ inputs.is-asan }}
upload-out-gen-artifacts: ${{ inputs.upload-out-gen-artifacts }}
- name: Set GN_EXTRA_ARGS for MAS Build
if: ${{ inputs.target-platform == 'macos' && (inputs.target-variant == 'all' ||
inputs.target-variant == 'mas') }}
run: |
echo "MAS_BUILD=true" >> $GITHUB_ENV
GN_EXTRA_ARGS='is_mas_build=true'
echo "GN_EXTRA_ARGS=$GN_EXTRA_ARGS" >> $GITHUB_ENV
- name: Build Electron (MAS)
if: ${{ inputs.target-platform == 'macos' && (inputs.target-variant == 'all' ||
inputs.target-variant == 'mas') }}
uses: ./src/electron/.github/actions/build-electron
with:
target-arch: ${{ inputs.target-arch }}
target-platform: ${{ inputs.target-platform }}
artifact-platform: mas
is-release: ${{ inputs.is-release }}
generate-symbols: ${{ inputs.generate-symbols }}
upload-to-storage: ${{ inputs.upload-to-storage }}
step-suffix: (mas)
Reference in New Issue View Git Blame Copy Permalink