Enable Cached Building on GitHub Actions— webkit2gtk_4_1 → 2.50.1 (#13001)

* Refactor and update webkit2gtk_4_1 Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add arm patch. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust env options Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add x86_64 build. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust build settings. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust arm build options. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * lint Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust arm build. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * lint Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust g++ in build. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add cache_build plumbing. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add NESTED_CI detection plumbing to see if we are running in a container on GitHub Actions. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust download options for cached builds. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust timed kill to kill cmake. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust triggering of cache_build. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Cleanup output. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Update cached build hash verification. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Rubyize #{build_cachefile}.sha256 write. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust documentation of cache_build trigger. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Also kill all ruby processes after finishing cache_build. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Make cached build download info more useful. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add --regenerate-filelist option. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Fix downloader. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Try newer git commit. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * remove arm patch. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust hash checking for build downloads. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add message for checksum calculation since that can take a while. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add cached build restart code block. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add max build time to build workflow. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * fixup buildsystems Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Set workflow max build time to 5.5 hours. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Indicate architectures for build in build workflow title. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust cached build uploading. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust workflow naming. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust installs after build. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust cached build logic. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * webkit => 2.50.1 Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust zstd options. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Move CREW_CACHE_DIR to /tmp in GitHub Action containers. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust build cache location. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * revert crew const variable changes. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust CREW_KERNEL_VERSION for CI usage. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Exclude @pkg.no_source_build? packages from cached builds. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Always create CREW_CACHE_DIR. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Clean up remnant @extract_dir folders from download command. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust permissions in workflow. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Sync up workflows. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * lint Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add x86_64 binaries Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Cleanup workflows. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Do not use build cache if package binary exists. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * webkit: Package File Update Run on linux/amd64 container. * webkit: Package File Update Run on linux/arm/v7 container. --------- Signed-off-by: Satadru Pramanik <satadru@gmail.com> Co-authored-by: satmandu <satmandu@users.noreply.github.com>
2026-01-08 23:18:10 -05:00 · 2025-10-14 05:06:32 -04:00
parent ed8f069a77
commit 6b35c08b2e
11 changed files with 368 additions and 148 deletions
--- a/lib/buildsystems/cmake.rb
+++ b/lib/buildsystems/cmake.rb
@@ -3,17 +3,25 @@ require_relative '../require_gem'
 require_relative '../report_buildsystem_methods'

 class CMake < Package
-  property :cmake_build_extras, :cmake_build_relative_dir, :cmake_install_extras, :cmake_options, :pre_cmake_options
+  property :cmake_build_extras, :cmake_build_relative_dir, :cmake_install_extras, :cmake_options, :cmake_pre_cache_build_extras, :pre_cmake_options

-  def self.build
+  def self.prebuild_config_and_report
    @cmake_build_relative_dir ||= '.'
    @crew_cmake_options = @no_lto ? CREW_CMAKE_OPTIONS.gsub('-ffat-lto-objects', '').gsub('-flto=auto', '').sub('-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=TRUE', '') : CREW_CMAKE_OPTIONS

    extend ReportBuildsystemMethods

    print_buildsystem_methods
+  end

-    system "#{@pre_cmake_options} cmake -S #{@cmake_build_relative_dir} -B #{@cmake_build_relative_dir}/builddir -G Ninja #{@crew_cmake_options} #{@cmake_options}"
+  def self.pre_cached_build
+    Dir.chdir(@cmake_build_relative_dir) do
+      @cmake_pre_cache_build_extras&.call
+    end
+  end
+
+  def self.build
+    system "#{@pre_cmake_options} cmake -S #{@cmake_build_relative_dir} -B #{@cmake_build_relative_dir}/builddir -G Ninja #{@crew_cmake_options} #{@cmake_options}" unless File.file?("#{@cmake_build_relative_dir}/build.ninja")
    system "#{CREW_PREFIX}/bin/jobserver_pool.py -j #{CREW_NPROC} #{CREW_NINJA} -C #{@cmake_build_relative_dir}/builddir"
    @cmake_build_extras&.call
  end
--- a/lib/buildsystems/meson.rb
+++ b/lib/buildsystems/meson.rb
@@ -3,9 +3,9 @@ require_relative '../require_gem'
 require_relative '../report_buildsystem_methods'

 class Meson < Package
-  property :meson_build_relative_dir, :meson_options, :meson_build_extras, :meson_install_extras, :pre_meson_options
+  property :meson_build_relative_dir, :meson_options, :meson_build_extras, :meson_install_extras, :meson_pre_cache_build_extras, :pre_meson_options

-  def self.build
+  def self.prebuild_config_and_report
    @meson_build_relative_dir ||= '.'
    @crew_meson_options = @no_mold ? CREW_MESON_OPTIONS.gsub('-fuse-ld=mold', '') : CREW_MESON_OPTIONS
    @crew_meson_options.gsub!('-Db_lto=true', '-Db_lto=false').gsub!('-flto=auto', '') if @no_lto
@@ -13,8 +13,16 @@ class Meson < Package
    extend ReportBuildsystemMethods

    print_buildsystem_methods
+  end

-    system "#{@pre_meson_options} meson setup #{@crew_meson_options} #{@meson_options} #{@meson_build_relative_dir}/builddir #{@meson_build_relative_dir}"
+  def self.pre_cached_build
+    Dir.chdir @meson_build_relative_dir do
+      @meson_pre_cache_build_extras&.call
+    end
+  end
+
+  def self.build
+    system "#{@pre_meson_options} meson setup #{@crew_meson_options} #{@meson_options} #{@meson_build_relative_dir}/builddir #{@meson_build_relative_dir}" unless File.file?("#{@meson_build_relative_dir}/build.ninja")
    system "meson configure --no-pager #{@meson_build_relative_dir}/builddir"
    system "#{CREW_PREFIX}/bin/jobserver_pool.py -j #{CREW_NPROC} #{CREW_NINJA} -C #{@meson_build_relative_dir}/builddir"
    @meson_build_extras&.call
--- a/lib/const.rb
+++ b/lib/const.rb
@@ -89,16 +89,27 @@ end

 # Use sane minimal defaults if in container and no override specified.
 CREW_KERNEL_VERSION ||=
-  if CREW_IN_CONTAINER && ENV.fetch('CREW_KERNEL_VERSION', nil).nil?
+  if (CREW_IN_CONTAINER && ENV.fetch('CREW_KERNEL_VERSION', nil).nil?) || ENV['CI']
    ARCH.eql?('i686') ? '3.8' : '6.12'
  else
    ENV.fetch('CREW_KERNEL_VERSION', Etc.uname[:release].rpartition('.').first)
  end

 # Local constants for contributors.
+CREW_CACHE_DIR          ||= ENV.fetch('CREW_CACHE_DIR', "#{HOME}/.cache/crewcache") unless defined?(CREW_CACHE_DIR)
+CREW_CACHE_FAILED_BUILD ||= ENV.fetch('CREW_CACHE_FAILED_BUILD', false) unless defined?(CREW_CACHE_FAILED_BUILD)
+CREW_CACHE_BUILD        ||= ENV.fetch('CREW_CACHE_BUILD', false) unless defined?(CREW_CACHE_BUILD)
 CREW_LOCAL_REPO_ROOT ||= `git rev-parse --show-toplevel 2>/dev/null`.chomp
 CREW_LOCAL_BUILD_DIR ||= "#{CREW_LOCAL_REPO_ROOT}/release/#{ARCH}"
+CREW_MAX_BUILD_TIME  ||= ENV.fetch('CREW_MAX_BUILD_TIME', '19800') unless defined?(CREW_MAX_BUILD_TIME) # GitHub Action containers are killed after 6 hours, so set to 5.5 hours.
 CREW_GITLAB_PKG_REPO ||= 'https://gitlab.com/api/v4/projects/26210301/packages'
+GITLAB_TOKEN ||= ENV.fetch('GITLAB_TOKEN', nil) unless defined?(GITLAB_TOKEN)
+GITLAB_TOKEN_USERNAME ||= ENV.fetch('GITLAB_TOKEN_USERNAME', nil) unless defined?(GITLAB_TOKEN_USERNAME)
+CREW_GITLAB_TOKEN_LABEL ||= if GITLAB_TOKEN.nil?
+                              ''
+                            else
+                              (GITLAB_TOKEN.split('-').first == 'glpat' ? 'PRIVATE-TOKEN' : 'DEPLOY-TOKEN')
+                            end

 CREW_LIB_PREFIX       ||= File.join(CREW_PREFIX, ARCH_LIB)
 CREW_MAN_PREFIX       ||= File.join(CREW_PREFIX, 'share/man')
@@ -123,9 +134,6 @@ CREW_DEST_MUSL_PREFIX ||= File.join(CREW_DEST_DIR, CREW_MUSL_PREFIX)
 MUSL_LIBC_VERSION     ||= File.executable?("#{CREW_MUSL_PREFIX}/lib/libc.so") ? `#{CREW_MUSL_PREFIX}/lib/libc.so 2>&1`[/\bVersion\s+\K\S+/] : nil unless defined?(MUSL_LIBC_VERSION)

 CREW_DEST_HOME          ||= File.join(CREW_DEST_DIR, HOME)
-CREW_CACHE_DIR          ||= ENV.fetch('CREW_CACHE_DIR', "#{HOME}/.cache/crewcache") unless defined?(CREW_CACHE_DIR)
-CREW_CACHE_BUILD        ||= ENV.fetch('CREW_CACHE_BUILD', false) unless defined?(CREW_CACHE_BUILD)
-CREW_CACHE_FAILED_BUILD ||= ENV.fetch('CREW_CACHE_FAILED_BUILD', false) unless defined?(CREW_CACHE_FAILED_BUILD)
 CREW_NO_GIT             ||= ENV.fetch('CREW_NO_GIT', false) unless defined?(CREW_NO_GIT)
 CREW_UNATTENDED         ||= ENV.fetch('CREW_UNATTENDED', false) unless defined?(CREW_UNATTENDED)

@@ -146,7 +154,7 @@ CREW_NPROC ||=
 # Set following as boolean if environment variables exist.
 # Timeout for agree questions in package.rb:
 CREW_AGREE_TIMEOUT_SECONDS           ||= ENV.fetch('CREW_AGREE_TIMEOUT_SECONDS', 10).to_i unless defined?(CREW_AGREE_TIMEOUT_SECONDS)
-CREW_CACHE_ENABLED                   ||= ENV.fetch('CREW_CACHE_ENABLED', false) unless defined?(CREW_CACHE_ENABLED)
+CREW_CACHE_ENABLED                   ||= ENV.fetch('CREW_CACHE_ENABLED', CREW_CACHE_FAILED_BUILD) unless defined?(CREW_CACHE_ENABLED)
 CREW_CONFLICTS_ONLY_ADVISORY         ||= ENV.fetch('CREW_CONFLICTS_ONLY_ADVISORY', false) unless defined?(CREW_CONFLICTS_ONLY_ADVISORY)
 # or use conflicts_ok
 CREW_DISABLE_ENV_OPTIONS             ||= ENV.fetch('CREW_DISABLE_ENV_OPTIONS', false) unless defined?(CREW_DISABLE_ENV_OPTIONS)
@@ -455,17 +463,17 @@ CREW_DOCOPT ||= <<~DOCOPT
    crew download [options] [-s|--source] [-v|--verbose] <name> ...
    crew files [options] <name> ...
    crew help [options] [<command>] [-v|--verbose] [<subcommand>]
-    crew install [options] [-f|--force] [-k|--keep] [-s|--source] [-S|--recursive-build] [-v|--verbose] <name> ...
+    crew install [options] [-f|--force] [-k|--keep] [--regenerate-filelist] [-s|--source] [-S|--recursive-build] [-v|--verbose] <name> ...
    crew list [options] [-v|--verbose] (available|compatible|incompatible|essential|installed)
    crew postinstall [options] [-v|--verbose] <name> ...
    crew prop [options] [<property>]
-    crew reinstall [options] [-f|--force] [-k|--keep] [-s|--source] [-S|--recursive-build] [-v|--verbose] <name> ...
+    crew reinstall [options] [-f|--force] [-k|--keep] [-s|--source] [--regenerate-filelist] [-S|--recursive-build] [-v|--verbose] <name> ...
    crew remove [options] [-f|--force] [-v|--verbose] <name> ...
    crew search [options] [-v|--verbose] <name> ...
    crew sysinfo [options] [-v|--verbose]
    crew update [options] [-v|--verbose]
    crew update_package_file [options] [-v|--verbose] [<name> ...]
-    crew upgrade [options] [-f|--force] [-k|--keep] [-s|--source] [-v|--verbose] [<name> ...]
+    crew upgrade [options] [-f|--force] [-k|--keep] [--regenerate-filelist] [-s|--source] [-v|--verbose] [<name> ...]
    crew upload [options] [-f|--force] [-v|--verbose] [<name> ...]
    crew upstream [options] [-j|--json|-u|--update-package-files|-v|--verbose] <name> ...
    crew version [options] [<name>]
@@ -482,6 +490,7 @@ CREW_DOCOPT ||= <<~DOCOPT
    -L --license               Display the crew license.
    -s --source                Build or download from source even if pre-compiled binary exists.
    -S --recursive-build       Build from source, including all dependencies, even if pre-compiled binaries exist.
+    --regenerate-filelist      Force regeneration of package filelists on install.
    -t --tree                  Print dependencies in a tree-structure format.
    -u --update-package-files  Attempt to update the package version.
    -v --verbose               Show extra information.
--- a/lib/downloader.rb
+++ b/lib/downloader.rb
@@ -1,5 +1,5 @@
-require 'digest/sha2'
 require 'io/console'
+require 'open3'
 require 'uri'
 require_relative 'const'
 require_relative 'color'
@@ -25,7 +25,7 @@ rescue RuntimeError => e
  end
 end

-def downloader(url, sha256sum, filename = File.basename(url), verbose: false)
+def downloader(url, sha256sum, filename = File.basename(url), no_update_hash: false, verbose: false)
  # downloader: wrapper for all Chromebrew downloaders (`net/http`,`curl`...)
  # Usage: downloader <url>, <sha256sum>, <filename::optional>, <verbose::optional>
  #
@@ -37,6 +37,9 @@ def downloader(url, sha256sum, filename = File.basename(url), verbose: false)
  puts "downloader(#{url}, #{sha256sum}, #{filename}, #{verbose})" if verbose
  uri = URI(url)

+  # Make sure the destination dir for the filename exists.
+  FileUtils.mkdir_p File.dirname(filename)
+
  if CREW_USE_CURL || !ENV['CREW_DOWNLOADER'].to_s.empty?
    # force using external downloader if either CREW_USE_CURL or ENV['CREW_DOWNLOADER'] is set
    puts "external_downloader(#{uri}, #{filename}, #{verbose})" if verbose
@@ -61,11 +64,15 @@ def downloader(url, sha256sum, filename = File.basename(url), verbose: false)
    end
  end

-  # verify with given checksum
-  calc_sha256sum = Digest::SHA256.hexdigest(File.read(filename))
+  # Verify with given checksum, using the external sha256sum binary so
+  # we do not load the entire file into ruby's process, which throws
+  # errors with large files on 32-bit architectures.
+  puts "Calculating checksum for #{filename}...".lightblue
+  sha256sum_out, _stderr, _status = Open3.capture3("sha256sum #{filename}")
+  calc_sha256sum = sha256sum_out.split[0]

  unless (sha256sum =~ /^SKIP$/i) || (calc_sha256sum == sha256sum)
-    if CREW_FORCE
+    if CREW_FORCE && !no_update_hash
      pkg_name = @pkg_name.blank? ? name : @pkg_name
      puts "Updating checksum for #{filename}".lightblue
      puts "from #{sha256sum} to #{calc_sha256sum}".lightblue
--- a/lib/package.rb
+++ b/lib/package.rb
@@ -26,35 +26,77 @@ def agree_with_default(yes_or_no_question_msg, character = nil, default:)
 end

 class Package
-  boolean_property :arch_flags_override, :conflicts_ok, :git_clone_deep,
-                   :git_fetchtags, :gem_compile_needed, :gnome,
-                   :ignore_updater, :is_fake, :is_musl, :is_static,
-                   :no_binaries_needed, :no_compile_needed,
-                   :no_compress, :no_env_options, :no_fhs, :no_filefix,
-                   :no_git_submodules, :no_links, :no_lto, :no_mold,
-                   :no_patchelf, :no_shrink, :no_source_build,
-                   :no_strip, :no_upstream_update, :no_zstd, :patchelf,
-                   :prerelease, :print_source_bashrc, :run_tests
+  boolean_property :arch_flags_override,
+                   :cache_build, # Cache build to gitlab.
+                   :conflicts_ok,
+                   :git_clone_deep,
+                   :git_fetchtags,
+                   :gem_compile_needed,
+                   :gnome,
+                   :ignore_updater,
+                   :is_fake,
+                   :is_musl,
+                   :is_static,
+                   :no_binaries_needed,
+                   :no_compile_needed,
+                   :no_compress,
+                   :no_env_options,
+                   :no_fhs,
+                   :no_filefix,
+                   :no_git_submodules,
+                   :no_links,
+                   :no_lto,
+                   :no_mold,
+                   :no_patchelf,
+                   :no_shrink,
+                   :no_source_build,
+                   :no_strip,
+                   :no_upstream_update,
+                   :no_zstd,
+                   :patchelf,
+                   :prerelease,
+                   :print_source_bashrc,
+                   :run_tests

-  property :description, :homepage, :version, :license, :compatibility,
-           :binary_compression, :binary_url, :binary_sha256, :source_url, :source_sha256,
-           :git_branch, :git_hashtag, :max_glibc, :min_glibc
+  property :description,
+           :homepage,
+           :version,
+           :license,
+           :compatibility,
+           :binary_compression,
+           :binary_url,
+           :binary_sha256,
+           :source_url,
+           :source_sha256,
+           :git_branch,
+           :git_hashtag,
+           :max_glibc,
+           :min_glibc

-  create_placeholder :preflight,   # Function for checks to see if install should occur.
-                     :patch,       # Function to perform patch operations prior to build from source.
-                     :prebuild,    # Function to perform pre-build operations prior to build from source.
-                     :build,       # Function to perform build from source.
-                     :postbuild,   # Function to perform post-build for both source build and binary distribution.
-                     :check,       # Function to perform check from source build. (executes only during `crew build`)
-                     :preinstall,  # Function to perform pre-install operations prior to install.
-                     :install,     # Function to perform install from source build.
-                     :postinstall, # Function to perform post-install for both source build and binary distribution.
-                     :preremove,   # Function to perform prior to package removal.
-                     :remove,      # Function to remove package.
-                     :postremove   # Function to perform after package removal.
+  create_placeholder :preflight,       # Function for checks to see if install should occur.
+                     :patch,           # Function to perform patch operations prior to build from source.
+                     :pre_cache_build, # Function to perform pre-build operations prior to starting & restarting a cached build.
+                     :prebuild,        # Function to perform pre-build operations prior to build from source.
+                     :prebuild_config_and_report, # Function to add some reporting for buildsystems.
+                     :build,           # Function to perform build from source.
+                     :postbuild,       # Function to perform post-build for both source build and binary distribution.
+                     :check,           # Function to perform check from source build. (executes only during `crew build`)
+                     :preinstall,      # Function to perform pre-install operations prior to install.
+                     :install,         # Function to perform install from source build.
+                     :postinstall,     # Function to perform post-install for both source build and binary distribution.
+                     :preremove,       # Function to perform prior to package removal.
+                     :remove,          # Function to remove package.
+                     :postremove       # Function to perform after package removal.

  class << self
-    attr_accessor :build_from_source, :built, :cached_build, :in_build, :in_install, :in_upgrade, :missing_binaries, :name
+    attr_accessor :build_from_source,
+                  :built,
+                  :cached_build,
+                  :in_build,
+                  :in_install,
+                  :in_upgrade,
+                  :missing_binaries,
+                  :name
  end

  def self.agree_default_no(message = nil)