Enable Cached Building on GitHub Actions— webkit2gtk_4_1 → 2.50.1 (#13001)

* Refactor and update webkit2gtk_4_1

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Add arm patch.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust env options

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Add x86_64 build.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust build settings.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust arm build options.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* lint

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust arm build.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* lint

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust g++ in build.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Add cache_build plumbing.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Add NESTED_CI detection plumbing to see if we are running in a container on GitHub Actions.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust download options for cached builds.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust timed kill to kill cmake.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust triggering of cache_build.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Cleanup output.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Update cached build hash verification.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Rubyize #{build_cachefile}.sha256 write.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust documentation of cache_build trigger.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Also kill all ruby processes after finishing cache_build.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Make cached build download info more useful.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Add --regenerate-filelist option.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Fix downloader.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Try newer git commit.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* remove arm patch.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust hash checking for build downloads.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Add message for checksum calculation since that can take a while.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Add cached build restart code block.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Add max build time to build workflow.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* fixup buildsystems

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Set workflow max build time to 5.5 hours.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Indicate architectures for build in build workflow title.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust cached build uploading.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust workflow naming.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust installs after build.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust cached build logic.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* webkit => 2.50.1

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust zstd options.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Move CREW_CACHE_DIR to /tmp in GitHub Action containers.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust build cache location.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* revert crew const variable changes.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust CREW_KERNEL_VERSION for CI usage.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Exclude @pkg.no_source_build? packages from cached builds.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Always create CREW_CACHE_DIR.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Clean up remnant @extract_dir folders from download command.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Adjust permissions in workflow.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Sync up workflows.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* lint

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Add x86_64 binaries

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Cleanup workflows.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* Do not use build cache if package binary exists.

Signed-off-by: Satadru Pramanik <satadru@gmail.com>

* webkit: Package File Update Run on linux/amd64 container.

* webkit: Package File Update Run on linux/arm/v7 container.

---------

Signed-off-by: Satadru Pramanik <satadru@gmail.com>
Co-authored-by: satmandu <satmandu@users.noreply.github.com>

lib/downloader.rb

@@ -1,5 +1,5 @@
-require 'digest/sha2'
 require 'io/console'
+require 'open3'
 require 'uri'
 require_relative 'const'
 require_relative 'color'
@@ -25,7 +25,7 @@ rescue RuntimeError => e
   end
 end
 
-def downloader(url, sha256sum, filename = File.basename(url), verbose: false)
+def downloader(url, sha256sum, filename = File.basename(url), no_update_hash: false, verbose: false)
   # downloader: wrapper for all Chromebrew downloaders (`net/http`,`curl`...)
   # Usage: downloader <url>, <sha256sum>, <filename::optional>, <verbose::optional>
   #
@@ -37,6 +37,9 @@ def downloader(url, sha256sum, filename = File.basename(url), verbose: false)
   puts "downloader(#{url}, #{sha256sum}, #{filename}, #{verbose})" if verbose
   uri = URI(url)
 
+  # Make sure the destination dir for the filename exists.
+  FileUtils.mkdir_p File.dirname(filename)
+
   if CREW_USE_CURL || !ENV['CREW_DOWNLOADER'].to_s.empty?
     # force using external downloader if either CREW_USE_CURL or ENV['CREW_DOWNLOADER'] is set
     puts "external_downloader(#{uri}, #{filename}, #{verbose})" if verbose
@@ -61,11 +64,15 @@ def downloader(url, sha256sum, filename = File.basename(url), verbose: false)
     end
   end
 
-  # verify with given checksum
-  calc_sha256sum = Digest::SHA256.hexdigest(File.read(filename))
+  # Verify with given checksum, using the external sha256sum binary so
+  # we do not load the entire file into ruby's process, which throws
+  # errors with large files on 32-bit architectures.
+  puts "Calculating checksum for #{filename}...".lightblue
+  sha256sum_out, _stderr, _status = Open3.capture3("sha256sum #{filename}")
+  calc_sha256sum = sha256sum_out.split[0]
 
   unless (sha256sum =~ /^SKIP$/i) || (calc_sha256sum == sha256sum)
-    if CREW_FORCE
+    if CREW_FORCE && !no_update_hash
       pkg_name = @pkg_name.blank? ? name : @pkg_name
       puts "Updating checksum for #{filename}".lightblue
       puts "from #{sha256sum} to #{calc_sha256sum}".lightblue