mirror of
https://github.com/chromebrew/chromebrew.git
synced 2026-01-07 22:54:11 -05:00
Use grep better to speed up crew remove of packages. (#10309)
* use grep to speed up crew remove Signed-off-by: Satadru Pramanik <satadru@gmail.com> * bump version Signed-off-by: Satadru Pramanik <satadru@gmail.com> * adjust logic Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add back CREW_ESSENTIAL_FILES to handle files like libC.so.6 Signed-off-by: Satadru Pramanik <satadru@gmail.com> * adjust CREW_ESSENTIAL_FILES logic Signed-off-by: Satadru Pramanik <satadru@gmail.com> * fix package_files_that_overlap Signed-off-by: Satadru Pramanik <satadru@gmail.com> * cleanup output of package_files_that_overlap Signed-off-by: Satadru Pramanik <satadru@gmail.com> * debug Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Back out CREW_ESSENTIAL_FILES change, and use Package.load_package('pkg.rb').get_deps_list to find dependent packages of CREW_ESSENTIAL_PACKAGES in commands/remove.rb, and also fix Package.load_package('pkg.rb').get_deps_list to enable using it from commands/remove.rb Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust removal message in crew. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust some punctuation and capitalization. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * fix pkg_file path detection Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Only use CREW_LOCAL_REPO_ROOT file if it exists. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * suggested changes Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Fix ESSENTIAL_PACKAGE dependency expansion. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Fix essential_deps logic. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust remove logic to return quietly if in_upgrade, else exit 1, and also properly figure out essential_packages list. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Fix docopt so options like '-d' work. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add suggested changes for testing, add testing file for remove.rb, also in testing. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * fixup Signed-off-by: Satadru Pramanik <satadru@gmail.com> * more testing Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Move recursive package function to package.rb Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust function name. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * simplify Signed-off-by: Satadru Pramanik <satadru@gmail.com> * simplify Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Use File.mtime for @last_update_check as per suggestion by @Zopolis4. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Move essential package determination to crew:generate_compatible. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Move load_json and save_json to package_utils and just use symbols for the json array in remove and package_utils. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Unify json usage globally. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Fix one remove test. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * lint Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Update remove.rb test. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * cleanup Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Adjust remove test. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Try to use new Unit Tests... Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Fix installsh: git config --local commands cannot be run unless the git repo is already setup. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Move json functions to package_utils. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Fix docopt for real, add docopt unit test, have install.sh generate ruby gem filelists just like a regular buildsystems/ruby gem install. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Use CREW_META_PATH in install.sh. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Cleanup remove.rb tests. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add 'crew list essential'. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Add unit test for list command. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Cleanup remove.rb test. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Cleanup wording. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Move print_deps_tree to lib/package.rb. Signed-off-by: Satadru Pramanik <satadru@gmail.com> * lint Signed-off-by: Satadru Pramanik <satadru@gmail.com> * Return changes from #10317 Signed-off-by: Satadru Pramanik <satadru@gmail.com> --------- Signed-off-by: Satadru Pramanik <satadru@gmail.com>
This commit is contained in:
Satadru Pramanik, DO, MPH, MEng
committed by
GitHub
GitHub
parent
0115387eb6
commit
c6b102fece
@@ -6,7 +6,7 @@ require_relative '../lib/package'
|
||||
require_relative '../lib/package_utils'
|
||||
|
||||
class Command
|
||||
def self.list(available, installed, compatible, incompatible, verbose)
|
||||
def self.list(available, compatible, incompatible, essential, installed, verbose)
|
||||
device_json = JSON.load_file(File.join(CREW_CONFIG_PATH, 'device.json'), symbolize_names: true)
|
||||
installed_packages = {}
|
||||
device_json[:installed_packages].each do |package|
|
||||
@@ -20,19 +20,6 @@ class Command
|
||||
pkg = Package.load_package(filename)
|
||||
puts pkg_name if PackageUtils.compatible?(pkg)
|
||||
end
|
||||
elsif installed
|
||||
if verbose
|
||||
installed_packages['======='] = '======='
|
||||
installed_packages['Package'] = 'Version'
|
||||
first_col_width = installed_packages.keys.max { |a, b| a.size <=> b.size }.size
|
||||
installed_packages.sort.to_h.each do |package, version|
|
||||
puts "#{package.ljust(first_col_width)} #{version}".lightgreen
|
||||
end
|
||||
else
|
||||
installed_packages.each_key do |package|
|
||||
puts package.lightgreen
|
||||
end
|
||||
end
|
||||
elsif compatible
|
||||
Dir["#{CREW_PACKAGES_PATH}/*.rb"].each do |filename|
|
||||
pkg_name = File.basename(filename, '.rb')
|
||||
@@ -46,6 +33,21 @@ class Command
|
||||
pkg = Package.load_package(filename)
|
||||
puts pkg_name.lightred unless PackageUtils.compatible?(pkg)
|
||||
end
|
||||
elsif essential
|
||||
puts device_json[:essential_deps].join("\n")
|
||||
elsif installed
|
||||
if verbose
|
||||
installed_packages['======='] = '======='
|
||||
installed_packages['Package'] = 'Version'
|
||||
first_col_width = installed_packages.keys.max { |a, b| a.size <=> b.size }.size
|
||||
installed_packages.sort.to_h.each do |package, version|
|
||||
puts "#{package.ljust(first_col_width)} #{version}".lightgreen
|
||||
end
|
||||
else
|
||||
installed_packages.each_key do |package|
|
||||
puts package.lightgreen
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
require 'fileutils'
|
||||
require 'json'
|
||||
require_relative '../lib/const'
|
||||
require_relative '../lib/package'
|
||||
require_relative '../lib/package_utils'
|
||||
|
||||
class Command
|
||||
def self.remove(pkg, verbose)
|
||||
device_json = JSON.load_file(File.join(CREW_CONFIG_PATH, 'device.json'))
|
||||
device_json = PackageUtils.load_json
|
||||
|
||||
# Make sure the package is actually installed before we attempt to remove it.
|
||||
unless PackageUtils.installed?(pkg.name)
|
||||
@@ -13,10 +13,23 @@ class Command
|
||||
return
|
||||
end
|
||||
|
||||
# Don't remove any of the packages ruby (and thus crew) needs to run.
|
||||
if CREW_ESSENTIAL_PACKAGES.include?(pkg.name)
|
||||
puts "Refusing to remove essential package #{pkg.name}.".lightred
|
||||
return
|
||||
# Determine dependencies of packages in CREW_ESSENTIAL_PACKAGES and
|
||||
# their dependencies, as those are needed for ruby and crew to run,
|
||||
# and thus should not be removed.
|
||||
# essential_deps = recursive_deps(CREW_ESSENTIAL_PACKAGES)
|
||||
essential_deps = device_json[:essential_deps]
|
||||
crewlog "Essential Deps are #{essential_deps}."
|
||||
if essential_deps.include?(pkg.name)
|
||||
return if pkg.in_upgrade
|
||||
|
||||
puts <<~ESSENTIAL_PACKAGE_WARNING_EOF.gsub(/^(?=\w)/, ' ').lightred
|
||||
#{pkg.name.capitalize} is considered an essential package needed for
|
||||
Chromebrew to function and thus cannot be removed.
|
||||
ESSENTIAL_PACKAGE_WARNING_EOF
|
||||
|
||||
# Exit with failure if attempt to remove an essential package
|
||||
# is made.
|
||||
exit 1
|
||||
end
|
||||
|
||||
# Perform any operations required prior to package removal.
|
||||
@@ -34,17 +47,40 @@ class Command
|
||||
# Remove the files and directories installed by the package.
|
||||
unless pkg.is_fake?
|
||||
Dir.chdir CREW_CONFIG_PATH do
|
||||
# Remove all files installed by the package.
|
||||
# Remove all files installed by the package in CREW_PREFIX and
|
||||
# HOME.
|
||||
# Exceptions:
|
||||
# 1. The file exists in another installed package.
|
||||
# 2. The file is in one of the filelists for packages in
|
||||
# CREW_ESSENTIAL_FILES, or a dependendent package of
|
||||
# CREW_ESSENTIAL_PACKAGES.
|
||||
flist = File.join(CREW_META_PATH, "#{pkg.name}.filelist")
|
||||
if File.file?(flist)
|
||||
File.foreach(flist, chomp: true) do |line|
|
||||
next unless line.start_with?(CREW_PREFIX)
|
||||
if system("grep --exclude #{pkg.name}.filelist -Fxq '#{line}' ./meta/*.filelist")
|
||||
puts "#{line} is in another package. It will not be removed during the removal of #{pkg.name}.".orange
|
||||
else
|
||||
puts "Removing file #{line}".yellow if verbose
|
||||
FileUtils.remove_file line, exception: false
|
||||
end
|
||||
# When searching for files to delete we exclude the files from
|
||||
# all packages and dependent packages of CREW_ESSENTIAL_PACKAGES.
|
||||
essential_deps_exclude_froms = essential_deps.map { |i| File.file?("#{File.join(CREW_META_PATH, i.to_s)}.filelist") ? "--exclude-from=#{File.join(CREW_META_PATH, i.to_s)}.filelist" : '' }.join(' ')
|
||||
|
||||
# When making a list of all files from crew filelists we again
|
||||
# ignore all files from packages and dependent packages of
|
||||
# CREW_ESSENTIAL_PACKAGES.
|
||||
essential_deps_excludes = essential_deps.map { |i| File.file?("#{File.join(CREW_META_PATH, i.to_s)}.filelist") ? "--exclude=#{File.join(CREW_META_PATH, i.to_s)}.filelist" : '' }.join(' ')
|
||||
|
||||
package_files = `grep -h #{essential_deps_exclude_froms} \"^#{CREW_PREFIX}\\|^#{HOME}\" #{flist}`.split("\n").uniq.sort
|
||||
all_other_files = `grep -h --exclude #{pkg.name}.filelist #{essential_deps_excludes} \"^#{CREW_PREFIX}\\|^#{HOME}\" #{CREW_META_PATH}/*.filelist`.split("\n").uniq.sort
|
||||
|
||||
# We want the difference of these arrays.
|
||||
unique_to_package_files = package_files - all_other_files
|
||||
|
||||
# We want the intersection of these arrays.
|
||||
package_files_that_overlap = all_other_files & package_files
|
||||
|
||||
unless package_files_that_overlap.empty?
|
||||
puts "The following file(s) in other packages will not be deleted during the removal of #{pkg.name}:".orange
|
||||
puts package_files_that_overlap.join("\n").orange
|
||||
end
|
||||
unique_to_package_files.each do |file|
|
||||
puts "Removing file #{file}".yellow if CREW_VERBOSE
|
||||
FileUtils.remove_file file, exception: false
|
||||
end
|
||||
FileUtils.remove_file flist
|
||||
end
|
||||
@@ -64,10 +100,10 @@ class Command
|
||||
|
||||
# Remove the package from the list of installed packages in device.json.
|
||||
puts "Removing package #{pkg.name} from device.json".yellow if verbose
|
||||
device_json['installed_packages'].delete_if { |entry| entry['name'] == pkg.name }
|
||||
device_json[:installed_packages].delete_if { |entry| entry[:name] == pkg.name }
|
||||
|
||||
# Update device.json with our changes.
|
||||
save_json(device_json)
|
||||
PackageUtils.save_json(device_json)
|
||||
|
||||
# Perform any operations required after package removal.
|
||||
pkg.postremove
|
||||
|
||||
Reference in New Issue
Block a user