Troubleshooting Patch Management issues - Server Automation Documentation

Unable to download shavlik metadata

The shavlik metadata may not be downloaded due to the following reasons:

• Files are not available at the shavlik site
• Firewall has blocked access to shavlik URLs

For more information about this issue, you can refer to the Error occurred while downloading shavlik metadata knowledge article in the BMC knowledge base.

Steps for debugging the shavlik downloader

To enable DEBUG logging, add the following text to appserver.cf file:

#Downloader debug
log4j.logger.com.bladelogic.model.job.compliance.patch.ShavlikResult=DEBUG
log4j.logger.com.bladelogic.app.util.DownloadServer=DEBUG

Product Mappings are subject to change due to updates by Microsoft (and therefore Shavlik). If the updates occur before BMC Server Automation has shipped updates in the product, you can use a Windows Filter Configuration File to update the mappings used by BMC Server Automation.

1. Navigate to the following BMC Communities document:
2. Download the most recent Windows Filter Configuration File file to a location available to the BMC Server Automation console.
3. From the console, select Configuration > Patch Global Configuration and click the Windows tab.
4. Next to the Windows Filter Configuration File (the last item in the tab), click the ellipsis button (...) and navigate to the downloaded file.
5. Select the product categories mapping file you downloaded from BMC Communities and click OK.
6. Click the Save icon (the blue diskette) at the top left corner of Patch Global Configuration window.
7. Edit the filter list in the patch catalog and select the correct Product Filter.
8. Run the Patch Catalog Update Job again.

Note

If the No mappings were found for the selected product message persists, the Windows Filter Configuration File needs further update. If you need assistance, contact BMC Customer Support.

For the latest information on this issue, see the BMC Server Automation Knowledge Article ID: KA419955.

For information about creating an ACL template or ACL policy and controlling server access with agent ACLs, see Managing access.

Windows Hotfix Patch is not found in the Catalog

For steps on troubleshooting this issue, see Windows Hotfix Patch not found in the Catalog.

Catalog update job fails because of PD5.cab, or HF7b.cab  configuration files.

If you have upgraded to BMC Server Automation 86 or later, you may still be using the PD5.cab, or HF7b.cab configuration files for Windows patching. However, BMC Server Automation 8.6 and later versions do not support the PD5.cab, or HF7b.cab configuration files, and you must use the PD5.xml, or HF7b.xml files instead. The Windows catalog update job fails if you use the .cab configuration files.

To update the configuration files used for Windows patching see, Global configuration parameters.

Shavlik metadata was not copied to the target

You can refer to the following articles in the BMC knowledge base for steps on troubleshooting this issue:

If your issue is still not resolved, file a ticket with BMC Support and provide the information from the following logs:

• Patch Analysis Job log
• rscd.log from file server and target

Turning on debug tracing for cl5.exe

To validate the payload, the shavlik engine runs the cl5.exe tool on the target. If this tool is not working properly, you can turn on tracing, using the following command:

CL5.exe 2097197 1 <Log Directory>

The customary value for maximum log size is 5000000. This command writes some values to the registry.

To turn off CL5 tracing, use the following command:

CL5.exe 2097197 0

One of the BLPatchCheck2 phases did not complete successfully

If your issue is still not resolved, file a ticket with BMC Support and provide the information from the following logs:

• Patch Analysis Job log
• rscd.log and Trace.txt from target

You can refer to the following articles in the knowledge base for steps on troubleshooting this issue:

If your issue is still not resolved, file a ticket with BMC Support and provide the information from the following logs:

• Patch Analysis Job log
• rscd.log and Trace.txt from target
• appserver.log

Analysis reported unexpected patch as missing

For information about troubleshooting this case, see Windows Patch Analysis Job reports an unexpected patch as missing.

You can also refer to the following articles in the BMC knowledge base:

Analysis did not report expected patch as missing

For information about troubleshooting this case, see Windows Patch Analysis Job does not report an expected missing patch.

You can also refer to the following articles in the BMC knowledge base:

For more information about troubleshooting this case, see Analyzing differences between Windows Patch Analysis Job results and third-party vendors.

You can also refer to the following articles in the BMC knowledge base:

Scenario 1: STDERR: cat: rpm-includes.lst: No such file or directory ERROR::YUM dry run failed. ERROR::cmd:  failed!

Scenario 2: Could not find repodata.tar.gz corresponding to OsArch: 

‘[ARCH]' in the catalog ( at location '//<repo>/catalog_XXX/[ARCH]/repodata.tar.gz').

Scenario 1: Missing Dependency: [rpm1] = [version] is needed by package [rpm2-version.arch] (installed)

For more information, see Yum failed to complete - Scenario 1.

Scenario 2: Missing Dependency: [rpm1] = [version] is needed by package [rpm2-version.arch] (repo)

For more information, see Yum failed to complete - Scenario 2.

Scenario 3: [rpm1] conflicts with [rpm2]

For more information, see Yum failed to complete - Scenario 3.

Scenario 1: [rpm-version.arch]: Caching enabled but no local cache of //<staging>/blrepos/repo/packages/[rpm-version.arch].rpm from repo.

Scenario 2: Transaction Check Error: package [rpm-version] is already installed.

For more information, see Deploy Job failed - Scenario 2.

The error messages suggest that either blyum is not found on the target or it is found but some libraries that are needed to run blyum were missing.

To fix this error or research further, review the analysis_err.log error log file in the staging directory.

This error is usually encountered when the supported agent version or architecture is not installed. So, to troubleshoot this issue, validate that the supported agent version or architecture is installed.

Log files to collect from the application server:

• Patch Analysis Job log

Log files to collect from the target:

• rscd.log
• agentinfo.log
• analysis bundle.log

This error message indicates that the Catalog Job did not succeed at a target where the repodata.tar.gz file was not generated. As a result, the Analysis Job was run against an invalid target whose OS level or architecture did not match. This problem might be caused by an issue on the Repo Server.

To fix this error, perform the following:

1. Run the Catalog Update Job again, and ensure that it ran successfully at all targets.
2. Validate that the target is correct by right-clicking and verifying the OS information.
3. Validate the ACLs on the Repo server.

• Patch Analysis Job log
• Catalog Update Job log

From the target

• rscd.log
• agentinfo.log

From the Repo server

Both rpm1 of the specified version and rpm2 are installed on the target. A newer version of rpm1 is found in the Catalog and set to be updated. The installed version of rpm1 is also a dependency to rpm2. To approve the installation of a newer rpm1, yum also needs to update rpm2. However, yum cannot find a newer version of rpm2 in repodata.tar.gz, so rpm2 is excluded from analysis.

As a new version of rpm2 is not found, the rpm is not offered an update. Because rpm2 is not updated, yum cannot allow the update of rpm1. An error is logged, alerting that rpm1 needs to remain installed to preserve the dependency of rpm2.

To troubleshoot this error, follow these steps:

1. Validate that rpm2 is missing from the catalog.
2. If rpm2 is missing from the Catalog, check if it exists in the RHN Channel.
   • If not in the RHN Channel, then analysis happens as expected.
     1. Acquire rpm2 by other means and update it along with rpm1.
     2. Exclude Rpm1 from the analysis.
        Note that another rpm might require the newer version of the excluded rpm.
     3. Uninstall rpm2.
        When you uninstall rpm2, consult your administrator and use caution.
   • If present in RHN Channel, run the Catalog Update Job again.
     • If rpm2 is still not in the catalog, and this is not an RBAC-related issue, then collect logs and contact BMC Support.
     • If rpm2 is now present in the catalog, run the analysis again.
       If Analysis still shows the same error, continue to the next step .
3. If rpm2 is present in the catalog, check whether rpm2 is present in repodata.tar.gz on the target by running the following command:

   # cd <staging>/LinuxCatalog_XXX_[target]/
# yum -C -c yum.conf search [rpm2]

   Proceed according to the results that you obtain:

   • If rpm2 is not found, then this could be why it is not offered during Analysis.

   • If rpm2 is found, collect logs and contact BMC Support.

Log files to collect from the application server

Log files to collect from the target
analysis bundle.log (including repodata.tar.gz)

Neither rpm1 of the specified version nor rpm2 are installed on the target. Yum sets rpm2 to be updated for one of two reasons:

• Yum detected an older version of rpm2 installed.
• Yum offered rpm2 as a dependency to some other rpm.

Yum checks if rpm2 needs any dependencies of its own, and detects that rpm1 of the specified version is needed as a dependency. Therefore, yum attempts to set rpm1 to be updated as well, but fails. An error is logged, alerting that rpm1 is required for rpm2 to be installed.

To troubleshoot this error, follow these steps:

1. Check if rpm1 of the specified version exists in the Catalog.
2. If rpm1 is not in the Catalog, check if it exists in the RHN Channel.
   • If not in RHN Channel, then analysis happens as expected.
     1. Acquire rpm1 by other means and install it manually.
     2. Exclude rpm2 from the analysis or uninstall rpm2.
        When you uninstall rpm2, consult your administrator and use caution.
        Note that another rpm might require the newer version of the excluded rpm. Furthermore, this step will not work if yum offered rpm2 as a dependency to some other rpm.
   • If present in the RHN Channel, run the Catalog Update Job again.
     1. If rpm1 is still not in catalog, and this is not an RBAC-related issue, then collect logs and contact BMC Support.
     2. If rpm1 is now present in the catalog, run the analysis again.
        If Analysis still shows the same error, continue to the next step.
3. If rpm1 is present in the catalog, check whether rpm1 is present in repodata.tar.gz on the target by running the following command:
# cd <staging>/LinuxCatalog_XXX_[target]/
# yum -C -c yum.conf search [rpm1]Proceed according to the results that you obtain:
   • If rpm1 is not found, then this could be why it is not offered during Analysis.
   • If rpm1 is found, something might be preventing it from being installed.
     In such a case, try performing a dry run installation of rpm1 using the following command:
     # rpm -iUv --test rpm1
     If the dry run fails, then yum predicted this failure and that is why yum rejected rpm1 from being updated, and rpm1 was no longer considered to be updated. An error is logged, indicating rpm1 as a missing dependency. For a possible cause for this dry run failure, see Knowledge Article KA347757.

Log files to collect from the application server

Log files to collect from the target

• rscd.log
• analysis bundle.log (including repodata.tar.gz)
• rpm –qa

[rpm1] conflicts with [rpm2]
 

Rpm2 is installed on the target, while rpm1 is not. Yum attempts to offer rpm1 to be installed or updated, but then yum detects that some of the files to be updated by rpm1 are used by rpm2. Therefore, yum rejects the installation or update of rpm1. An error is logged, alerting that rpm1 cannot be installed. In general, the conflict suggests that rpm1 and rpm2 cannot coexist if rpm versions matter.

1. Check the RHN site for known conflict issues.
   A conflict may arise if either rpm1 or rpm2 are not in the repository.
2. Validate that the rpm is in RHN channel, in the Catalog, and in repodata.tar.gz, as described in scenario 2.
   Note that resolution cannot be guaranteed if the RPMs come from different channels.
3. Proceed according to your needs:
   • If rpm1 must be installed, check whether rpm2 can be uninstalled, downgraded, or upgraded to a non-conflicting version.
   • If rpm2 must be preserved, consider the following actions:
     • Check whether rpm1 can be excluded. Note that another rpm might require the newer version of the excluded rpm.
     • Check whether another non-conflicting version of rpm1 can be installed.
     • Check whether multiple versions of rpm2 are installed, where the conflicting one can be uninstalled.
4. If the previous steps do not resolve the problem, collect logs and contact BMC Support.

Log files to collect from the application server

Log files to collect from the target

• analysis bundle.log (including repodata.tar.gz)
• rpm –qa

Prior to installing the packages, the Deploy Job runs a preliminary yum analysis. During this analysis scan, rpm-version.arch is found as missing. The Deploy Job does not find rpm-version.arch in the list of staged missing patches, and, therefore, the job cannot proceed.

An error is logged, alerting you that the patch payload is not found and cannot be installed. This error indicates that a discrepancy was found between the regular Patch Analysis and the preliminary yum scan during Deploy, or that rpm-version.arch was not copied during staging. The error message is written in either the Deploy Job log or the bldeploy log.

1. Review Patch Analysis results and check whether rpm-version.arch was found as missing.
2. If the file is missing, check whether it was packaged during patch remediation.
   
   • If it was packaged, check whether it points to an existing source. Open the object from inside the Catalog and review its Installable source value.
     
     • The root cause of the problem might be that the installable source points to a missing payload. In this case, rerun the Catalog or download the patch manually.
     • If the source points to an available payload, review the stage phase of the Deploy Job. Check for issues that might have occurred during the copying of the payload to the target.
   • If the object was not packaged, check for possible RBAC issues.

3. If patch was not found missing, compare yum scans of Analysis and Deploy:

   The deploy log exists if the Deploy Job was set to preserve staging area on failure. If not, configure this setting now, then reproduce the error and collect the log file.

4. Compare the lists of set to be updated patches.
   This comparison might reveal that for some RPMs, the analysis finds one arch file missing, while Deploy finds the other one missing. This might be the cause of the original issue. If so, a fix is available in BMC Server Automation 8.1 Service Pack 3.

5. If the preceding steps did not reveal the root cause of the issue, collect several logs and contact BMC Support. From the application server, obtain the Patch Analysis Job log. In addition, obtain the following logs from the target:

   • Analysis bundle log
   • bldeploy log
   • <staging>/XXXXX/blres.yum.depl

package [rpm-version] is already installed

The Deploy Job installs the patches that were found missing during Analysis. In this case, the Analysis Job packaged the rpm-version, even though it is already installed.

1. Review the Analysis logs to troubleshoot the issue. 

   The error message is located in either the Deploy Job log or the bldeploy log.
   You might see the following log records:

   • The analysis_log.log is expected to reveal that the Patch Analysis ran with an include filter. The signs for this are:
     
     • After 'update' you see a list of RPMs.
     • YUM Command: for i in 0 1 2 3 4 5 :; do echo n; done | blyum -c ./yum.conf -C update "rpm-version.arch" "…" ...
   • Command rpm –q [rpm] reveals that multiple versions of the rpm are installed.
2. The expected findings described in the previous step are not the standard configuration. Therefore, perform the following further testing:
   • Run yum update with include list.
     This is expected to produce the following error:
latest rpm as missing [issue reproduced]
     The updated rpm is offered for the older version of RPM that is installed even if the new version is also already installed.
   • Run yum update without include list.
     This should return the following message:
no missing rpm [good result]
     If we do not forcefully analyze for patches (with an include filter), then if the latest version is found, the rpm is not offered.
   • Run yum install with include list.
     This should return the following message:
no missing rpm [good result]
     The Install option checks only for the specific version of the rpm that we are attempting to install. Since our specific version of rpm is already installed, it will not be installed again.
3. If you must preserve multiple versions of such RPMs, create a separate analysis job where you would use Install option and include the RPM.
4. Modify the existing Analysis Job include filter and remove the RPM.
   If the RPM must be included, try excluding the opposite set of RPMs. For example, exclude all where RPM does not equal [rpm-version].
5. Yum does not install multiple versions of such RPMs by default, so if they are not required, then uninstall the old versions.
   The Analysis Job should no longer report the latest RPM as missing, because there will no longer be multiple versions.

Log files to collect from the application server

• Patch Analysis Job log

Log files to collect from the target

• analysis bundle log
• bldeploy log
• <staging>/XXXXX/blres.yum.depl
• rpm –q [rpm]