VCF
VCF SDDC Manager Bundle Checksum Issue
LcmException: bundle checksum does NOT match the manifest value
The Issue? SDDC Manager Bundle Downloads Fail with the above issue
When attempting to upgrade from VCF 4.2.1 to VCF 4.3, all the bundles required for the download were set to download. However, when checking later, 2 were successful and the rest had failed. When looking in Bundle Management in SDDC Manager, error message in the image below presented itself.
When this occurs, the bundle cannot be retried and all options are greyed out. This may not occur all too often, but without knowing what to do, administrators are generally stuck.
Troubleshooting The Issue
The easiest options to troubleshoot this issue are.
- SSH onto SDDC Manager, su to root and restart the lcm service systemctl restart lcm. The service can take up to 15 minutes to come back and initialize the depot. This did not resolve the issue for me.
- Reboot SDDC Manager. This also did not resolve the problem for me.
- After looking in /nfs/vmware/vcf/nfs-mount/bundle, I noticed the failed bundles had folders created with some content. At this point, it would seem the bundles need to be cleaned up and re-downloaded. There is a utility on SDDC Manager that can assist administrators in achieving this.
The Fix
To properly clean up the SDDC Manager database and repo, the bundles and their associations must be removed completely.
- SSH onto SDDC Manager and su to root.
- Navigate to the directory with the tool cd /opt/vmware/vcf/lcm/lcm-app/bin
- Identify bundle ID’s for the failed bundles in SDDC Manager, this can be done by navigating to Bundle Management and clicking more details on one of the failed bundles.
- For each failed bundle, issue the following command followed by the bundle ID, ./bundle_cleanup.py <bundleID>. The command is shown in the image below, and the output from the command is also displayed.
LOG FILE : /var/log/vmware/vcf/lcm/bundle_cleanup.log
-----------------------------------------------------
2021-08-27 04:13:48,922 [INFO] root: Performing cleanup for bundle with IDs : ['ed86b356-e99c-45c7-add8-92791cb28757']
2021-08-27 04:13:48,922 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from upgrade where upgrade_status in ('INPROGRESS','CANCELLING');"
2021-08-27 04:13:48,949 [INFO] root: b' 0\n'
2021-08-27 04:13:48,950 [INFO] root: b'\n'
2021-08-27 04:13:48,950 [INFO] root: RC: 0
2021-08-27 04:13:48,950 [INFO] root: Out: 0
2021-08-27 04:13:48,950 [INFO] root: Stopping LCM service.
2021-08-27 04:13:48,951 [INFO] root: Execute cmd: systemctl stop lcm
2021-08-27 04:13:49,087 [INFO] root: RC: 0
2021-08-27 04:13:49,087 [INFO] root: Out:
2021-08-27 04:13:49,088 [INFO] root: Removing LCM NFS mount.
2021-08-27 04:13:49,088 [INFO] root: Execute cmd: rm -rf /nfs/vmware/vcf/nfs-mount/bundle/ed86b356-e99c-45c7-add8-92791cb28757
2021-08-27 04:13:49,127 [INFO] root: RC: 0
2021-08-27 04:13:49,128 [INFO] root: Out:
2021-08-27 04:13:49,128 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select upload_id from bundle_upload where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,145 [INFO] root: b'\n'
2021-08-27 04:13:49,146 [INFO] root: RC: 0
2021-08-27 04:13:49,146 [INFO] root: Out:
2021-08-27 04:13:49,146 [INFO] root: Bundle with ID : ed86b356-e99c-45c7-add8-92791cb28757 not found in bundle upload table
2021-08-27 04:13:49,146 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select download_id from bundledownload_by_id where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,163 [INFO] root: b' fb61d3c8-c791-4ace-9f09-18d57e3ba48c\n'
2021-08-27 04:13:49,164 [INFO] root: b'\n'
2021-08-27 04:13:49,164 [INFO] root: RC: 0
2021-08-27 04:13:49,164 [INFO] root: Out: fb61d3c8-c791-4ace-9f09-18d57e3ba48c
2021-08-27 04:13:49,164 [INFO] root: Execute cmd: curl -s -X DELETE localhost/tasks/registrations/fb61d3c8-c791-4ace-9f09-18d57e3ba48c
2021-08-27 04:13:49,211 [INFO] root: RC: 0
2021-08-27 04:13:49,211 [INFO] root: Out:
2021-08-27 04:13:49,211 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select upgrade_id from upgrade where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,230 [INFO] root: b'\n'
2021-08-27 04:13:49,230 [INFO] root: RC: 0
2021-08-27 04:13:49,230 [INFO] root: Out:
2021-08-27 04:13:49,231 [INFO] root: Bundle with ID : ed86b356-e99c-45c7-add8-92791cb28757 not found in upgrade table
2021-08-27 04:13:49,231 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select upgrade_id from upgrade where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,249 [INFO] root: b'\n'
2021-08-27 04:13:49,249 [INFO] root: RC: 0
2021-08-27 04:13:49,249 [INFO] root: Out:
2021-08-27 04:13:49,249 [INFO] root: Bundle with ID : ed86b356-e99c-45c7-add8-92791cb28757 not found in upgrade table
2021-08-27 04:13:49,250 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from bundle where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,267 [INFO] root: b' 1\n'
2021-08-27 04:13:49,268 [INFO] root: b'\n'
2021-08-27 04:13:49,268 [INFO] root: RC: 0
2021-08-27 04:13:49,268 [INFO] root: Out: 1
2021-08-27 04:13:49,268 [INFO] root: Deleting bundle & upgrade info for bundle ID : ed86b356-e99c-45c7-add8-92791cb28757
2021-08-27 04:13:49,268 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -c "delete from bundle where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,290 [INFO] root: b'DELETE 1\n'
2021-08-27 04:13:49,290 [INFO] root: RC: 0
2021-08-27 04:13:49,290 [INFO] root: Out: DELETE 1
2021-08-27 04:13:49,290 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from image where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,311 [INFO] root: b' 1\n'
2021-08-27 04:13:49,311 [INFO] root: b'\n'
2021-08-27 04:13:49,311 [INFO] root: RC: 0
2021-08-27 04:13:49,311 [INFO] root: Out: 1
2021-08-27 04:13:49,311 [INFO] root: Deleting bundle ed86b356-e99c-45c7-add8-92791cb28757 in image table
2021-08-27 04:13:49,311 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -c "delete from image where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,330 [INFO] root: b'DELETE 1\n'
2021-08-27 04:13:49,331 [INFO] root: RC: 0
2021-08-27 04:13:49,331 [INFO] root: Out: DELETE 1
2021-08-27 04:13:49,331 [INFO] root: Execute cmd: psql --host=localhost -U postgres -d lcm -tc "select count(*) from partner_bundle_metadata where bundle_id = 'ed86b356-e99c-45c7-add8-92791cb28757';"
2021-08-27 04:13:49,350 [INFO] root: b' 0\n'
2021-08-27 04:13:49,350 [INFO] root: b'\n'
2021-08-27 04:13:49,350 [INFO] root: RC: 0
2021-08-27 04:13:49,350 [INFO] root: Out: 0
2021-08-27 04:13:49,351 [INFO] root: Bundle with ID : ed86b356-e99c-45c7-add8-92791cb28757 not found in partner_bundle_metadata table
2021-08-27 04:13:49,351 [INFO] root: Starting LCM service.
2021-08-27 04:13:49,351 [INFO] root: Execute cmd: systemctl start lcm
2021-08-27 04:13:49,466 [INFO] root: RC: 0
2021-08-27 04:13:49,466 [INFO] root: Out:
5. Once the execution completes, the bundle should be removed. You will also notice some of the commands not finding the bundle and where it is found, it is being deleted. Once the process is complete, the LCM service is restarted. It may take up to 15 minutes for all services to be initialized again, and you should now be able to download the bundles again.
Conclusion
Whilst the underlying issue is not known, the resolution is straightforward and quick. If you run into this issue, and this does not resolve it, leave a comment and we’ll see what we can do!
For more VCF Troubleshooting tips check out my Unofficial VCF Troubleshooting Guide
All my other VCF related articles can be found here
Nice find Shank
Thanks Michael!