TPE installation and upgrade issues

Cockpit is unreachable

SYMPTOM: After customizing the Cockpit certificate during the first installation or by editing the infrastructure configuration, Cockpit is unreachable ("CONNECTION_REFUSED").

SOLUTION:

1. Check the status of the Cockpit systemd service.

   systemctl status cockpit

2. If you see the error The certificate and the given key do not match, run the following commands to clean up the Cockpit certificate then redeploy the cluster using TPE Service Cockpit module with correct certificate and private key.

   sudo rm /etc/cockpit/ws-certs.d/cockpit.cert
   sudo rm /etc/cockpit/ws-certs.d/cockpit.key
   systemctl restart cockpit

GUI is unreachable

SYMPTOM: After a first installation, browser is not able to load the page ("Loading... please wait") or displays a certificate error.

SOLUTION:

1. Check the system monitoring (For more information, see Listing Containers and Health Check).
2. If errors are raised when you try to access the TPE Portal GUI after TPE installation, try to reconfigure the hostname of the TPE system to "actility.local". Save & Apply the change and try to access the TPE portal GUI using https://enterprise.actility.local/tpe. If it is working, your certificate is incorrect.
3. If you stay on page "Loading... please wait" when you try to access the TPE Portal GUI, the TLS certificate for HTTP traffic is maybe expired. If the TLS certificate has expired, on TPE cockpit, an error message is displayed at the top of the "TPE Configuration page"

In this case, you must generate a new certificate and upload it.

GUI displays "Unexpected error" messages

SYMPTOM: After an HA upgrade, the GUI displays connection errors (popup message like "Unexpected error occurred", "Unknown error").

SOLUTION:

1. Disconnect and reconnect to the GUI (may require up to 10 disconnect/reconnect iterations to flush the SQL connection pool),
2. If the issue persists, restart both SQL proxy containers and retry to connect to the TPE GUI.

YUM update fails

SYMPTOM: The TPE update procedure fails with the following error:

There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help).

SOLUTION:

1. Run the following command through ssh:

   sudo yum-complete-transaction

2. Then retry TPE update procedure.

Update procedure fails with "ansible_memtotal_mb is undefined" message

SYMPTOM: The TPE update procedure fails with the following error:

tpe_node1 failed | msg: The conditional check 'ansible_memtotal_mb < max_xs_host_memory_size_mb' failed. The error was: error while evaluating conditional (ansible_memtotal_mb < max_xs_host_memory_size_mb): 'ansible_memtotal_mb' is undefined^M

SOLUTION:

Retry TPE update procedure.

Update procedure fails with "Not enough disk space to perfom the upgrade" message

SYMPTOM: The TPE Update procedure fails with the following error:

NOTE: There is not enough disk space to perform the upgrade (at least 10GB required).

SOLUTION:

• Clean up unnecessary container images, here is the procedure
• Free up disk space

Service actility_post-upgrade remains in failed state after reboot

SYMPTOM: After TPE instance reboot, the service actility_post-upgrade remains in failed state in Cockpit. This is because post-upgrade service has not been restarted automatically after the reboot.

SOLUTION:

The solution consists of redeploying the actility_post-upgrade service manually. For that:

1. Connect to Cockpit
2. Go to the TPE Services module
3. Under "others" directory, for service "actility_post-upgrade", click on "Actions" button then on "Redeploy"

Container images provisioning failed

SYMPTOM: During an install, an upgrade or a cluster redeployment, the container images provisioning might fail and the procedure (install, upgrade or redeploy) stops.

tpe_node1 -> localhost failed | item: {u'key': u'mongo', u'value': {u'image': u'tpe/mongo', u'version': u'5.2.0-4'}} | msg: Error pushing image registry1.actility.local:5000/tpe/mongo: dial tcp X.X.X.X:5000: connect: no route to host

tpe_node2 -> localhost failed | item: {u'key': u'twa_dev', u'value': {u'image': u'tpe/twa-dev', u'version': u'7.12.6-1'}} | msg: Error pushing image registry2.actility.local:5000/tpe/twa-dev: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

Several reasons can explain this issue:

• network connection issue to the TPE repository
• local container images registry instability

SOLUTION:

The solution is to restart the procedure (install, upgrade, redeploy).

Container images cleanup failed

SYMPTOM: During an upgrade, the container images cleanup might fail with the error:

An error occurs during container images cleaning.

Since this is not blocking, the upgrade will continue to process but uncleaned container images will take up disk space.

SOLUTION:

After the upgrade is complete, run the following script to clean up container images:

/usr/bin/tpe-cleanup-container-images

RF Region configuration issue for Tektelic Kona Mega gateways

SYMPTOM: After a self-hosted upgrade in 7.3.4, Tektelic Kona Mega gateways fail to retrieve their RF Region tarball for the following radio configurations:

• Gateways configured with 2 antenna with more than 8 channels
• Gateways configured with 1 antenna with more than 16 channels

SOLUTION: Execute the following procedure on node1, and node2 for a TPE OCP HA.

1. Connect the node with the SUPPORT account, for instance using the Terminal accessible from Cockpit.

2. Start a bash on the lrc container:

   docker exec -it $(docker ps -q -f "name=actility_lrc1") bash

   NOTE: Use actility_lrc1 for node1 and actility_lrc2 for node2

3. Regenere RF_Region tarball for all RF Regions (takes 1 or 2 minutes):

    for file in $(find /home/actility/FDB_lora/RFRegion/ ! -name '*DEFAULT-RFREGION-TEMPLATE' -type f |  xargs -n 1 basename);do /home/actility/rfregtool/shells/dnrfregionid.sh -d /var/ftp/ -g $file; done

Then, execute the following verification procedure:

1. Got the the FTP folder of the LRC container:

   cd /var/ftp/

2. Check that b1 TARBAL have been generated for US 72 channels:

    ls *US*72*b1_v2.tar.gz

   The command must return two TARBALL files.