Upgrading an external ZooKeeper server

Use this procedure to upgrade the external ZooKeeper servers that are used by Teracloud® Streams enterprise domains.

About this task

The upgrade procedure that you use depends on how your ZooKeeper ensemble is configured.

  • Rolling upgrade: If each ZooKeeper server is running on a dedicated machine and the hostname or zkconnect string is not changing, you can upgrade the server while the Teracloud® Streams domains are running.

  • Standard upgrade: If the ZooKeeper servers share a machine or there are changes to the hostname or zkconnect string, you must perform a standard upgrade on each server. Teracloud® Streams domains must be stopped before performing the upgrade.

Procedure

  • For a rolling upgrade, complete the following steps:
    1. Stop any ZooKeeper watchdog or monitor that restarts the ZooKeeper server.

    2. Verify that the ZooKeeper ensemble is up and running by using the ZooKeeper srvr or stat command. For a srvr command example, see the ZooKeeper setup procedure.

    3. Stop one of the ZooKeeper servers that is defined in the zoo.cfg file and verify that a quorum still exists.

      In the following example, the ZooKeeper ensemble includes zkserver1, zkserver2 and zkserver3. The client port number is 2181.

      1. To stop zkserver1, enter the following command on zkserver1:
        ZooKeeper-installation-directory/bin/zkServer.sh stop
      2. To verify that a quorum still exists, use the srvr command and specify another server in the ZooKeeper ensemble on the command, for example:
        echo srvr | nc zkserver2 2181
        If a quorum exists, the srvr command output includes information about the ZooKeeper version, latency, and other information as shown in the following example:
        Zookeeper version: 3.8.4-9316c2a7a97e1666d8f4593f34dd6fc36ecc436c, built on 2024-02-12 22:16 UTC
Latency min/avg/max: 0/4/2834
Received: 14529333776
Sent: 14758752055
Connections: 1616
Outstanding: 4
Zxid: 0x952925303
Mode: follower
Node count: 237674

    4. Back up the following directories on the server:
      • dataDir
      • dataLogDir, if specified in the zoo.cfg file

    5. Perform the upgrade on the server.

    6. Copy the content in the following directories to the upgraded server:
      • dataDir
      • dataLogDir, if specified in the zoo.cfg file

    7. If applicable, transfer the following files to the upgraded server:
      • zoo.cfg
      • java.env
      • log4j.properties

    8. Verify that the zoo.cfg file references the correct file path for the dataDir and dataLogDir directories, and that the myid files are configured correctly.

    9. Restart the upgraded ZooKeeper server, for example: 
      ZooKeeper-installation-directory/bin/zkServer.sh start

    10. Repeat Steps 2-9 for the other ZooKeeper servers in the ensemble.

    11. After all servers in the ensemble are upgraded, restart the ZooKeeper watchdog or monitor, if applicable.

  • For a standard upgrade, complete the following steps:
    1. Stop the Teracloud® Streams domains that are using the ZooKeeper ensemble. To stop a domain, enter the following command: 
      streamtool stopdomain -d domain-id

    2. Log in as user root on each Teracloud® Streams resource that is running the domain controller service as a Linux™ system service, and enter the following command: 
      streamtool stopdomainhost -d domain-id

    3. Stop any ZooKeeper watchdog or monitor that restarts the ZooKeeper server.

    4. Stop the ZooKeeper servers. To stop a ZooKeeper server, enter the following command on the server:
      ZooKeeper-installation-directory/bin/zkServer.sh stop

    5. Back up the following directories on each server:
      • dataDir
      • dataLogDir, if specified in the zoo.cfg file

    6. Perform the upgrade on each server.

    7. If applicable, transfer the following files to each server:
      • zoo.cfg
      • java.env
      • log4j.properties

    8. Verify that the zoo.cfg file references the correct file path for the dataDir and dataLogDir directories, and that the myid files are configured correctly for each server.

    9. Copy the content in the following directories to each server:
      • dataDir
      • dataLogDir, if specified in the zoo.cfg file

    10. Restart each of the upgraded ZooKeeper servers. To start a ZooKeeper server, enter the following command on the server:
      ZooKeeper-installation-directory/bin/zkServer.sh start
      When a quorum is formed, the data are automatically replicated to the servers in the quorum.

    11. Verify that the ZooKeeper ensemble is up and running by using the ZooKeeper srvr or stat command. For a srvr command example, see the ZooKeeper setup procedure.

    12. After all servers in the ensemble are upgraded, restart the ZooKeeper watchdog or monitor, if applicable.

    13. Log in as user root on each Teracloud® Streams resource that is running the domain controller service as a Linux™ system service, and complete the following steps for each domain:
      1. Update the zkconnect property in the following file to the proper value:
        /etc/teracloud/streams/domain-id/controller.properties

      2. Start the domain controller service on the resource by entering the following command:
        streamtool startdomainhost -d domain-id

    14. Start the Teracloud® Streams domains. To start a domain, enter the following command:
      streamtool startdomain -d domain-id