Transcript
In this screencast, we will walk through the process of setting up the high-availability of OpenAML's front-end, also known as the control plane. Afterwards, we will verify that the HA functionality actually works. The number of nodes must be odd. It is recommended to have all nodes equally sized to avoid surprises after the failover. The bare minimum of nodes equals to 3. When the leader is down, other nodes are voting amongst themselves to elect the new leader. The new leader will take over the responsibilities, while the failed node, once joined back, will be demoted to the follower. Only one node is active or in the read-write state, while all other nodes remain in passive or read-only state. The environment must have a shared storage and ability to use a floating IP. OpenAML nodes are using the RPC port 2633-TCP to communicate with each other. Each of the nodes maintains and updates its own database. The manual database sync is necessary only during the configuration process or when there is a need to add another node. The environment in this demonstration is the following. Three nodes, each with OpenAML front-end software installed. The OpenAML daemon is only running on the 1FE0 node. All nodes have the shared storage mounted. The shared storage exposes the image and system data stores. There is one floating virtual IP to be assigned to the leader node. The acquisition and release of this virtual IP will be configured along with the HA itself. We are starting the whole process on the leader node. The hostname of this node is 1FE0. For now, it is going to be considered as a leader. Remember that the OpenAML front-end high availability is configured at the zone level, meaning that in case of a federated environment, each of the zones can have its own HA setup. Use the one-zone command line to add the 1FE0 to the HA configuration. As seen in the command line parameters, we are using the RPC port. Make sure that this port is available for external connections and not blocked by network security tools. However, it is very important to mention that allowing every IP on the network connect to this port is a very bad practice. Make sure that only the necessary IPs can communicate over the RPC port. Once the command was executed, verify that it was added and note the ID. We will be using this in the future. Now stop the OpenAML daemon. Now we must edit the OpenAML main configuration file etc.1.1d.conf. Let's look for federation configuration set and change the ID from negative one to the ID we've seen in the output of the one-zone show command. We're going to set it to zero. Additionally, we should configure the floating IP. You may want to manage this floating IP in any other way. However, OpenAML has the necessary toolset, so it works out of the box. In the configuration file you can find two actions. One is to acquire the floating IP when this particular node has been promoted to the leader. The second is to remove the floating IP if this node become a follower. Save the configuration file and move to the next one, the monitord.conf. This configuration file defines the monitoring settings. Because monitoring agent must communicate with OpenAML frontend node, we must change the default setting to the floating IP as well. Let's locate the monitor underscore address parameter and change it from auto to the floating IP. Save the configuration file and start the OpenAML daemon. Let's verify that the floating IP was acquired successfully. As seen on the screen, the IP address has been configured without any issues. Meanwhile on the leader node, export the database using the wandb command. We will need to transfer the database export along with all of the contents from walib1.1 directory for a few times, so for convenience purposes let's create the transfer directory under the tmp. Move the database export file and all the files from the .1 directory to the transfer directory. Then create an archive for easier transfer. Copy the archive to one fe1 node and switch to the command line of this node. Decompress an archive and copy all files from the extracted .1 directory to the .1 directory that's located under the walib1 on the follower node. Then execute the wandb command to restore the database. Since the HA implementation may be performed anytime, the size of a database from a mature environment is greater than from a relatively fresh one, that's why the import process may take a while. Now when the database is restored, as the last step, make sure that the owner of the files is set to an admin. We were operating under the root user, so an additional command must be executed to ensure the ownership. Once the node is prepared, switch back to the leader's command line, in this case 1fe0 node. Using the wanzone command, add another server, this time 1fe1. Execute the wanzone show0 command and record the ID of the newly added node. For now the node is in the error state. It's normal, we haven't joined this node to the HA cluster yet and the openable software is not running. Switch to the follower node and edit the etc1 1d.conf file. As with the leader, locate the federation setting and set the server ID parameter to the correct one, in this case the ID equals to 1. As with the leader, you must set up the floating IP commands. Last but not least is the monitoring configuration file. You can either set it manually or use the wanzone server sync command to sync this and other necessary files between the leader and the followers. Start the openable software and switch over to the leader. Execute the wanzone show0 command to print the current state of the frontend's HA cluster. As seen on the screen, the 1fe1 node is added as a follower. Index and commit values may be lagging behind for a while since the nodes just have been but it should be in sync in a while. Now we must perform the identical steps with the 1fe2 node with the only difference being the ID of the node. All three nodes are connected and their index and commit values are in sync, we can consider that the frontend HA is configured successfully. The setup is over, but it's always better to test it. The first test is to access the sunstone. In this environment, sunstone is running on each node separately. Each time the node goes down, the newly elected leader will take over the frontend and the web UI responsibilities. This can be further improved by decoupling the sunstone role from the frontend and moving it to an independent node. The sunstone is reachable via the FQDN and one admin can be logged in without any issues. The second test is to power off the current leader and make sure that the next leader has been elected. Verify that your location is correct and issue the power off command. Now connect to either of the remaining nodes and execute the 1zone show0 command. As seen on the screen, the 1fe0 is in the error state since we've just powered it down. 1fe2 was elected as the new leader and 1fe1 remained as a follower. We're going to power on the 1fe0 node and rerun the 1zone show0 command once again. This time 1fe0 is online and was demoted to a follower. Switching to the 1fe2 we can confirm that the floating IP was moved to the current leader as well. Third and the last test is to put the FQDN on ping and power down the leader. Switch to 1fe2 and execute the 1zone show0 command to verify that we are on the right host and execute the power off command. Let's switch over to the Jamf host and observe the pings. The FQDN responds without any interruptions. However by executing the 1zone show0 command one more time we can confirm that the 1fe2 has been down for a while and that the new leader was elected. While you can configure the frontend high availability manually, it is recommended and much more convenient to deploy the HA using the 1deploy tool. It covers automated deployment of OpenEBOLA in HA mode as well as many other use cases. Please visit the official GitHub page of 1deploy or official OpenEBOLA documentation to learn more about the tool. And this concludes the screencast where we have shown how to set up the OpenEBOLA frontend in high availability mode and performed three different tests to confirm that the setup is ready to be used. Thank you for watching and see you in the next screencast.
