Transcript
In this video we will show how to set up Virtual Machine High Availability and verify that a virtual machine will be migrated to another host when the initial host is powered down. Virtual Machine High Availability is a powerful feature with a lot of fine-tuning. However, it requires shared storage to be in place. Without shared storage, virtual machine can be respawned on a different host, but will lose all the data. This may be suitable for stateless machines where the data is not a concern. However, true Virtual Machine High Availability is only available in an environment with shared storage. This functionality relies on the Hooks subsystem. The Hooks subsystem is powerful automation and integration functionality available in OpenEBOLA. This functionality allows the execution of any script based on a virtual machine, host, or image state change, or an API call. To learn more about the Hooks system, please visit the official OpenEBOLA documentation. Fencing is important to prevent split-brain issues. It is crucial to ensure that the host is fenced. You must edit the fence-host.shell script and supply it with the correct fencing command. Virtual Machine High Availability also relies on the built-in monitoring of hosts and virtual machines. You may need to adjust the monitoring settings as well to achieve faster failover. The environment for this demonstration consists of one host with OpenEBOLA frontend software, two KVM hosts managed by OpenEBOLA, each host has a virtual machine running. One virtual machine is a Flask-based web application, and the second one is the database that stores data for this app. Shared storage that exposes system and image data stores and is mounted to each node, including the frontend, using the NFS protocol. To enable Virtual Machine High Availability, we must configure the hook first. You can find the hook template in the official OpenEBOLA documentation. There are additional flags that can be passed to the script, each of them is described in the documentation as well. To create a hook, we're going to create a file and insert the hook template contents. The default hook settings are very cautious, which may lead to longer virtual machine migration times. In the scope of this demonstration, we're going to make them much more aggressive. Additionally, there is no need to fence the host because we're going to power it down ourselves, hence the no-fencing flag has been added. Please note that the no-fencing flag works in a demo environment but should never be used in the production environment. The hook will execute the host error script if any host changes its state to error. Save the file and then use the one-hook create command line tool and point to the file template. Now the hook has been created successfully. Using the one-hook list command, we can find all hooks configured in this environment. Using the one-hook show command, we print the details of a specific hook and its execution log. In your production environment, please don't forget to configure the fencing mechanism by editing the following file. Once again, you shouldn't run virtual machine high availability without fencing in a production environment. Now it's time to test and verify whether it works as expected. We have two virtual machines running on two different hosts. These two represent a typical application with a database, a flask-based web application is serving the data while the database stores it. In this test scenario, the web application will remain running while the host with the database virtual machine is brought down. Let's navigate to the web page and confirm that the database isn't empty and contains the data. The data is there. In our test scenario, we are supposed to lose connectivity with the database for a brief period, but once it's back online, the data must not be lost. Now let's open a VNC connection to the application virtual machine and log in using the configured credentials. Once logged in, let's start pinging the second virtual machine. We are going to use the ping command to capture the moment when the virtual machine goes down along with the host and when it successfully started on another host. Switch back to the command line interface, this time to the KVM node that will be powered down. Now let's execute the shutdown command. Now let's observe the state change of a host using the OpenAbility built-in command line tool one-host-top. Let's switch to the VNC session and see whether the virtual machine is still responding to sent ICMP packets. As we can see, the virtual machine stopped responding. Back to the command line, we can confirm that the host state switched to error. This will trigger the execution of the high availability hook. Now let's look back at the ICMP packets. We can see that the virtual machine starts to respond to ICMP requests once again. The virtual machine went down after the 44th request and started responding on the 66th, meaning that only 21 ICMP packets went unresponded. Switching to the application, we can see that the database is not yet running as the operating system and database software are in the process of starting up. By trying one more time, we can confirm that the database software is back online and the data has been preserved. By switching to the SunStone web UI, we can see that both virtual machines are now running on the same host. This is the result of the virtual machine high availability hook execution. And this concludes our screencast. We have shown how to configure the virtual machine HA capability of OpenNebula and verify that the functionality works as expected. Thank you for watching and we will see you in the next screencast.
