A less known jewel of Solaris 11 is RAD (Remote Administration Daemon). Since, as I just found out, I don’t have any RAD posts yet, let’s talk about what it is and offers before we go on.
what RAD does is it provides programmatic interfaces to manage Solaris. Users, zones, zfs and smf are just a few examples. RAD offers APIs for C, Java, Python and REST. It can be used locally as well as remotely. It can be used to read data but also to write data. As an example you can get zfs informations as well as create new datasets or change current settings. There are a couple of great examples and posts out there from e.g. Glynn Foster and Robert Milkowski
Why am I telling you this? Because you can PROGRAMMATICALLY manage your enterprise operating system now.
Imagine an environment of SPARC T4, T5, T7 or S7 servers running quiet few non-global zones whether in LDOMs or not.
Over the last weeks I tested kernel zones pretty heavily. The chance of getting rid of LDOMs is just too good to not go for it. Don’t understand me wrong, LDOMs work fine and are a key part of our current DR concept. But there are also things I really don’t like at all. Guess this will make a good post in the nearer future. ;)
As I said, I was using kernel zones but for DR purposes (let’s say one data center dies) I need to be able to boot the kzones from the other data center. In order to do so the zone configuration has to be available. Well, shared storage and so on too but for the purpose of this post let’s say that’s all taking care of automatically (it really is ;) ).
At the moment Zone configurations are saved via a SMF service on a NFS server. But I don’t want to create zones first while the angry mob can’t work out there and tries to figure out where I am sitting.
When I used kernel zone live migration I started thinking about how I want to solve this issue. For those who haven’t used kzone live migration yet, it creates a zone configuration on the target side and leaves the old one in configured state. Which means once you have live migrated a zone the problem seems to be solved. But what if something changes? What if it runs on a different server (hw) by the time of a disaster.
These two facts, programmatic interfaces and LDOM replacement, lead me to the idea of actually having a SMF scheduled service that takes care of zone configurations. For that I use RAD’s zonemanager and Python.
Besides the RAD IPS packages (rad and rad-zonemgr) being installed you will need a user with sufficient privileges for rad and zones administration/management.
The script zones-sync is part of a larger RAD script that makes all sort of stuff.
Simply said it checks which zone’s configuration is missing on a target server and then imports/creates it. Quiet trivial.
Let’s start wit the imports.
import rad.connect as radc import rad.bindings.com.oracle.solaris.rad.zonemgr_1 as zonemgr import string import socket import sys
Line 36 let’s you connect to RAD as the name already says while the import in line 37 adds the ability to us zone management. Quiet self-explaining I would say.
In order to know which global zones are suppose to be synced I started with getting the source and target hostname straight, depending on the used arguments. So, if only one hostname is given the other one will be considered to be localhost. For remote purposes two hostnames need to be provided. For the purpose of automation I added [-service|-svc] which in this case maps certain pattern of hostnames. The name pattern is used to find the corresponding global zone.
In the end anything that helps to automate getting the hostnames should be put here.
def getHostnames(): global source_hostname global target_hostname if sys.argv == "-service" or sys.argv == "-svc": if source_hostname == 'A': target_hostname = source_hostname[:3]+'S'+source_hostname[4:] elif source_hostname == 'a': target_hostname = source_hostname[:3]+'s'+source_hostname[4:] elif source_hostname == 'S': target_hostname = source_hostname[:3]+'A'+source_hostname[4:] elif source_hostname == 's': target_hostname = source_hostname[:3]+'a'+source_hostname[4:] elif len(sys.argv) == 3: source_hostname = sys.argv target_hostname = sys.argv elif len(sys.argv) == 2: target_hostname = sys.argv return (source_hostname,target_hostname)
Now that it is clarified which systems will be involved the next step is to connect to RAD. As you can see in lines 99 and 104 I am using ssh to connect to remote systems and Unix socket for local connections as shown in line 102.
Again, the user that executes this script must have sufficient privileges on the involved systems. In addition to that the rad:remote and/or rad:local service has to be enabled and online.
def connectRAD(): global source_rc global target_rc if len(sys.argv) == 3: source_uri = radc.RadURI("ssh://"+source_hostname) source_rc = source_uri.connect() else: source_uri = radc.RadURI("unix:///") source_rc = source_uri.connect() target_uri = radc.RadURI("ssh://"+target_hostname) target_rc = target_uri.connect() return (source_rc,target_rc)
The next step is to get a list of zones of each global zone. What it actually is is a list of the objects of each zone.
def getZoneLists(): global zones_s global zones_t zones_s = source_rc.list_objects(zonemgr.Zone()) zones_t = target_rc.list_objects(zonemgr.Zone()) return (source_zones,target_zones)
Each object includes the values of a zone. For example name, state, brand, etc..
The previous is done to get each ng/kzone’s state and therefore to decide whether it is synced or not. Incomplete zones for example are not worth synchronizing. A configured zone’s config will be replaced by the one of a running zone in order to have the most current version configured.
def getSourceZones(): printHeader(source_hostname) for name_s in zones_s: zone_s = source_rc.get_object(name_s) print "\t%-16s %-11s %-6s" % (zone_s.name, zone_s.state,zone_s.brand) if zone_s.state != 'incomplete': source_zones.append(zone_s.name) if zone_s.state == 'configured': source_conf_zones.append(zone_s.name) return (source_zones,source_conf_zones) def getInstalledTargetZones(): printHeader(target_hostname) for name_t in zones_t: zone_t = target_rc.get_object(name_t) print "\t%-16s %-11s %-6s" % (zone_t.name, zone_t.state,zone_t.brand) if zone_t.state != 'configured' and zone_t.state != 'incomplete': target_zones.append(zone_t.name) if zone_t.state == 'configured': target_conf_zones.append(zone_t.name) return (target_zones,target_conf_zones)
After comparing the states, the script deletes existing configurations that are about to be replaced.
In line 152 you can see the preparation of connecting to the target machine’s RAD zonemgr (rad.bindings.com.oracle.solaris.rad.zonemgr_1). The class that is used here is ZoneManager(rad.client.RADInterface)
Quote from the python help for rad.bindings.com.oracle.solaris.rad.zonemgr_1.ZoneManager:
| Create and delete zones. Changes in the state of zones can be
| monitored through the StateChange event.
def deleteExistingConfiguredTargetZone(): delete_zone = target_rc.get_object(zonemgr.ZoneManager()) global zones_t for name_d in import_zones: for name_t in zones_t: zone_t = target_rc.get_object(name_t) if name_d.name == zone_t.name: delete_zone.delete(name_d.name) print "DELETED: %s" % name_d.name zones_t.remove(name_t) break
As you can see above in line 159 the method used is called delete.
When this is done it is time to export and import configurations.
To export a zone configuration via RAD (line 168) the proper class is called Zone with its method exportConfig(*args, **kwargs). Imports (line 170) are done by using the class ZoneManager and (importConfig(*args, **kwargs) as the method.
zones-sync.py [-h|help] [-service|-svc] [target hostname/ip]
def expImpConfig(): mgr = target_rc.get_object(zonemgr.ZoneManager()) for name_i in import_zones: zone_i = source_rc.get_object(name_i) z_config = zone_i.exportConfig() split_config = z_config.splitlines(True) mgr.importConfig(False,zone_i.name,split_config) print "IMPORTED: %s" % zone_i.name
Well, all that is left to do is to close the connections.
def closeRc(): source_rc.close() target_rc.close()
And for you to have an idea what this may look like and what it does, let’s check out the following outputs.
In the following the script was used with -service. This is the way I use for scheduled/periotic services. The hostname’s pattern is used to define the source and target hostnames.
u2034611@AC6A000:~$ /net/ap6shr1/data/shares/soladm/intern/tm/zones-sync.py -service AC6A000: NAME STATUS BRAND kzone1 configured solaris-kz zone2 configured solaris zone3 configured solaris fuu1 configured solaris fu1 configured solaris oo1 configured solaris kzone configured solaris zone1 configured solaris AC6S000: NAME STATUS BRAND kzone1 running solaris-kz zone2 installed solaris zone3 configured solaris fuu1 configured solaris fu1 configured solaris oo1 configured solaris kzone configured solaris zone1 configured solaris
As you can see nothing was deleted or imported. This is what it looks like when everything is in sync.
Let’s tell the script which server is suppose the target in order to sync it with the localhost.
u2034611@AC6A000:~$ /net/ap6shr1/data/shares/soladm/intern/tm/zones-sync.py AC4S000 AC6A000: NAME STATUS BRAND kzone1 configured solaris-kz zone2 configured solaris zone3 configured solaris fuu1 configured solaris fu1 configured solaris oo1 configured solaris kzone configured solaris zone1 configured solaris AC4S000: NAME STATUS BRAND IMPORTED: kzone1 IMPORTED: zone2 IMPORTED: zone3 IMPORTED: fuu1 IMPORTED: fu1 IMPORTED: oo1 IMPORTED: kzone IMPORTED: zone1
Above you can see that on the one server (AC6A000, in this case the local machine) a bunch of different zones are configured and none exist on the target side. Therefore all of the zone configurations are imported.
Let’s say you have a central server or your local workstation from which you want to sync two global zones. In the following example two hostnames are passed on.
G u2034611@r0065262 % /var/tmp/zones-sync.py ac6s000 10.1.30.107 ac6s000: NAME STATUS BRAND kzone1 running solaris-kz zone2 installed solaris zone3 configured solaris fuu1 configured solaris fu1 configured solaris oo1 configured solaris kzone configured solaris zone1 configured solaris 10.1.30.107: NAME STATUS BRAND kzone1 configured solaris-kz zone2 configured solaris zone3 configured solaris fuu1 configured solaris fu1 configured solaris oo1 configured solaris kzone configured solaris zone1 configured solaris DELETED: kzone1 DELETED: zone2 IMPORTED: kzone1 IMPORTED: zone2
This time both global zones have several zones in the configured state and on one host one zone is running and another one is in the installed state. Which means that two zone configurations (configured state). The more current ones, which in this case means rather “runnable” or running, are synced and the “only” configured zones’ configuration was deleted.
There are many more things to explore about RAD, so have fun and stay calm ;) !