RAD – syncing Solaris zone configs

RAD

A less known jewel of Solaris 11 is RAD (Remote Administration Daemon). Since, as I just found out, I don’t have any RAD posts yet, let’s talk about what it is and offers before we go on.
what RAD does is it provides programmatic interfaces to manage Solaris. Users, zones, zfs and smf are just a few examples. RAD offers APIs for C, Java, Python and REST. It can be used locally as well as remotely. It can be used to read data but also to write data. As an example you can get zfs informations as well as create new datasets or change current settings. There are a couple of great examples and posts out there from e.g. Glynn Foster and Robert Milkowski

Why am I telling you this? Because you can PROGRAMMATICALLY manage your enterprise operating system now.

Use case

Imagine an environment of SPARC T4, T5, T7 or S7 servers running quiet few non-global zones whether in LDOMs or not.
Over the last weeks I tested kernel zones pretty heavily. The chance of getting rid of LDOMs is just too good to not go for it. Don’t understand me wrong, LDOMs work fine and are a key part of our current DR concept. But there are also things I really don’t like at all. Guess this will make a good post in the nearer future. ;)
As I said, I was using kernel zones but for DR purposes (let’s say one data center dies) I need to be able to boot the kzones from the other data center. In order to do so the zone configuration has to be available. Well, shared storage and so on too but for the purpose of this post let’s say that’s all taking care of automatically (it really is ;) ).
At the moment Zone configurations are saved via a SMF service on a NFS server. But I don’t want to create zones first while the angry mob can’t work out there and tries to figure out where I am sitting.
When I used kernel zone live migration I started thinking about how I want to solve this issue. For those who haven’t used kzone live migration yet, it creates a zone configuration on the target side and leaves the old one in configured state. Which means once you have live migrated a zone the problem seems to be solved. But what if something changes? What if it runs on a different server (hw) by the time of a disaster.
These two facts, programmatic interfaces and LDOM replacement, lead me to the idea of actually having a SMF scheduled service that takes care of zone configurations. For that I use RAD’s zonemanager and Python.
Besides the RAD IPS packages (rad and rad-zonemgr) being installed you will need a user with sufficient privileges for rad and zones administration/management.

RAD Python

The script zones-sync is part of a larger RAD script that makes all sort of stuff.
Simply said it checks which zone’s configuration is missing on a target server and then imports/creates it. Quiet trivial.

Let’s start wit the imports.

import rad.connect as radc
import rad.bindings.com.oracle.solaris.rad.zonemgr_1 as zonemgr
import string
import socket
import sys

Line 36 let’s you connect to RAD as the name already says while the import in line 37 adds the ability to us zone management. Quiet self-explaining I would say.

In order to know which global zones are suppose to be synced I started with getting the source and target hostname straight, depending on the used arguments. So, if only one hostname is given the other one will be considered to be localhost. For remote purposes two hostnames need to be provided. For the purpose of automation I added [-service|-svc] which in this case maps certain pattern of hostnames. The name pattern is used to find the corresponding global zone.
In the end anything that helps to automate getting the hostnames should be put here.

def getHostnames():
    global source_hostname
    global target_hostname
    
    if sys.argv[1] == "-service" or sys.argv[1] == "-svc":
        if source_hostname[3] == 'A':
            target_hostname = source_hostname[:3]+'S'+source_hostname[4:]
        elif source_hostname[3] == 'a':
            target_hostname = source_hostname[:3]+'s'+source_hostname[4:]
        elif source_hostname[3] == 'S':
            target_hostname = source_hostname[:3]+'A'+source_hostname[4:]
        elif source_hostname[3] == 's':
            target_hostname = source_hostname[:3]+'a'+source_hostname[4:]
    elif len(sys.argv) == 3:
        source_hostname = sys.argv[1]
        target_hostname = sys.argv[2]
    elif len(sys.argv) == 2:
        target_hostname = sys.argv[1]
    return (source_hostname,target_hostname)

Now that it is clarified which systems will be involved the next step is to connect to RAD. As you can see in lines 99 and 104 I am using ssh to connect to remote systems and Unix socket for local connections as shown in line 102.
Again, the user that executes this script must have sufficient privileges on the involved systems. In addition to that the rad:remote and/or rad:local service has to be enabled and online.

def connectRAD():
    global source_rc
    global target_rc
    
    if len(sys.argv) == 3:
        source_uri = radc.RadURI("ssh://"+source_hostname)
        source_rc = source_uri.connect()
    else:
        source_uri = radc.RadURI("unix:///")
        source_rc = source_uri.connect()
    target_uri = radc.RadURI("ssh://"+target_hostname)
    target_rc = target_uri.connect()

    return (source_rc,target_rc)

The next step is to get a list of zones of each global zone. What it actually is is a list of the objects of each zone.

def getZoneLists():
    global zones_s
    global zones_t
    
    zones_s = source_rc.list_objects(zonemgr.Zone())
    zones_t = target_rc.list_objects(zonemgr.Zone())

    return (source_zones,target_zones)

Each object includes the values of a zone. For example name, state, brand, etc..
The previous is done to get each ng/kzone’s state and therefore to decide whether it is synced or not. Incomplete zones for example are not worth synchronizing. A configured zone’s config will be replaced by the one of a running zone in order to have the most current version configured.

def getSourceZones():
    printHeader(source_hostname)
    for name_s in zones_s:
        zone_s = source_rc.get_object(name_s)
        print "\t%-16s %-11s %-6s" % (zone_s.name, zone_s.state,zone_s.brand)
        if zone_s.state != 'incomplete':
            source_zones.append(zone_s.name)
        if zone_s.state == 'configured':
            source_conf_zones.append(zone_s.name)
    return (source_zones,source_conf_zones) 

def getInstalledTargetZones():
    printHeader(target_hostname)
    for name_t in zones_t:
        zone_t = target_rc.get_object(name_t)
        print "\t%-16s %-11s %-6s" % (zone_t.name, zone_t.state,zone_t.brand)
        if zone_t.state != 'configured' and zone_t.state != 'incomplete':
            target_zones.append(zone_t.name)
        if zone_t.state == 'configured':
            target_conf_zones.append(zone_t.name)
    return (target_zones,target_conf_zones)

After comparing the states, the script deletes existing configurations that are about to be replaced.
In line 152 you can see the preparation of connecting to the target machine’s RAD zonemgr (rad.bindings.com.oracle.solaris.rad.zonemgr_1). The class that is used here is ZoneManager(rad.client.RADInterface)

Quote from the python help for rad.bindings.com.oracle.solaris.rad.zonemgr_1.ZoneManager:


| Create and delete zones. Changes in the state of zones can be
| monitored through the StateChange event.

def deleteExistingConfiguredTargetZone():
    delete_zone = target_rc.get_object(zonemgr.ZoneManager())
    global zones_t

    for name_d in import_zones:
        for name_t in zones_t:
            zone_t = target_rc.get_object(name_t)
            if name_d.name == zone_t.name:
                delete_zone.delete(name_d.name)
                print "DELETED: %s" % name_d.name
                zones_t.remove(name_t)
                break 

As you can see above in line 159 the method used is called delete.

When this is done it is time to export and import configurations.
To export a zone configuration via RAD (line 168) the proper class is called Zone with its method exportConfig(*args, **kwargs). Imports (line 170) are done by using the class ZoneManager and (importConfig(*args, **kwargs) as the method.
zones-sync.py [-h|help] [-service|-svc] [target hostname/ip]

def expImpConfig():
    mgr = target_rc.get_object(zonemgr.ZoneManager())
    for name_i in import_zones:
        zone_i = source_rc.get_object(name_i)
        z_config = zone_i.exportConfig()
        split_config = z_config.splitlines(True)
        mgr.importConfig(False,zone_i.name,split_config)
        print "IMPORTED: %s" % zone_i.name

Well, all that is left to do is to close the connections.

def closeRc():
    source_rc.close()
    target_rc.close()

And for you to have an idea what this may look like and what it does, let’s check out the following outputs.

In the following the script was used with -service. This is the way I use for scheduled/periotic services. The hostname’s pattern is used to define the source and target hostnames.

u2034611@AC6A000:~$ /net/ap6shr1/data/shares/soladm/intern/tm/zones-sync.py -service 
AC6A000: 
        NAME             STATUS      BRAND 
        kzone1           configured  solaris-kz 
        zone2            configured  solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 
AC6S000: 
        NAME             STATUS      BRAND 
        kzone1           running     solaris-kz 
        zone2            installed   solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 

As you can see nothing was deleted or imported. This is what it looks like when everything is in sync.

Let’s tell the script which server is suppose the target in order to sync it with the localhost.

u2034611@AC6A000:~$ /net/ap6shr1/data/shares/soladm/intern/tm/zones-sync.py AC4S000 
AC6A000: 
        NAME             STATUS      BRAND 
        kzone1           configured  solaris-kz 
        zone2            configured  solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 
AC4S000: 
        NAME             STATUS      BRAND  

IMPORTED: kzone1 
IMPORTED: zone2 
IMPORTED: zone3 
IMPORTED: fuu1 
IMPORTED: fu1 
IMPORTED: oo1 
IMPORTED: kzone 
IMPORTED: zone1 

Above you can see that on the one server (AC6A000, in this case the local machine) a bunch of different zones are configured and none exist on the target side. Therefore all of the zone configurations are imported.

Let’s say you have a central server or your local workstation from which you want to sync two global zones. In the following example two hostnames are passed on.

G u2034611@r0065262 % /var/tmp/zones-sync.py ac6s000 10.1.30.107                         
ac6s000: 
        NAME             STATUS      BRAND 
        kzone1           running     solaris-kz 
        zone2            installed   solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris  
10.1.30.107: 
        NAME             STATUS      BRAND 
        kzone1           configured  solaris-kz 
        zone2            configured  solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 

DELETED: kzone1 
DELETED: zone2 
IMPORTED: kzone1 
IMPORTED: zone2 

This time both global zones have several zones in the configured state and on one host one zone is running and another one is in the installed state. Which means that two zone configurations (configured state). The more current ones, which in this case means rather “runnable” or running, are synced and the “only” configured zones’ configuration was deleted.

There are many more things to explore about RAD, so have fun and stay calm ;) !

Attachments

4 Replies to “RAD – syncing Solaris zone configs”

  1. Xpack

    Nice post and very interesting, but how about something more complex like send a zfs snapshot to a remote system, can be done?

    Reply
    1. muehle Post author

      Oh sure.
      You can do pretty much everything you normally do with the zfs command.

      These are the available CLASSES (python -c “help(‘rad.bindings.com.oracle.solaris.rad.zfsmgr_1’)”)

      rad.client.RADEnum(rad.client.RADObject)
      PropSrc
      SocketType
      ZfsOpOption
      ZfsRADErrType
      ZfsRecvNameOptions
      rad.client.RADInterface(rad.client.RADObject)
      ZfsDataset
      ZfsSnapshot
      ZfsUtil
      Zpool
      rad.client.RADStruct(rad.client.RADObject)
      RecvSocketInfo
      SendSocketInfo
      SendStreamInfo
      ZfsProp
      ZfsPropDetail
      ZfsPropRequest
      ZfsRADError
      ZfsSendRecvStatus
      ZpoolVdev

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *