New RAD in town with Oracle Solaris 11.4 Beta

The journey of making an admins life easier continues with the next release of Oracle Solaris.
Solaris 11.4 beta comes with quiet a view new RAD modules. While Solaris 11.3 included 12 modules

system/management/rad/module/rad-dlmgr            0.5.11-0.175.3.29.0.4.0    i--
system/management/rad/module/rad-evs-controller   0.5.11-0.175.3.32.0.1.0    ---
system/management/rad/module/rad-files            0.5.11-0.175.3.29.0.4.0    i--
system/management/rad/module/rad-kstat            0.5.11-0.175.3.29.0.4.0    i--
system/management/rad/module/rad-network          0.5.11-0.175.3.32.0.1.0    i--
system/management/rad/module/rad-panels           0.5.11-0.175.3.29.0.4.0    i--
system/management/rad/module/rad-smf              0.5.11-0.175.3.29.0.4.0    i--
system/management/rad/module/rad-time             0.5.11-0.175.3.29.0.4.0    i--
system/management/rad/module/rad-usermgr          0.5.11-0.175.3.32.0.1.0    i--
system/management/rad/module/rad-zfsmgr           0.5.11-0.175.3.32.0.1.0    i--
system/management/rad/module/rad-zonemgr          0.5.11-0.175.3.32.0.1.0    i--
system/management/rad/module/rad-zones-bridge     0.5.11-0.175.1.0.0.14      --o

Solaris 11.4 doubles the amount of modules. Yes, 24 (12 additional) RAD modules free to use to programatically control your Solaris servers.

system/management/rad/module/rad-archivemgr       11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-bemgr            11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-compliance       11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-dlmgr            11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-etraceprovider   11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-etracequery      11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-etraceserver     11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-evs-controller   11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-files            11.4-11.4.0.0.0.1.0        --o
system/management/rad/module/rad-ips              11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-kstat            11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-labelmgr         11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-network          11.4-11.4.0.0.0.1.0        --o
system/management/rad/module/rad-odocprovider     11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-panels           11.4-11.4.0.0.0.1.0        --o
system/management/rad/module/rad-smf              11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-sstore           11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-sysmgr           11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-time             11.4-11.4.0.0.0.1.0        --o
system/management/rad/module/rad-usermgr          11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-webuiprefs       11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-zfsmgr           11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-zonemgr          11.4-11.4.0.0.1.3.1        i--
system/management/rad/module/rad-zones-bridge     0.5.11-0.175.1.0.0.14      --o

Very exciting.
If you haven’t used RAD or looked into it yet, do it. Once you get started you will not want to stop using it.

RAD – get ZFS properties

After starting out with Python RAD zonemgr module I thought it is time to write about another available Python RAD module. In the following I want to give you a short and simple look and feel of how to get ZFS property information (name, type, atime, compression, compressratio, dedup, mounted and mountpoint). The puprpose is to get people interested in Solaris going with RAD and show how easily it can be used.

Let’s start with the usual imports which are pretty self-explaining. One is needed to open a connection and the other one depends on the purpose of the script. In this case we want to get ZFS properties which means we will use zfsmgr.
In case it is not already installed just run: pkg install rad-zfsmgr

import rad.connect as radc
import rad.bindings.com.oracle.solaris.rad.zfsmgr_1 as zfsmgr

Next we need to connect to a RAD client. In this case it is a local one and a unix socket can be used.
If you like to use a remote connection go ahead and use ssh://[hostname] instead.

uri = radc.RadURI("unix:///")
rc = uri.connect()

We now have an open connection (rc) and we can move on to the ZFS part.
First we need is to know what the existing datasets are. rc.list_object(zfsmgr.ZfsDataset()) will do exactly this for us. It lists all the ZFS dataset objects (ZfsDataset) the RAD zfs manager (zfsmgr) has to offer for the chosen connection (rc).

zfsDataSets = rc.list_objects(zfsmgr.ZfsDataset())

Well, the list of datasets is complete. But we will need more than just the objects of each dataset.
In order to get the information we are interested in we need to define it first. Therefore ZfsPropRequest is used.

prop0 = zfsmgr.ZfsPropRequest(name="name")
prop1 = zfsmgr.ZfsPropRequest(name="type")
prop2 = zfsmgr.ZfsPropRequest(name="atime")
prop3 = zfsmgr.ZfsPropRequest(name="compression")
prop4 = zfsmgr.ZfsPropRequest(name="compressratio")
prop5 = zfsmgr.ZfsPropRequest(name="dedup")
prop6 = zfsmgr.ZfsPropRequest(name="mounted")
prop7 = zfsmgr.ZfsPropRequest(name="mountpoint")

After defining the properties we just loop through each of the object list of the ZFS datasets and request the value for the just defined keys of the current object (zobj).

for dataset in zfsDataSets:
    zobj = rc.get_object(dataset)
    zvalues = zobj.get_props([prop0, prop1, prop2, prop3, prop4, prop5, prop6, prop7])
    print "%-40s%-14s%-8s%-13s%-15s%-7s%-9s%s" % (zvalues[0].value, zvalues[1].value, zvalues[2].value, zvalues[3].value, zvalues[4].value, zvalues[5].value, zvalues[6].value, zvalues[7].value)

Done. Got the information and therefore can close the connection at this point.

rc.close()

The output will remind you of a regular zfs list -o … output. One may say why should I use RAD then and the answer is quiet simple. Because this is just a trivial example of how you can make use RAD of the zfsmgr to get dataset information. The next steps would be to use the above and automate whatever comes to your mind. Juggle around with the objects, keys, values, etc.. Add more functions to it and even combine it with more RAD modules (e.g.: rad-zonemgr). That’s where you will benefit the most. But even small automation tasks are perfect for this.

Last but not least, here is an example of what it might look like. I had to take out a few lines because it included Solaris beta content.

rad-zfs2

Remember, the purpose was to make the very first step with RAD together with ZFS. Try it out and you will most probably like it and stick to it.

RAD – syncing Solaris zone configs

RAD

A less known jewel of Solaris 11 is RAD (Remote Administration Daemon). Since, as I just found out, I don’t have any RAD posts yet, let’s talk about what it is and offers before we go on.
what RAD does is it provides programmatic interfaces to manage Solaris. Users, zones, zfs and smf are just a few examples. RAD offers APIs for C, Java, Python and REST. It can be used locally as well as remotely. It can be used to read data but also to write data. As an example you can get zfs informations as well as create new datasets or change current settings. There are a couple of great examples and posts out there from e.g. Glynn Foster and Robert Milkowski

Why am I telling you this? Because you can PROGRAMMATICALLY manage your enterprise operating system now.

Use case

Imagine an environment of SPARC T4, T5, T7 or S7 servers running quiet few non-global zones whether in LDOMs or not.
Over the last weeks I tested kernel zones pretty heavily. The chance of getting rid of LDOMs is just too good to not go for it. Don’t understand me wrong, LDOMs work fine and are a key part of our current DR concept. But there are also things I really don’t like at all. Guess this will make a good post in the nearer future. ;)
As I said, I was using kernel zones but for DR purposes (let’s say one data center dies) I need to be able to boot the kzones from the other data center. In order to do so the zone configuration has to be available. Well, shared storage and so on too but for the purpose of this post let’s say that’s all taking care of automatically (it really is ;) ).
At the moment Zone configurations are saved via a SMF service on a NFS server. But I don’t want to create zones first while the angry mob can’t work out there and tries to figure out where I am sitting.
When I used kernel zone live migration I started thinking about how I want to solve this issue. For those who haven’t used kzone live migration yet, it creates a zone configuration on the target side and leaves the old one in configured state. Which means once you have live migrated a zone the problem seems to be solved. But what if something changes? What if it runs on a different server (hw) by the time of a disaster.
These two facts, programmatic interfaces and LDOM replacement, lead me to the idea of actually having a SMF scheduled service that takes care of zone configurations. For that I use RAD’s zonemanager and Python.
Besides the RAD IPS packages (rad and rad-zonemgr) being installed you will need a user with sufficient privileges for rad and zones administration/management.

RAD Python

The script zones-sync is part of a larger RAD script that makes all sort of stuff.
Simply said it checks which zone’s configuration is missing on a target server and then imports/creates it. Quiet trivial.

Let’s start wit the imports.

import rad.connect as radc
import rad.bindings.com.oracle.solaris.rad.zonemgr_1 as zonemgr
import string
import socket
import sys

Line 36 let’s you connect to RAD as the name already says while the import in line 37 adds the ability to us zone management. Quiet self-explaining I would say.

In order to know which global zones are suppose to be synced I started with getting the source and target hostname straight, depending on the used arguments. So, if only one hostname is given the other one will be considered to be localhost. For remote purposes two hostnames need to be provided. For the purpose of automation I added [-service|-svc] which in this case maps certain pattern of hostnames. The name pattern is used to find the corresponding global zone.
In the end anything that helps to automate getting the hostnames should be put here.

def getHostnames():
    global source_hostname
    global target_hostname
    
    if sys.argv[1] == "-service" or sys.argv[1] == "-svc":
        if source_hostname[3] == 'A':
            target_hostname = source_hostname[:3]+'S'+source_hostname[4:]
        elif source_hostname[3] == 'a':
            target_hostname = source_hostname[:3]+'s'+source_hostname[4:]
        elif source_hostname[3] == 'S':
            target_hostname = source_hostname[:3]+'A'+source_hostname[4:]
        elif source_hostname[3] == 's':
            target_hostname = source_hostname[:3]+'a'+source_hostname[4:]
    elif len(sys.argv) == 3:
        source_hostname = sys.argv[1]
        target_hostname = sys.argv[2]
    elif len(sys.argv) == 2:
        target_hostname = sys.argv[1]
    return (source_hostname,target_hostname)

Now that it is clarified which systems will be involved the next step is to connect to RAD. As you can see in lines 99 and 104 I am using ssh to connect to remote systems and Unix socket for local connections as shown in line 102.
Again, the user that executes this script must have sufficient privileges on the involved systems. In addition to that the rad:remote and/or rad:local service has to be enabled and online.

def connectRAD():
    global source_rc
    global target_rc
    
    if len(sys.argv) == 3:
        source_uri = radc.RadURI("ssh://"+source_hostname)
        source_rc = source_uri.connect()
    else:
        source_uri = radc.RadURI("unix:///")
        source_rc = source_uri.connect()
    target_uri = radc.RadURI("ssh://"+target_hostname)
    target_rc = target_uri.connect()

    return (source_rc,target_rc)

The next step is to get a list of zones of each global zone. What it actually is is a list of the objects of each zone.

def getZoneLists():
    global zones_s
    global zones_t
    
    zones_s = source_rc.list_objects(zonemgr.Zone())
    zones_t = target_rc.list_objects(zonemgr.Zone())

    return (source_zones,target_zones)

Each object includes the values of a zone. For example name, state, brand, etc..
The previous is done to get each ng/kzone’s state and therefore to decide whether it is synced or not. Incomplete zones for example are not worth synchronizing. A configured zone’s config will be replaced by the one of a running zone in order to have the most current version configured.

def getSourceZones():
    printHeader(source_hostname)
    for name_s in zones_s:
        zone_s = source_rc.get_object(name_s)
        print "\t%-16s %-11s %-6s" % (zone_s.name, zone_s.state,zone_s.brand)
        if zone_s.state != 'incomplete':
            source_zones.append(zone_s.name)
        if zone_s.state == 'configured':
            source_conf_zones.append(zone_s.name)
    return (source_zones,source_conf_zones) 

def getInstalledTargetZones():
    printHeader(target_hostname)
    for name_t in zones_t:
        zone_t = target_rc.get_object(name_t)
        print "\t%-16s %-11s %-6s" % (zone_t.name, zone_t.state,zone_t.brand)
        if zone_t.state != 'configured' and zone_t.state != 'incomplete':
            target_zones.append(zone_t.name)
        if zone_t.state == 'configured':
            target_conf_zones.append(zone_t.name)
    return (target_zones,target_conf_zones)

After comparing the states, the script deletes existing configurations that are about to be replaced.
In line 152 you can see the preparation of connecting to the target machine’s RAD zonemgr (rad.bindings.com.oracle.solaris.rad.zonemgr_1). The class that is used here is ZoneManager(rad.client.RADInterface)

Quote from the python help for rad.bindings.com.oracle.solaris.rad.zonemgr_1.ZoneManager:


| Create and delete zones. Changes in the state of zones can be
| monitored through the StateChange event.

def deleteExistingConfiguredTargetZone():
    delete_zone = target_rc.get_object(zonemgr.ZoneManager())
    global zones_t

    for name_d in import_zones:
        for name_t in zones_t:
            zone_t = target_rc.get_object(name_t)
            if name_d.name == zone_t.name:
                delete_zone.delete(name_d.name)
                print "DELETED: %s" % name_d.name
                zones_t.remove(name_t)
                break 

As you can see above in line 159 the method used is called delete.

When this is done it is time to export and import configurations.
To export a zone configuration via RAD (line 168) the proper class is called Zone with its method exportConfig(*args, **kwargs). Imports (line 170) are done by using the class ZoneManager and (importConfig(*args, **kwargs) as the method.
zones-sync.py [-h|help] [-service|-svc] [target hostname/ip]

def expImpConfig():
    mgr = target_rc.get_object(zonemgr.ZoneManager())
    for name_i in import_zones:
        zone_i = source_rc.get_object(name_i)
        z_config = zone_i.exportConfig()
        split_config = z_config.splitlines(True)
        mgr.importConfig(False,zone_i.name,split_config)
        print "IMPORTED: %s" % zone_i.name

Well, all that is left to do is to close the connections.

def closeRc():
    source_rc.close()
    target_rc.close()

And for you to have an idea what this may look like and what it does, let’s check out the following outputs.

In the following the script was used with -service. This is the way I use for scheduled/periotic services. The hostname’s pattern is used to define the source and target hostnames.

u2034611@AC6A000:~$ /net/ap6shr1/data/shares/soladm/intern/tm/zones-sync.py -service 
AC6A000: 
        NAME             STATUS      BRAND 
        kzone1           configured  solaris-kz 
        zone2            configured  solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 
AC6S000: 
        NAME             STATUS      BRAND 
        kzone1           running     solaris-kz 
        zone2            installed   solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 

As you can see nothing was deleted or imported. This is what it looks like when everything is in sync.

Let’s tell the script which server is suppose the target in order to sync it with the localhost.

u2034611@AC6A000:~$ /net/ap6shr1/data/shares/soladm/intern/tm/zones-sync.py AC4S000 
AC6A000: 
        NAME             STATUS      BRAND 
        kzone1           configured  solaris-kz 
        zone2            configured  solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 
AC4S000: 
        NAME             STATUS      BRAND  

IMPORTED: kzone1 
IMPORTED: zone2 
IMPORTED: zone3 
IMPORTED: fuu1 
IMPORTED: fu1 
IMPORTED: oo1 
IMPORTED: kzone 
IMPORTED: zone1 

Above you can see that on the one server (AC6A000, in this case the local machine) a bunch of different zones are configured and none exist on the target side. Therefore all of the zone configurations are imported.

Let’s say you have a central server or your local workstation from which you want to sync two global zones. In the following example two hostnames are passed on.

G u2034611@r0065262 % /var/tmp/zones-sync.py ac6s000 10.1.30.107                         
ac6s000: 
        NAME             STATUS      BRAND 
        kzone1           running     solaris-kz 
        zone2            installed   solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris  
10.1.30.107: 
        NAME             STATUS      BRAND 
        kzone1           configured  solaris-kz 
        zone2            configured  solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 

DELETED: kzone1 
DELETED: zone2 
IMPORTED: kzone1 
IMPORTED: zone2 

This time both global zones have several zones in the configured state and on one host one zone is running and another one is in the installed state. Which means that two zone configurations (configured state). The more current ones, which in this case means rather “runnable” or running, are synced and the “only” configured zones’ configuration was deleted.

There are many more things to explore about RAD, so have fun and stay calm ;) !

Attachments