RAD – syncing Solaris zone configs


A less known jewel of Solaris 11 is RAD (Remote Administration Daemon). Since, as I just found out, I don’t have any RAD posts yet, let’s talk about what it is and offers before we go on.
what RAD does is it provides programmatic interfaces to manage Solaris. Users, zones, zfs and smf are just a few examples. RAD offers APIs for C, Java, Python and REST. It can be used locally as well as remotely. It can be used to read data but also to write data. As an example you can get zfs informations as well as create new datasets or change current settings. There are a couple of great examples and posts out there from e.g. Glynn Foster and Robert Milkowski

Why am I telling you this? Because you can PROGRAMMATICALLY manage your enterprise operating system now.

Use case

Imagine an environment of SPARC T4, T5, T7 or S7 servers running quiet few non-global zones whether in LDOMs or not.
Over the last weeks I tested kernel zones pretty heavily. The chance of getting rid of LDOMs is just too good to not go for it. Don’t understand me wrong, LDOMs work fine and are a key part of our current DR concept. But there are also things I really don’t like at all. Guess this will make a good post in the nearer future. ;)
As I said, I was using kernel zones but for DR purposes (let’s say one data center dies) I need to be able to boot the kzones from the other data center. In order to do so the zone configuration has to be available. Well, shared storage and so on too but for the purpose of this post let’s say that’s all taking care of automatically (it really is ;) ).
At the moment Zone configurations are saved via a SMF service on a NFS server. But I don’t want to create zones first while the angry mob can’t work out there and tries to figure out where I am sitting.
When I used kernel zone live migration I started thinking about how I want to solve this issue. For those who haven’t used kzone live migration yet, it creates a zone configuration on the target side and leaves the old one in configured state. Which means once you have live migrated a zone the problem seems to be solved. But what if something changes? What if it runs on a different server (hw) by the time of a disaster.
These two facts, programmatic interfaces and LDOM replacement, lead me to the idea of actually having a SMF scheduled service that takes care of zone configurations. For that I use RAD’s zonemanager and Python.
Besides the RAD IPS packages (rad and rad-zonemgr) being installed you will need a user with sufficient privileges for rad and zones administration/management.

RAD Python

The script zones-sync is part of a larger RAD script that makes all sort of stuff.
Simply said it checks which zone’s configuration is missing on a target server and then imports/creates it. Quiet trivial.

Let’s start wit the imports.

import rad.connect as radc
import rad.bindings.com.oracle.solaris.rad.zonemgr_1 as zonemgr
import string
import socket
import sys

Line 36 let’s you connect to RAD as the name already says while the import in line 37 adds the ability to us zone management. Quiet self-explaining I would say.

In order to know which global zones are suppose to be synced I started with getting the source and target hostname straight, depending on the used arguments. So, if only one hostname is given the other one will be considered to be localhost. For remote purposes two hostnames need to be provided. For the purpose of automation I added [-service|-svc] which in this case maps certain pattern of hostnames. The name pattern is used to find the corresponding global zone.
In the end anything that helps to automate getting the hostnames should be put here.

def getHostnames():
    global source_hostname
    global target_hostname
    if sys.argv[1] == "-service" or sys.argv[1] == "-svc":
        if source_hostname[3] == 'A':
            target_hostname = source_hostname[:3]+'S'+source_hostname[4:]
        elif source_hostname[3] == 'a':
            target_hostname = source_hostname[:3]+'s'+source_hostname[4:]
        elif source_hostname[3] == 'S':
            target_hostname = source_hostname[:3]+'A'+source_hostname[4:]
        elif source_hostname[3] == 's':
            target_hostname = source_hostname[:3]+'a'+source_hostname[4:]
    elif len(sys.argv) == 3:
        source_hostname = sys.argv[1]
        target_hostname = sys.argv[2]
    elif len(sys.argv) == 2:
        target_hostname = sys.argv[1]
    return (source_hostname,target_hostname)

Now that it is clarified which systems will be involved the next step is to connect to RAD. As you can see in lines 99 and 104 I am using ssh to connect to remote systems and Unix socket for local connections as shown in line 102.
Again, the user that executes this script must have sufficient privileges on the involved systems. In addition to that the rad:remote and/or rad:local service has to be enabled and online.

def connectRAD():
    global source_rc
    global target_rc
    if len(sys.argv) == 3:
        source_uri = radc.RadURI("ssh://"+source_hostname)
        source_rc = source_uri.connect()
        source_uri = radc.RadURI("unix:///")
        source_rc = source_uri.connect()
    target_uri = radc.RadURI("ssh://"+target_hostname)
    target_rc = target_uri.connect()

    return (source_rc,target_rc)

The next step is to get a list of zones of each global zone. What it actually is is a list of the objects of each zone.

def getZoneLists():
    global zones_s
    global zones_t
    zones_s = source_rc.list_objects(zonemgr.Zone())
    zones_t = target_rc.list_objects(zonemgr.Zone())

    return (source_zones,target_zones)

Each object includes the values of a zone. For example name, state, brand, etc..
The previous is done to get each ng/kzone’s state and therefore to decide whether it is synced or not. Incomplete zones for example are not worth synchronizing. A configured zone’s config will be replaced by the one of a running zone in order to have the most current version configured.

def getSourceZones():
    for name_s in zones_s:
        zone_s = source_rc.get_object(name_s)
        print "\t%-16s %-11s %-6s" % (zone_s.name, zone_s.state,zone_s.brand)
        if zone_s.state != 'incomplete':
        if zone_s.state == 'configured':
    return (source_zones,source_conf_zones) 

def getInstalledTargetZones():
    for name_t in zones_t:
        zone_t = target_rc.get_object(name_t)
        print "\t%-16s %-11s %-6s" % (zone_t.name, zone_t.state,zone_t.brand)
        if zone_t.state != 'configured' and zone_t.state != 'incomplete':
        if zone_t.state == 'configured':
    return (target_zones,target_conf_zones)

After comparing the states, the script deletes existing configurations that are about to be replaced.
In line 152 you can see the preparation of connecting to the target machine’s RAD zonemgr (rad.bindings.com.oracle.solaris.rad.zonemgr_1). The class that is used here is ZoneManager(rad.client.RADInterface)

Quote from the python help for rad.bindings.com.oracle.solaris.rad.zonemgr_1.ZoneManager:

| Create and delete zones. Changes in the state of zones can be
| monitored through the StateChange event.

def deleteExistingConfiguredTargetZone():
    delete_zone = target_rc.get_object(zonemgr.ZoneManager())
    global zones_t

    for name_d in import_zones:
        for name_t in zones_t:
            zone_t = target_rc.get_object(name_t)
            if name_d.name == zone_t.name:
                print "DELETED: %s" % name_d.name

As you can see above in line 159 the method used is called delete.

When this is done it is time to export and import configurations.
To export a zone configuration via RAD (line 168) the proper class is called Zone with its method exportConfig(*args, **kwargs). Imports (line 170) are done by using the class ZoneManager and (importConfig(*args, **kwargs) as the method.
zones-sync.py [-h|help] [-service|-svc] [target hostname/ip]

def expImpConfig():
    mgr = target_rc.get_object(zonemgr.ZoneManager())
    for name_i in import_zones:
        zone_i = source_rc.get_object(name_i)
        z_config = zone_i.exportConfig()
        split_config = z_config.splitlines(True)
        print "IMPORTED: %s" % zone_i.name

Well, all that is left to do is to close the connections.

def closeRc():

And for you to have an idea what this may look like and what it does, let’s check out the following outputs.

In the following the script was used with -service. This is the way I use for scheduled/periotic services. The hostname’s pattern is used to define the source and target hostnames.

u2034611@AC6A000:~$ /net/ap6shr1/data/shares/soladm/intern/tm/zones-sync.py -service 
        NAME             STATUS      BRAND 
        kzone1           configured  solaris-kz 
        zone2            configured  solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 
        NAME             STATUS      BRAND 
        kzone1           running     solaris-kz 
        zone2            installed   solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 

As you can see nothing was deleted or imported. This is what it looks like when everything is in sync.

Let’s tell the script which server is suppose the target in order to sync it with the localhost.

u2034611@AC6A000:~$ /net/ap6shr1/data/shares/soladm/intern/tm/zones-sync.py AC4S000 
        NAME             STATUS      BRAND 
        kzone1           configured  solaris-kz 
        zone2            configured  solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 
        NAME             STATUS      BRAND  

IMPORTED: kzone1 
IMPORTED: zone2 
IMPORTED: zone3 
IMPORTED: kzone 
IMPORTED: zone1 

Above you can see that on the one server (AC6A000, in this case the local machine) a bunch of different zones are configured and none exist on the target side. Therefore all of the zone configurations are imported.

Let’s say you have a central server or your local workstation from which you want to sync two global zones. In the following example two hostnames are passed on.

G u2034611@r0065262 % /var/tmp/zones-sync.py ac6s000                         
        NAME             STATUS      BRAND 
        kzone1           running     solaris-kz 
        zone2            installed   solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 
        NAME             STATUS      BRAND 
        kzone1           configured  solaris-kz 
        zone2            configured  solaris 
        zone3            configured  solaris 
        fuu1             configured  solaris 
        fu1              configured  solaris 
        oo1              configured  solaris 
        kzone            configured  solaris 
        zone1            configured  solaris 

DELETED: kzone1 
DELETED: zone2 
IMPORTED: kzone1 
IMPORTED: zone2 

This time both global zones have several zones in the configured state and on one host one zone is running and another one is in the installed state. Which means that two zone configurations (configured state). The more current ones, which in this case means rather “runnable” or running, are synced and the “only” configured zones’ configuration was deleted.

There are many more things to explore about RAD, so have fun and stay calm ;) !


Benefits of Solaris Immutable Zones

Over the last couple of weeks or actually months I was lucky to talk to a lot of other Oracle Solaris customers and other Linux and Unix users/admins. One question that I got asked most of the time is why I use immutable zones? Where is the benefit if your data can still be read and stolen?
Since I finally got some time for a blog post I figured I will share the answer with whoever might be interested. Originally I planned on writing a HowTo post ever since this feature was released. But time has passed by and the more important question at the moment seems to be the WHY rather then the HOW.

The answer to why I use immutable (global)zones is security, simplicity, and speed. And all of this for no extra costs.
Let me explain this in more detail.


Security is often just looked at as something that keeps your data safe and protects the IT from attackers/hackers. This is definitely a part of what security is but there is so much more to it. Why are mostly hackers, attacker or let’s say external people considered a threat to the system but hardly ever the admins or users on the system itself. Why trust yourself? And what is it I want to protect or prevent?

Immutable zones aims for preventing data manipulation and protects you from headless admin mistakes like rm -r * in the wrong directory, wrong terminal or what so ever, misconfiguration of the system, and therefor of course also from anyone (attackers or users) reconfiguring your system. Yes, the data can be read but not changed. I am very sure most of you who read this have been dealing with a sudden appearance of a fault and when the question is asked what happened what was changed nobody has done anything. Why bother with this question??? I don’t want to think about what the application users might be doing in /etc or hope that the new admin is not going to destroy datasets, cleans out directories or even mis-configures RBAC.
Immutable zones ensures me that the system stays exactly the same way until I change something intentionally.

That was the technical point of view. But a growing field of security is compliance. I wonder how much money and time is spend by companies just to be able to somehow ensure the auditor that your system configuration is compliant throughout the year. And even more how much was spend to make sure it did stay the same. Scripts were written, mistakes were corrected, you got more gray hairs over the last couple of months and the meetings with the auditors will most probably not be the highlight of the year. Save time and money and just tell/show the auditor that the system was and is immutable (read-only) and therefor nothing changed!!! Easiest way to do so by the way is use the Solaris compliance framework.

Another very important fact is that this is not achieved by just mounting datasets read-only. It is rather deeply integrated in Solaris and based on privileges.


The chances are high that not every out there is working with Solaris but rather Windows or any Linux distro. So let me start with a short comment what simplicity does not mean to me as a Solaris guy. Simplicity does not mean I don’t have to install additional software or spend time on integrating a feature. That is what I am used to and what is normal to me.
I just start using security features or in case of for example RBAC “have” to use it. It is always there.

So what is it that makes immutable zones simple then? Well, let me just show you the steps it takes to turn a non-immutable zone into an immutable one.

root@GZ:~# zoneadm list -cv
  ID NAME             STATUS      PATH                         BRAND      IP
   0 global           running     /                            solaris    shared
   2 ap1s001          running     /zones/ap1s001               solaris    excl

root@GZ:~# zonecfg -z ap1s001 set file-mac-profile=fixed-configuration

root@GZ:~# zlogin ap1s001 init 6

That’s it
You enable immutable Zones by simply changing the file-mac-profile value to either strict(be careful!), fixed-configuration or flexible-configuration and then reboot the zone. For the global zone dynamic-zones is available as well. In case you want to go back to a regular type of zone just use none as a value for file-mac-profile.
Here is a quote of the zonecfg man page:

none makes the zone exactly the same as a normal, r/w zone. strict
allows no exceptions to the read-only policy. fixed-configuration
allows the zone to write to files in and below /var, except direc-
tories containing configuration files:


dynamic-zones is equal to fixed-configuration but allows creating
and destroying non-global zones and kernel zones. This profile is
only valid for global zones, including the global zone of a kernel

flexible-configuration is equal to dynamic-zones, but allows writ-
ing to files in /etc in addition.

zoneadm list -p shows whether a zone is immutable or not.

root@GZ:~# zoneadm list -p

Listed are the fields zoneid:zonename:state:zonepath:uuid:brand:ip-type:r/w:file-mac-profile.

To add more simplicity a trusted path (zlogin -T|U) from the global zone / console can be used to do necessary changes. For example adding a directory or adding a user.
You also don’t have to do wild things when it comes to updating/patching. Just use the pkg update command as you always do.

As you can see it is just simple!


Thanks to the sophisticated integration of immutable zones there is no overhead. No software installed on top of the operating system. No daemon running and checking fo/preventing open system calls or what so ever. Immutable zones run just as fast as non-immutable zones.
Changing back and worth is just a reboot. I could imagine this might not even be necessary anymore somewhen.
Besides that this feature will speed up the auditor meetings as mentioned before.
And the process of setting it up is lightning fast compared to other tools out there.
Not worrying about your configuration of your system anymore will speed up other projects/topics you are working on by saving time and thoughts/distractions.

Short answer

To tell you the truth this is what my very first answer to the why question always is before getting into details:

Why not?!?!?! Why shouldn’t I use a security feature that is there for free, that works and is a no-brainer to use?!
I don’t wanna worry about my own stupid mistakes or even the ones of others. I don’t trust application admins/users. Auditor meetings are over before they even begin. It’s just a great feature!

As I said at the beginning I was lucky to be able to talk to quiet a lot of different admins, engineers, managers, etc. and it was really nice to see how most of them started thinking “Why not, true!”. This feature might not fit in every single environment. But does every machine have IPsec, IPfilter, … enabled? Probably not.

I hope this will encourage some of you to make your own experience with this great Solaris feature.