ZFS Device Removal with Solaris 11.4 Beta

The earlier mentioned Solaris 11.4 Beta build refresh is here.

Which means: ZFS DEVICE REMOVAL is available to all of you. :-)

Since I am short of time right now here is the very quick and trivial proof:

root@wacken:~# zpool list
NAME    SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
rpool  99.5G  55.5G  44.0G  55%  1.00x  ONLINE  -
root@wacken:~# for i in 1 2 3;do mkfile 1g diskfile$i;done
root@wacken:~# zpool create rempool /root/diskfile1 /root/diskfile2 /root/diskfile3
root@wacken:~# zpool list
NAME      SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
rempool  2.98G   152K  2.98G   0%  1.00x  ONLINE  -
rpool    99.5G  55.5G  44.0G  55%  1.00x  ONLINE  -
root@wacken:~# zpool status rempool
  pool: rempool
 state: ONLINE
  scan: none requested
config:

        NAME               STATE      READ WRITE CKSUM
        rempool            ONLINE        0     0     0
          /root/diskfile1  ONLINE        0     0     0
          /root/diskfile2  ONLINE        0     0     0
          /root/diskfile3  ONLINE        0     0     0

errors: No known data errors
root@wacken:~# zpool remove rempool /root/diskfile2
root@wacken:~# zpool status rempool
  pool: rempool
 state: ONLINE
  scan: resilvered 1K in 1s with 0 errors on Fri Mar  9 13:08:34 2018

config:

        NAME                      STATE      READ WRITE CKSUM
        rempool                   ONLINE        0     0     0
          /root/diskfile1         ONLINE        0     0     0
          /root/diskfile3         ONLINE        0     0     0

errors: No known data errors

Works!

This is what it looks like when there is not enough space left for removing a device from a pool.

root@wacken:~# zpool status rempool
  pool: rempool
 state: ONLINE
  scan: none requested
config:

        NAME               STATE      READ WRITE CKSUM
        rempool            ONLINE        0     0     0
          /root/diskfile1  ONLINE        0     0     0
          /root/diskfile2  ONLINE        0     0     0
          /root/diskfile3  ONLINE        0     0     0

errors: No known data errors
root@wacken:~# zpool list rempool
NAME      SIZE  ALLOC  FREE  CAP  DEDUP  HEALTH  ALTROOT
rempool  2.98G  2.54G  443M  85%  1.00x  ONLINE  -
root@wacken:~# zpool remove rempool /root/diskfile2
cannot remove device(s): not enough space to migrate data

It actually doesn’t matter what your top-level vdev is. Her is an example of a multiple mirrored vdevs:

root@wacken:~# zpool destroy rempool
root@wacken:~# zpool create rempool2 mirror /root/diskfile1 /root/diskfile2 mirror /root/diskfile3 /root/diskfile4 mirror /root/diskfile5 /root/diskfile6
root@wacken:~# zpool status
  pool: rempool2
 state: ONLINE
  scan: none requested
config:

        NAME                 STATE      READ WRITE CKSUM
        rempool2             ONLINE        0     0     0
          mirror-0           ONLINE        0     0     0
            /root/diskfile1  ONLINE        0     0     0
            /root/diskfile2  ONLINE        0     0     0
          mirror-1           ONLINE        0     0     0
            /root/diskfile3  ONLINE        0     0     0
            /root/diskfile4  ONLINE        0     0     0
          mirror-2           ONLINE        0     0     0
            /root/diskfile5  ONLINE        0     0     0
            /root/diskfile6  ONLINE        0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: resilvered 0 in 5s with 0 errors on Thu Mar  8 23:27:12 2018

config:

        NAME    STATE      READ WRITE CKSUM
        rpool   ONLINE        0     0     0
          c1d0  ONLINE        0     0     0

errors: No known data errors
root@wacken:~# zpool remove rempool2 mirror-1
root@wacken:~# zpool status
  pool: rempool2
 state: ONLINE
  scan: resilvered 1.50K in 1s with 0 errors on Tue Apr 10 05:25:51 2018

config:

        NAME                      STATE      READ WRITE CKSUM
        rempool2                  ONLINE        0     0     0
          mirror-0                ONLINE        0     0     0
            /root/diskfile1       ONLINE        0     0     0
            /root/diskfile2       ONLINE        0     0     0
          mirror-2                ONLINE        0     0     0
            /root/diskfile5       ONLINE        0     0     0
            /root/diskfile6       ONLINE        0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: resilvered 0 in 5s with 0 errors on Thu Mar  8 23:27:12 2018

config:

        NAME    STATE      READ WRITE CKSUM
        rpool   ONLINE        0     0     0
          c1d0  ONLINE        0     0     0

errors: No known data errors

Very nice additional improvement to an already feature-rich and fantastic release!

Solaris is dead – long lives Solaris

The year 2018 just started and Oracle just released Solaris 11.4 Beta.
Only a few months after the media and social networks called Oracle Solaris dead, and if it was to them it would have been buried and gone by now.
Oracle drew the curtain on what was formerly known as Solaris 12. Oracle Solaris 11.4 is as mentioned by quite a few people (Oracle and non-Oracle) before not just another Solaris 11 release. It is new! Go grab the public beta version and find out yourself. The list of changes, improvements and new features is long.
I am in the privileged position that I have been using this for a couple of years now. I saw it grow, shrink, mutate and evolve. Glad it is finally available publicly.

A lot of people have worked very hard on this over the last couple of years.
And no matter where all of you are now or what you work on now, CONGRATULATIONS! Well done!

Update: After a forced offline time I will see check the blogs and get finally going with cool features and enhancements.

Oracle Solaris Blogs

Enterprise is not enterprise

Over the last couple of weeks and months I looked a bit more into Linux than before and especially Red Hat. Just to check out what new features and technologies it offers and also if anything might have improved that was already offered in the past. The list of possible topics for posts about comparisons between Linux and Unix and pros and cons grew rapidly and was too long. But one thing became clearer and clearer – Enterprise is not enterprise. Putting the word enterprise in your name is not either.

Calling yourself Enterprise is not the same as being called enterprise.

As I mentioned, the list of topics is long but there is a perfect example that I want to use in this post to emphasize my statement above.

One might get the feeling that I believe Linux or in this case Red Hat is totally garbage. No, it’s not. There are things it fits better than Solaris. But now it is about enterprise. The Red Hat Product Security Center for example is fantastic. I love it. Lots of information and tools to stay on top of security topics.
But at the same time the Red Hat Product Security Center is making me wonder why the name of the Linux distribution includes the word Enterprise.
As Red Hat shows on their website they are able to provide a great value of data in terms of compliance, cve, etc., but where is this value when it comes to the operating system? Gone!

Openscap has been available in Red Hat Linux longer than in Oracle Solaris but just making it available is nothing but a nice offer that takes the work of downloading and installing away from the customer and that’s it. That’s just as much enterprise as offering a shell with default settings or any other application/program.

I am well aware that the meaning of the word enterprise depends on quiet a lot of factors and mostly just on the subjective point of view. It might be facts like scalability, stability, usability, different performance aspects or the rate of consolidation one can achieve by using the product. These and many others are important but often depend on what you need it for.
One thing though always matters and that’s the rate and quality the product improves and matures. Wouldn’t it make you sad and mad at the same time when you have to put quiet a lot of time, nerves and effort into something that is already there but not passed on to you as a customer? Have you ever used the openscap command? As great as the tool is just as annoying it is to use and especially to get going with it. As usual, once you get the hang of it it’s ok. But really not more than that. What happened the last couple of years while the security topic got pushed all the way to the front of IT? Well, let’s look at the facts. Red Hat created a nice, no actually an extremely nice, website with a fantastic security section. You will get all the information on for example current CVEs that you want and need. Really enterprise like if you ask me! Chapeau! Love it. I actually use it as an example of what I expect when I talk to the Oracle Solaris team. Yesterday was the last time even. :-) Greetings, guys.
So obviously there was some work that happened on the compliance topic. Here is what did not happen over the years. Nothing close to what the Red Hat Product Security Center offers can be easily done when you are running a Red Hat Enterprise Linux server. Let me just show you a “simple” command used to start a compliance run of a certain benchmark for RHEL and Solaris.

RHEL:

# oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rht-ccp --results scan-xccdf-results.xml /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml

Solaris 11:

# compliance assess -p pci-dss

In case you want to run the benchmark and profile that is set as the default you even only need the following:

# compliance assess

Can you tell the difference? This is a customer orientated implementation. And again, it is just one small example.
Well, there really is not much more to say. I just much rather spend money on enterprise features and implementations than Enterprise names.
For all the admins and engineers go and check it out. And even if you don’t want to use Solaris and just simply can’t bother your OS vendor. Sorry it is RHEL in this case but that’s just because I am closer to RHEL than to other distributions.

All in all, I promise you, it is worth checking it out! Have fun.

Tailoring meets Solaris Compliance

With Solaris 11.3 Oracle addded a new feature to compliance. Tailoring it is called and pretty much does exactly that. Instead of having to manually customize benchmark files tailoring will do the job for you. That’s the trivial description of what tailoring does.
But underneath the hood tailoring is capable of so much more. Used the right way it takes the automation of compliance reporting to a more sophisticated level.

How to get started

Before talking about how tailoring can enhance the way you use and customize compliance in Solaris let me quickly walk you through how it works.
Using tailoring is as simple and intuitive as running an assessment. All you need to do is type “compliance tailor -t “. The -t option declares which tailoring shall be loaded. In case none exists it will be created. It is not a required option but in order to store the tailoring you will have to set the name manually by using “set tailoring=” later on anyway.

Example without the option:

ROOT@AP6S500 > compliance tailor

Documented commands (type help ):
========================================
clear   delete   exit    include  list  pick  value 
commit  exclude  export  info     load  set   values

Miscellaneous help topics:
==========================
tailoring

tailoring> set tailoring=tailoring.tm
tailoring:tailoring.tm> info
Properties:
        tailoring=tailoring.tm
        benchmark: not set
        profile: not set
tailoring:tailoring.tm>

Example with -t:

ROOT@AP6S500 > compliance tailor -t tailoring.tm2 
*** compliance tailor: Can't load tailoring 'tailoring.tm2': no existing tailoring: 'tailoring.tm2', initializing

tailoring:tailoring.tm2> info
Properties:
        tailoring=tailoring.tm2
        benchmark: not set
        profile: not set

As the examples already showed the tailoring CLI command info shows which tailoring, benchmark and profile are set.
From this point on you could use set …=… all the way till your tailoring is done and you commit it. If you rather would like to save some time and typing pick will be the command of your choice.

tailoring:tailoring.tm2> pick

tailoring-pick-tm
Use the arrow-keys to navigate up and down and pick the benchmark and profile that you would like to take for your tailoring. This can be seen as sort of a template. When you have done your selection pres ESC. info will show what you selected.

tailoring:tailoring.tm2> info
Properties:
        tailoring=tailoring.tm2
        benchmark=tm
        profile=tm

tailoring, benchmark and profile are set, which means tests can be picked now.

tailoring:tailoring.tm2> pick

tailoring-pick-tests
The picture above shows the tests of the earlier chosen benchmark and profile. “x” stands for excluded while “>” indicates an activated test. This is where you tailor your compliance check. As before press “ESC” when you are done.
With the command export yo can see what changes you have made. The output that is shown then are the commands that can be used to manually include and exclude tests instead of using pick.

tailoring:tailoring.tm2> export
set tailoring=tailoring.tm2
# version=2016-02-26T16:44:36.000+00:00
set benchmark=tm
set profile=tm
tailoring:tailoring.tm2> pick
tailoring:tailoring.tm2> export
set tailoring=tailoring.tm2
# version=2016-02-26T17:02:10.000+00:00
set benchmark=tm
set profile=tm
# ivv-000: Compliance integrity is given
exclude ivv-000
# ivv-001: LDAP client configuration is ok
include ivv-001
# OSC-54005: Package integrity is verified
exclude OSC-54005
# OSC-53005: The OS version is current
exclude OSC-53005
# OSC-53505: Package signature checking is globally activated
exclude OSC-53505

Should you be interested in how the tailoring file itself will look like simply use the option -x. This will give you the XML output.
tailoring-export-x
All that is left to do is commit your changes et voilá … exit and done!
In case you have been fiddling around and create a few tailorings already the list will list all the existing tailorings.

Tailoring vs. Benchmarks/Profiles only

After we flew through the basics of Solaris compliance tailoring we are already know enough to talk about why EVERYONE should use tailoring.
Maybe you have read one or even all of my earlier Solaris Compliance posts or heard me talking about it, if you might remember me saying it is really quiet fast and simple to customize. Well, it just got way easier. Not all out of the box yet but almost and I am sure someone already requested an enhancement. :-D
So what am I talking about?!
The files for Solaris compliance can be found under two paths. One is /usr/lib/compliance. This was probably the only one that you might have been working in in case you customized anything. For adding benchmarks, adding tests or editing profiles this was/is where you do it. Other than that all the content here is pretty much static until a change might come with an update (SRU). With Solaris 11.3 and tailoring the compliance benchmark directories received another directory called tailorings. By default this is empty.
All the changes and information done while using the compliance command are done under /var/share/compliance. It is important to understand that this content should stay untouched. Just leave this path to Solaris and the engineering. But it is always nice and helpful to know where to look for changes.
Let’s take a look at /var/share/compliance/tailorings.

G muehle@AP6S500 % ls -l /var/share/compliance/tailorings 
total 60
-rw-r--r--   1 root     root         495 Feb 16 14:21 ivv-tailor.xccdf.xml
-rw-r--r--   1 root     root         964 Feb 16 14:05 tailoring.tm.xccdf.xml
-rw-r--r--   1 root     root         952 Feb 26 18:07 tailoring.tm2.xccdf.xml
-rw-r--r--   1 root     root         489 Feb 17 14:03 test.xccdf.xml
-rw-r--r--   1 root     root       24844 Feb 17 15:11 test123.xccdf.xml

This is the place compliance tailor saves the tailorings after committing it. The content of /var/share/compliance/tailorings/tailoring.tm2.xccdf.xml is exactly what export -x showed us earlier.

Another very interesting directory is /var/share/compliance/assessments. I will write more about why this is hopefully soon. I am working on customizing Solaris compliance for a larger scale environment and this directory plays an important role for that.

But let’s get back on track and talk about how much of an enhancement tailoring is.
At the moment we have different IPS packages with different benchmarks. Each with different profiles. Just so different scenarios are covered.
Which means we spend some time customizing large XML files and we also do have to spend time on maintaining it.
Now, all we do is package up your tailoring file or a compliance tailor -f command file with includes and excludes in IPS. Less complexity and less maintaining! No more duplicating lines and lines of code only to have a different set of tests that is suppose to be used.
When you think about it tailorings are the delta to a certain benchmark. So, what if you would have one large benchmark that includes all the available tests and let’s say a preconfigured profile for solaris, pci-dss and a “complete profile”. To cerate your own profile just place your tailoring in /usr/lib/compliance/benchmark/benchmark-name/tailorings/ and run the following:

# compliance assess -t tailoring-name

Using different tests depending on the application has become really simple and quick to prepare and do. Your tailoring works everywhere no matter if a benchmarks has tests included or excluded. Really nice! Add IPS and Puppet to all of this and you can much more time on other topics.

Right now this “complete” benchmark needs to be created by the customer. Not much of a problem if you already took care of that but I would guess not too many have. But even if you have your own all containing benchmark with each update you might be missing something in it. Tests or what so ever. So you still have to maintain thousands of lines of XML content. :-(
So hopefully such a benchmark will make it into a future release of compliance.

Tailoring simplifies Solaris Compliance a lot and saves you a lot of time. It is great! Try it!

New Compliance Report Design

Oracle just released the beta version of Solaris 11.3 which means we finally get to use new features and improvements.

If you already use Solaris compliance you will run right into one of the improvements that come with the latest release. A new design for the html report. In case you haven’t used compliance yet (you should definately take a look) or just don’t remember what a html compliance report looked like with Solaris 11.2 here is a small example.

compliance-basic

Quiet static besides the links to the details of each check.
With Solaris 11.3 bootstrap is used to give the html reports a new look and feel. And it is great! Here is an example of the new design.

compliance.report113

Besides the fresh look the major improvement that comes with bootstrap is flexibility. Whenever you wanted to see only passed or failed checks you ran “compliance report -s …” twice. One report for passed and one for failed checks in your report. Now you just need one report that includes all the checks and you choose what kind of result will be visible. Multiple selections are possible.

compliance.report113.selection

As you can see you can also search for certain rules/checks. Which is quiet handy.
Moreover an enhanced grouping of the results is included now too. All in all the new design gives the user a better overview of the results and additional information.

Solaris 11.3 compliance has way more than just this to offer but it’s the small changes that matter too.