How to set up ILB for an apache server

In order to install implement a loadbalancing via ILB for the WLS’ apache server you need to be able to set an additional default route in the apache ng-zone. Therefore the ip-type should be set to exclusive and the LDOM needs to be configured with enough alt-mac-addr.
So these are the steps for setting up an ILB environment and configurational changes needed on the existing apache ng-zone.

As mentioned before it is recommended to use exclusive ip-type for the ng-zones’ configurations.

In this HowTo I am creating an ip-address for each IP used in a ILB rule for vip. This way ipadm takes care of the arp entries. If you do not want to assign ip-addresses you can just use

 arp -s 10.1.34.125 00:14:4f:f8:e2:c0 pub permanent

instead.

These are the actual commands it takes:

on the ILB ng-zone (ilb-zone1)

ROOT@ilb-zone1 > pkg install ilb

ROOT@ilb-zone1 > routeadm -u -e ipv4-forwarding

ROOT@ilb-zone1 > routeadm
              Configuration   Current              Current
                     Option   Configuration        System State
---------------------------------------------------------------
               IPv4 routing   disabled             disabled
               IPv6 routing   disabled             disabled
            IPv4 forwarding   enabled              enabled
            IPv6 forwarding   disabled             disabled

           Routing services   "route:default ripng:default"

Routing daemons:

                      STATE   FMRI
                   disabled   svc:/network/routing/ripng:default
                     online   svc:/network/routing/ndp:default
                   disabled   svc:/network/routing/route:default
                   disabled   svc:/network/routing/rdisc:default
                   disabled   svc:/network/routing/legacy-routing:ipv4
                   disabled   svc:/network/routing/legacy-routing:ipv6

ROOT@ilb-zone1 > svcadm enable ilb

ROOT@ilb-zone1 > dladm
LINK                CLASS     MTU    STATE    OVER
ilb0                vnic      1500   up       ?
ilb1                vnic      1500   up       ?

ROOT@ilb-zone1 > ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
ilb0              ip         ok           --         --
   ilb0/v4        static     ok           --         10.1.34.124/24
lo0               loopback   ok           --         --
   lo0/v4         static     ok           --         127.0.0.1/8
   lo0/v6         static     ok           --         ::1/128

ROOT@ilb-zone1 > ipadm create-ip ilb1

ROOT@ilb-zone1 > ipadm create-addr -T static -a local=10.1.34.125/24 ilb1/test1

ROOT@ilb-zone1 > ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
ilb0              ip         ok           --         --
   ilb0/v4        static     ok           --         10.1.34.124/24
ilb1              ip         ok           --         --
   ilb1/test1     static     ok           --         10.1.34.125/24
lo0               loopback   ok           --         --
   lo0/v4         static     ok           --         127.0.0.1/8
   lo0/v6         static     ok           --         ::1/128

ROOT@ilb-zone1 > arp -a
Net to Media Table: IPv4
Device   IP Address               Mask      Flags      Phys Addr
------ -------------------- --------------- -------- ---------------
...
ilb1   10.1.34.125          255.255.255.255 SPLA     00:14:4f:f8:e2:c0
ilb0   ilb-zone1              255.255.255.255 SPLA     00:14:4f:fa:26:f7
...

ROOT@ilb-zone1 > ilbadm create-servergroup -s server=10.1.34.118 web-test1

ROOT@ilb-zone1 > ilbadm show-sg
SGNAME         SERVERID            MINPORT MAXPORT IP_ADDRESS
web-test1      _web-test1.0        --      --      10.1.34.118

ROOT@ilb-zone1 > ilbadm create-hc -h hc-timeout=3,hc-count=2,hc-interval=8,hc-test=PING web-test_hc1

ROOT@ilb-zone1 > ilbadm create-hc -h hc-timeout=3,hc-count=2,hc-interval=8,hc-test=TCP web-test_hc2

ROOT@ilb-zone1 > ilbadm show-hc
HCNAME        TIMEOUT COUNT   INTERVAL DEF_PING TEST
web-test_hc1  3       2       8        Y        PING
web-test_hc2  3       2       8        Y        TCP

ROOT@ilb-zone1 > ilbadm create-rule -e -p -i vip=10.1.34.125,port=80,protocol=tcp -m lbalg=rr,type=HALF-NAT,pmask=24 -h hc-name=web-test_hc1 -o servergroup=web-test1 test-rule1

ROOT@ilb-zone1 > ilbadm create-rule -e -p -i vip=10.1.34.125,port=443,protocol=tcp -m lbalg=rr,type=HALF-NAT,pmask=24 -h hc-name=web-test_hc2 -o servergroup=web-test1 test-rule2

ROOT@ilb-zone1 > ilbadm create-rule -e -p -i vip=10.1.34.125,port=52224,protocol=tcp -m lbalg=rr,type=HALF-NAT,pmask=24 -h hc-name=web-test_hc2 -o servergroup=web-test1 test-rule3

ROOT@ilb-zone1 > ilbadm show-hc-result
RULENAME      HCNAME        SERVERID      STATUS   FAIL LAST     NEXT     RTT
test-rule1    web-test_hc1  _web-test1.0  alive    0    17:04:39 17:04:51 174
test-rule2    web-test_hc2  _web-test1.0  alive    0    17:04:45 17:04:55 194
test-rule3    web-test_hc2  _web-test1.0  alive    0    17:04:45 17:04:54 177

ROOT@ilb-zone1 > ilbadm show-rl
RULENAME            STATUS LBALG       TYPE    PROTOCOL VIP         PORT
test-rule1          E      roundrobin  HALF-NAT TCP 10.1.34.125     80
test-rule2          E      roundrobin  HALF-NAT TCP 10.1.34.125     443
test-rule3          E      roundrobin  HALF-NAT TCP 10.1.34.125     52224

ROOT@ilb-zone1 > ilbadm show-rl -f
       RULENAME: test-rule1
         STATUS: E
           PORT: 80
       PROTOCOL: TCP
          LBALG: roundrobin
           TYPE: HALF-NAT
      PROXY-SRC: --
          PMASK: /24
        HC-NAME: web-test_hc1
        HC-PORT: ANY
     CONN-DRAIN: 0
    NAT-TIMEOUT: 120
PERSIST-TIMEOUT: 60
    SERVERGROUP: web-test1
            VIP: 10.1.34.125
        SERVERS: _web-test1.0

       RULENAME: test-rule2
         STATUS: E
           PORT: 443
       PROTOCOL: TCP
          LBALG: roundrobin
           TYPE: HALF-NAT
      PROXY-SRC: --
          PMASK: /24
        HC-NAME: web-test_hc2
        HC-PORT: ANY
     CONN-DRAIN: 0
    NAT-TIMEOUT: 120
PERSIST-TIMEOUT: 60
    SERVERGROUP: web-test1
            VIP: 10.1.34.125
        SERVERS: _web-test1.0

       RULENAME: test-rule3
         STATUS: E
           PORT: 52224
       PROTOCOL: TCP
          LBALG: roundrobin
           TYPE: HALF-NAT
      PROXY-SRC: --
          PMASK: /24
        HC-NAME: web-test_hc2
        HC-PORT: ANY
     CONN-DRAIN: 0
    NAT-TIMEOUT: 120
PERSIST-TIMEOUT: 60
    SERVERGROUP: web-test1
            VIP: 10.1.34.125
        SERVERS: _web-test1.0

on the apache ng-zone (apache-ngz1)

ROOT@apache-ngz1 > dladm
LINK                CLASS     MTU    STATE    OVER
vnic1               vnic      1500   up       ?
vnic2               vnic      1500   up       ?

ROOT@apache-ngz1 > ipadm create-ip vnic1

ROOT@apache-ngz1 > ipadm create-ip vnic2

ROOT@apache-ngz1 > ipadm create-addr -T static -a local=10.1.34.118/24 vnic1/v4

ROOT@apache-ngz1 > ipadm create-addr -T static -a local=10.1.34.123/24 vnic2/v4

ROOT@apache-ngz1 > ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
lo0               loopback   ok           --         --
   lo0/v4         static     ok           --         127.0.0.1/8
   lo0/v6         static     ok           --         ::1/128
vnic1             ip         ok           --         --
   vnic1/v4       static     ok           --         10.1.34.118/24
vnic2             ip         ok           --         --
   vnic2/v4       static     ok           --         10.1.34.123/24

ROOT@apache-ngz1 > netstat -r

Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface 
-------------------- -------------------- ----- ----- ---------- --------- 
10.1.34.0            10.1.34.123          U         3          1 vnic2     
10.1.34.0            10.1.34.118          U         3          3 vnic1     
apache-ngz1              apache-ngz1              UH        2          0 lo0       

Routing Table: IPv6
  Destination/Mask            Gateway                   Flags Ref   Use    If   
--------------------------- --------------------------- ----- --- ------- ----- 
apache-ngz1                     apache-ngz1                     UH      2       0 lo0   

ROOT@apache-ngz1 > route -p add default 10.1.34.1
add net default: gateway 10.1.34.1
add persistent net default: gateway 10.1.34.1

ROOT@apache-ngz1 > route -p add default 10.1.34.125
add net default: gateway 10.1.34.125
add persistent net default: gateway 10.1.34.125

ROOT@apache-ngz1 > netstat -r

Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface 
-------------------- -------------------- ----- ----- ---------- --------- 
default              10.1.34.125          UG        1          0           
default              10.1.34.1 UG        1          0           
10.1.34.0            apache2-ngz1 U         3          4 vnic2     
10.1.34.0            apache-ngz1 U         3          3 vnic1     
apache-ngz1              apache-ngz1              UH        2          0 lo0       

Routing Table: IPv6
  Destination/Mask            Gateway                   Flags Ref   Use    If   
--------------------------- --------------------------- ----- --- ------- ----- 
apache-ngz1                     apache-ngz1                     UH      2       0 lo0   

This are already all the configurations that are needed.

What is left is testing and monitoring.
Open a browser and use the following URL to check if accessing apache directly still works:

http://apache-ngz1/server-status

Now we want to find out if the access via the ILB works. Therefore use the URL

http://10.1.34.125/server-status

Use the ilbadm show-stats command to monitor what the ilb is doing.

ROOT@ilb-zone1 > ilbadm show-stats -it -s _web-test1.0 2
RULENAME            PKT_P   BYTES_P   TIME
test-rule1          33      20230     2013-02-16:17.08.53
test-rule2          42      9269      2013-02-16:17.08.53
test-rule3          0       0         2013-02-16:17.08.53
test-rule1          0       0         2013-02-16:17.08.55
test-rule2          0       0         2013-02-16:17.08.55
test-rule3          0       0         2013-02-16:17.08.55
test-rule1          0       0         2013-02-16:17.08.57
test-rule2          0       0         2013-02-16:17.08.57
test-rule3          0       0         2013-02-16:17.08.57
test-rule1          0       0         2013-02-16:17.08.59
test-rule2          0       0         2013-02-16:17.08.59
test-rule3          0       0         2013-02-16:17.08.59
test-rule1          0       0         2013-02-16:17.09.01
test-rule2          0       0         2013-02-16:17.09.01
test-rule3          0       0         2013-02-16:17.09.01
test-rule1          18      13954     2013-02-16:17.09.03
test-rule2          0       0         2013-02-16:17.09.03
test-rule3          0       0         2013-02-16:17.09.03
test-rule1          0       0         2013-02-16:17.09.05
test-rule2          13      3813      2013-02-16:17.09.05
test-rule3          0       0         2013-02-16:17.09.05

ROOT@ilb-zone1 > ilbadm show-stats -vt -s _web-test1.0 2
PKT_P   BYTES_P   PKT_U   BYTES_U   PKT_D   BYTES_D   ICMP_P  ICMP_D  ICMP2BIG_P ICMP2BIG_D NOMEMP_D NOPORTP_D NOMEMB_D NOPORTB_D TIME
154     66042     0       0         0       0         0       0       0          0          0       0       0       0       2013-02-16:17.10.55
0       0         0       0         0       0         0       0       0          0          0       0       0       0       2013-02-16:17.10.57
0       0         0       0         0       0         0       0       0          0          0       0       0       0       2013-02-16:17.10.59
0       0         0       0         0       0         0       0       0          0          0       0       0       0       2013-02-16:17.11.01
20      14182     0       0         0       0         0       0       0          0          0       0       0       0       2013-02-16:17.11.03
0       0         0       0         0       0         0       0       0          0          0       0       0       0       2013-02-16:17.11.05
12      1593      0       0         0       0         0       0       0          0          0       0       0       0       2013-02-16:17.11.07

Leave a Reply

Your email address will not be published. Required fields are marked *