Thursday, July 8, 2010

Firmware

Briefly, What is Firmware?


Firmware is a low level program which lies between the hardware and OS and is used only when the particular hardware is initialized before OS is loaded and not after that.The firmware of a system board does the initialization of its hardware devices cpu /mem and then loads the OS boot loader to memory (In AIX this firmware is called ROS program which provides us the SMS menu)Apart of main system firmware, there are pieces of firmware inside the controller chips of adapters and some disks/tapes to initialize/control their operations called micro codes.System Microcode initializes the system hardware and controls the boot process enabling the system to boot up and operate correctly; it also provides the interface between the operating system software and the hardware. Adapter Microcode is the operating code of the adapter; it initializes the adapter when power is applied and it controls many of the ongoing operations executed by the adapter. Device Microcode (Similar to adapter microcode) provides these same functions for devices such as tape drives, disks etc.Firmwares in IBM eServer pSeries, and IBM System p5.These systems (and many other IBM systems) include a Service Processor, which contains System Firmware and other key System code. High-end systems also include "Bulk Power Controllers" (BPC) which each have a separate Service Processor. In addition, a System Power Control Network provides the interface to the BPCs or other power controllers.

The Flexible Service Processor (FSP) firmware provides diagnostics, initialization, configuration, run-time error detection, and correction.

The Power Hypervisor firmware (which is based on the pSeries hypervisor) provides Partitioning, VLAN, virtual I/O, and subprocessor partitioning support.

The Platform Firmware (PFW) supports the "Power Architecture Platform Requirements+" interface

The Bulk Power Control (BPC) firmware controls each bulk power unit in CEC and towers. This firmware is model dependent.

The System Power Control Network (SPCN) firmware interfaces with bulk power for power monitoring and control. In addition, many systems are likely to have a Hardware Management Console (indeed, it is a requirement for all systems which have Bulk Power Controllers). An HMC is required for Logical Partitioning (LPAR), Service Focal Point etc.The Hardware Management
Console (HMC) firmware provides platform configuration, management, and services.


HMC Command line than graphics

To view HMC information like serial number, Model, Version, release, hostname its lshmc -v and -V
hscroot@apdochmc1p:~> lshmc -v
"vpd=*FC ????????
*VC 20.0
*N2 Thu Jul 08 18:49:42 SGT 2010
*FC ????????
*DS Hardware Management Console
*TM 7042-CR4
*SE SERLNUMBER
*MN IBM
*PN Unknown
*OS Embedded Operating Systems
*NA
*FC ????????
*DS Platform Firmware
*RM V7R3.4.0.0
"
hscroot@apdochmc1p:~>
hscroot@apdochmc1p:~>
hscroot@apdochmc1p:~> lshmc -V
"version= Version: 7
Release: 3.4.0
Service Pack: 0
HMC Build level 20081204.1
MH01152: Required fix for HMC V7R3.4.0 (12-04-2008)
","base_version=V7R3.4.0
"
hscroot@apdochmc1p:~>

To view HMC Network configuration

#lshmc -n

To show processor usage by partition

#lshwres -m msname -r proc --level lpar -F lpar_name,run_procs --header

To start a virtual console

#mkvterm -m mansystem -p lparname


To enable ssh remote execution from another server without password

#ssh-keygen -t rsa as 'root' in NIM server (or the host from u connect)
#scp hmchostname:.ssh/authorized_keys2 ~/.ssh/temp_hmcauthkeys2

To copy the hmc keys file to local

#cat ~/.ssh/id_rsa.pub >> ~/.ssh/temp_hmcauthkeys2

To add pub keys from local system

#scp ~/.ssh/temp_hmcauthkeys2 hmchostname:.ssh/authorized_keys2

To update hmc keys file
On first login it will ask for keys exchange give yes



There are thre options to start a manged system from hmc

1 Partition standby ( from here one by one partitions are started)
2 System profile ( they are collection of partition profiles named for usage. select a profile and start)
3 Partition auto start ( the partitions are started with last running configuration)

#chsysstate -m mndsys -o onstandby -r sys
#chsysstate -m mndsys -o onsysprof -r sys -f profilename
#chsysstate -m mndsys -o on -r sys

respectively

To view state of manged system

#lssyscfg -r sys -m managedsys -F state --header( header to list options under header)

To poweroff

Shutdown all lpars from os. Or use hmc option operating system shutdown
Then power off managed system
#chsysstate -r sys -m managedsys -o off

MEMORY FOR LPARS

LMB - is the size of unit of memory can be allocated to a partition. You have to add remove or adjust an lpars memory amount in multiples of LMB.

LMB varies depending on the Total Physical memory available as below.

4GB n less LMB 16 MB default
4-8 GB 32 MB default
8-16 GB 64MB and so on to 256MB

Partition profiles - Configuration for lpars. Contains lpar_name, memory, cpu, IO, startup mode etc.
System profile - Collection of partition profiles. An lpar can have multiple profiles with different mem/cpu/io parameters provided that only one profile is active at a time. For example you have a set of UAT servers to run on less resources at the begining of the month you can put those profiles in to a group managed sys profile and activate /deactivate easily.

Order of creating profiles - Create a partition with default profile, then create additional profiles, create system profile

CPU, MEM Allocation for lpars

Minimum - Lpar won't start without the minimum even to the firmware level. An lpar will be running memory/cpu much more than the minimum normally depends on the desired amount selected. The amount can be reduced as low as minimum using dlpar property online.

Desired - Lpar Starts with this amount if available or an amount between the minmum and desired if not fully available.

Maximum- Normally not used by startup. We can add resources (mem/cpu) up to this amount using dlpar on the fly.

An amount of memory is used by the hypervisor, Translation Control Entries, page tables for each partition (memory mapping) etc
for 32 to 64 system memory partition table takes 1GB
from 64-128 2GB and so on

To list memory info
#lshwres -r mem -m mndsys --level sys

Installed mem - Total physical memory available in system.
Deconfigured - faulty
Configurable - Total physical memory after reducing faulty mem or COD memory inactive amount if any
Available - Free after usage by the lpars


I/O resources are added as either required or desired

Required - lpar won't start if not available.
Desired - Can be moved using dlpar, Lpar starts even when they not available..

To know the lpar info from aix

# uname -L

To Know the HMC name from an lpar commandline

#lsrsrc IBM.ManagementServer

To activate partition

#chsysstate -m mndsys -r lpar -o on -f profilename -n LPARname

For a manged system the states are

Power off -off
Initializing - powering on
Standby - partition standby mode
Operating - At least 1 partition on
Pending authorzation - hmc is yet to get authorized with the HMC user password of SP
Authorization failure - The hmc user password mismatch
Incomplete - hmc not able to gather proper info
No connection - handshake for ip conn failed
Recovery - hmc / SP nvram mismatch. You may need to restore from backup or initialize system with SP contents
Error- error need to log call with IBM
Cod click to accept - enter activation code for COD

To view state of managed systems

#lssyscfg -r sys -m mandsys -F state --header

To open terminal console to lpar

#mkvterm -m mndsys -p lparname

To exit from console press ~.

To remove an active console connection

#rmvterm -m mndsys -p lparname

For stopping partition

#chsysstate -m mndsys -o shutdown -r lpar -n lparname --immed (after os is hsutdown)
or
#chsysstate -m mndsys -o osshutdown -r lpar -n lparname (Before os is hsutdown)


The managed systems profiles backup is taken with command

#bkprofdata -m mndsysname


To restore

#rstprofdata -m mndsys -f /mn/floppy/mybackup -l 1 (full restore) 2 ( bavckup prior) 3 (mandsys priority)

To see processor info lpar vice

#lshwres -m mandsys -r proc --level lpar

To see dedicated or virtual proc info

#lshwres -m mangedsys -r proc --level lpar -F lpar_name,run_procs --header

To see magedsys mem info

#lshwres -r mem -m mndsys --level sys

To see mem by lpar

#lshwres -r mem -m mndsys --level lpar

To view info virualio scsi slots

#lshwres-r virtualio --rsubtype scsi -m mndsys

For moving memory from one partition to other using DLPAR

#chhwres -r mem -m SYS:9131-52A-SNXXE2G -o m -p lparapp1t -t lparapp1d -q 28672

For adding memory in one

#chhwres -r mem -m SYS:9131-52A-SNXXE2G -o a -p lparapp1d -q 28672

For adding processors in one partition

#chhwres -r proc -m SYS:9131-52A-SNXXE2G -o a -p lparapp1d --procs 3

For moving processor from one partition to other

#chhwres -r proc -m SYS:9131-52A-SNXXE2G -o m -p lparapp1d -t fincnapp1t --procs 3


To restart partition

#chsysstate -m SYS:9131-52A-SNXXE2G -r lpar -o shutdown --id 3 --immed --restart


To reactivate partition

#chsysstate -m SYS:9131-52A-SNXXE2G -r lpar -o shutdown --id 3 --immed
#chsysstate -r lpar -m SYS:9131-52A-SNXXE2G -n lparapp1d -o on -f lparapp1d_profile


To update managed system from microcode rpm file copied under hscroot home dir of hmc.

#updlic -m SYS:9117-570-SNXXE2G-t all -l latest -o a -r mountpoint -d /home/hscroot/Microcode

To view microcode after update

#lslic -t sys -m SYS:9117-570-SNXXE2G

To delete entries of mndsys from exclude file for non detecting Manageddsystem SP objects

#mksysconn -o auto

To list all systems connections to HMC

#lssyconn –r all

To see remote access settings for HMC

#lshmc –r

To change remote access settings

#chhmc -c xntp -s enable

To backup critical console data to DVD

#bkconsdata -r dvd (It can also be backed up to nfs or ftp site.)

To save upgrade data (Normally to a particular area of disk before upgrade)

#saveupgdata -r disk (or for saving to disk and then dvd) saveupgdata -r diskdvd

To restore upgrade data

#rstupgdata (or rstupgdata –r dvd to get data from dvd. Once command issued HMC restores it on the reboot.)

To update hmc code from local file

#updhmc –t l –f /home/hscroot/Update1.zip

To create an hmc user

#mkhmcusr -u sysadmin -a hmcsuperadmin

To change attribute of hmc user

#chhmcusr -u tester -t taskrole -v hmcoperator
To list HMC serviceable events for last three days

#lssvcevents -t hardware -d 3 (-t console for hmc related)

To close all serviceable events

#chsvcevent -o closeall

To activate COD Advanced power virtualization features (To enter feature code from HMC)

#chvet -m mySystem -o e -k AlphaNumericString12345

To view COD features activation log for a managed system

#lsvet -m 9117-570FC54678 -t hist

To display CUoD processor capacity information of a manged system

#lscod -m 9406-570*98127634 -t cap -r proc -c cuod

To enter a CoD code

#chcod -m sys1 -o e -k AlphaNumericString12345

To activate 2 GB of On/Off CoD memory for 10 days

#chcod -m 9406-570*1001CA -o a -c onoff -r mem -q 2048 -d 10

To increase to 5 GB of On/Off CoD memory for 3 days

#chcod -m 9406-570*1001CA -o a -c onoff -r mem -q 5120 -d 3

To activate 3 Reserve CoD processors

#chcod -m 9406-570*1001CA -o a -c reserve -r proc -q 3

To remove temporary HMC files which have not been modified during the last 36 hours from the /var file system

#chhmcfs -o f -h 36 -f /var

To view hmc filesystems

#lshmcfs

To display the physical SA LED for the system:

#lsled -m 9406-579*100103C -r sa -t phys

To display the virtual partition SA LEDs for partitions lpar1 and lpar2:

#lsled -m system1 -r sa -t virtuallpar --filter ""lpar_names=lpar1,lpar2""





To list all storage devices available on HMC

#lsmediadev

To migrate the partition with ID 5 from managed system system1 to managed system system2:

#migrlpar -o m -m system1 -t system2 --id 5 -n migprof -i "virtual_scsi_mappings=12//1, dest_msp_id=1,source_msp_id=3"

To Create new LPAR using command line

#mksyscfg -r lpar -m MACHINE -i name=LPARNAME, profile_name=normal, lpar_env=aixlinux, shared_proc_pool_util_auth=1,min_mem=512, desired_mem=2048, max_mem=4096, proc_mode=shared, min_proc_units=0.2, desired_proc_units=0.5,max_proc_units=2.0, min_procs=1, desired_procs=2, max_procs=2, sharing_mode=uncap, uncap_weight=128,boot_mode=norm, conn_monitoring=1, shared_proc_pool_util_auth=1


If we have multiple lpars to create you can use the below method of specitying them in a file name and creating.


#vi lpar_profiles.txt

name=LPAR1,profile_name=normal,lpar_env=aixlinux,all_resources=0,min_mem=1024,desired_mem=9216,max_mem=9216,proc_mode=shared,min_proc_units=0.3,desired_proc_units=1.0,max_proc_units=3.0,min_procs=1,desired_procs=3,max_procs=3,sharing_mode=uncap,uncap_weight=128,lpar_io_pool_ids=none,max_virtual_slots=10,"virtual_scsi_adapters=6/client/4/vio1a/11/1,7/client/9/lpar1/11/1","virtual_eth_adapters=4/0/3//0/1,5/0/4//0/1",boot_mode=norm,conn_monitoring=1,auto_start=0,power_ctrl_lpar_ids=none,work_group_id=none,shared_proc_pool_util_auth=1
name=LPAR2,profile_name=normal,lpar_env=aixlinux,all_resources=0,min_mem=1024,desired_mem=9216,max_mem=9216,proc_mode=shared,min_proc_units=0.3,desired_proc_units=1.0,max_proc_units=3.0,min_procs=1,desired_procs=3,max_procs=3,sharing_mode=uncap,uncap_weight=128,lpar_io_pool_ids=none,max_virtual_slots=10,"virtual_scsi_adapters=6/client/4/vio1a/12/1,7/client/9/lpar2/12/1","virtual_eth_adapters=4/0/3//0/1,5/0/4//0/1",boot_mode=norm,conn_monitoring=1,auto_start=0,power_ctrl_lpar_ids=none,work_group_id=none,shared_proc_pool_util_auth=1

Copy this file to HMC and run

#mksyscfg -r lpar -m MANGDSYS -f /home/hscroot/lpar_profiles.txt

Commands to edit partition profiles

To change a partition profile's memory amounts (reduce the profile's current memory amounts each by 256 MB), and number of desired processors:

#chsyscfg -r prof -m sys1 -i "name=profile1, lpar_name=partition3,min_mem-=256,desired_mem-=256, max_mem-=256,desired_procs=2"

To change a system profile (add 2 new partition profiles):

#chsyscfg -r sysprof -m sys1 -i "name=sysprof1, "lpar_names+=partition3,partition4", "profile_names+=3_prof1,4_defaultProf""



While adding /removing IO slots to profile or while moving IO slots we need to use Drc Index number for the particular IO slot. It can be seen from the commands

#lshwres -r io -m FINACLE-TAIWA-9117-570-ORQSrv01 --rsubtype slot grep lpar_name or drc_index number


Scheduling operations like managed systems on/off, activate system profiles for managed systems and activate/stop, conduct DLPAR operations for lpars is to be done from GUI.

HMC and FSP

Flexible Service Processor Basics and Connecting a Pseries system to HMC

Flexible Service Processor is the component which resides as part of the system board or as a card in your P-series system which provides diagnostics, initialization, configuration, run-time error detection and correction.
It contains the firmware which boots as sooon as AC Power is given and handles the startup of LPARs communication to HMC etc in a partitioned environment.

HMC

The HMC is a dedicated desktop PC workstation that provides several functions for configuring and operating pSeries servers functioning either partitioned or in
the Full System Partition, using the graphical user interface1(GUI) or command line interface2 (CLI). The functions provided by HMC include:

Logical partitioning management

The HMC provides a set of tasks that are necessary to manage logical
partitions. These tasks include:

– Starting, stopping, resetting, and shutting down a partition.

– Opening a virtual console for each partition or connected pSeries server
system.

– Creating partition profiles that define the processor, memory, and I/O
resources allocated to an individual partition.

Connecting FSP (Or the managed system) to HMC

When the FSP is in it's factory default configuration, it's Ethernet ports are configured as DHCP clients. When AC power applied to the system unit, the
FSP will automatically begin it's boot process. Assuming that a network link on either of the FSP's Ethernet ports is in an active
state, The FSP will issue a DHCP request on the active port(s).

Note that active means that the link light is illuminated which indicates that the FSP is communicating with another device on the
network (but NOT necessarily an HMC) The DHCP request will be issued during the first 60 second period after AC power is
applied however the completion of the boot process to FSP standby will take 3 to 5 min. The DHCP request process will be
repeated every time one of the following actions is performed:

• AC power is cycled.
• The FSP is reset (rebooted) via the Operator Panel pinhole reset method
• The ASMI Reset Service Processor menu option is utilized

Notes:
1) A common misconception is that if a Managed System has had AC power applied to it before the HMC is configured
that extensive recovery steps (flipping FSP Dip switches, removing time of day battery etc) must be performed to allow
the FSP to communicate correctly. This is NOT the case!
2) In addition to the 3 actions listed above, numerous other ASMI options exist that will cause the FSP to reboot when
certain configuration changes have been made.
3) There is one other condition under which the FSP will generate a DHCP request. If the FSP Ethernet link is not
active (no cable connected) during the FSP boot process and then an active connection is made after the FSP is
booted to standby, a DHCP request will now be generated. Keep in mind that this operation will occur as soon as the
link goes active, regardless of what device is on the other end of the link.
Example: If an Ethernet switch is in the path between the FSP and the HMC, the connection between the switch and
the HMC needs to be established before the connection between the FSP and the switch otherwise, the DHCP request
will not be seen by the HMC.
4) Future Firmware releases beginning with the 01SF225 release will also allow a DHCP request to be generated
anytime the network link is dropped for approximately 60 seconds and then re-established.
Now we’ve established how and when a DHCP request is generated. What happens next depends on whether the FSP receives
a response to it's request:


Scenario #1: Response to DHCP request is NOT received

The FSP will assign the following default IP addresses to it's ports:
HMC1 - 192.168.2.147
HMC2 - 192.168.3.147
The assignment of these default addresses allows the user to configure and attach a network capable device such as a PC or
Laptop, connect it directly to one of the FSP network ports and access the FSP's ASMI interface. Note that these addresses are
NOT permanent, (static) but are just defaults which the FSP continues to use until the next DHCP request cycle is performed and
a response is received.


Scenario #2: Response to DHCP request IS received

In this case, the FSP simply accepts the IP address it was administered by the DHCP server however this new IP address now
replaces the default IP address the FSP would have assigned itself above in scenario #1. The new IP address now becomes the
new default IP address for that port however just like in scenario #1, the DHCP request process will take place again on the next
power or FSP reset cycle. If a DHCP response containing a different IP address is received, the FSP will accept it AND replace
the previous default IP address with the new one. A few examples of how this can happen are listed below:
• System (FSP) is attached to a different HMC (using the same FSP port that was previously configured)
• If the HMC's disk drive is reloaded or replaced, the address may change, particularly if multiple systems are attached
and are discovered in a different order.
• An additional DHCP server such as another HMC configured as a DHCP server appears on the network
As long as no configuration changes are made, the HMC should always assign the same IP address to a respective Ethernet
port. It does this by keeping track of the network mac (hardware) addresses assigned to each Ethernet port. These mac
addresses are unique for every Ethernet device.

Connecting a Managed System to an HMC

The information contained in this section assumes that DHCP as been chosen as the desired method of assigning IP addresses
and that the HMC itself has been configured as the DHCP server. Prior to discussing troubleshooting options, we will describe
the expected behavior.
The HMC should be configured as a DHCP server prior to connection of the system unit. For this example, we'll assume that the
HMC is set up properly, the Managed System has been attached and AC power has been applied. As discussed in the last
section, the DHCP address request and assignment process takes place within the first 60 seconds of the FSP's boot process.
The HMC should now automatically detect the Managed System.
On the HMC, Click Server and Partitions > Server management to view the status of your managed system. An object
representing the Managed System with an IP address listed next to it should be displayed within a few minutes. If the Managed
System has never been configured on this OR another HMC, the state of the system will be Pending
Authentication. This indicates the that the HMC's access password (to the FSP) has not yet been set.
When this password is set, it is written to the FSP and also stored in a file on the HMC. This password
establishes secure connection between the HMC and the FSP.
At this point, the HMC access password must now be set as well as the ASMI admin and general user's
passwords. This can be done by selecting the object with the left mouse buttons and then right clicking on
the object and choosing the "Update Managed System Password" option from the menu.


Actual passwords chosen are up to you. If not specified the common passwords are as below :

HMC Access: abc123
Admin user: admin
General user: general


If the state of the system is Failed Authentication, it indicates that the FSP already has a stored HMC
access password which is not known by the HMC. This condition can occur if the Managed System was previously connected to
a different HMC or it was previously connected to the current HMC and it's connection was then logically removed using the
Reset or Remove connection option. When the connection is removed the HMC deletes it’s copy of the password.
If the password is known, it can be input now. If the password is NOT known, a couple of options are available. These options
are covered in the Common Symptoms and Solutions section which appears later in this document.
Once the passwords have been entered, the system will go to a Power Off state and the IP address in
the icon will be replaced with the machine type and serial number of the system (or a customized setting
for systems that were previously installed)


What if no Managed System object appears?
The first step is to establish whether or not the FSP is requesting an address and if so, is the HMC
responding?
The easiest way to do this is to open a rshterm on the HMC and run the command "lshmc -n". The output
from the command will look something like the following:


Example 1:
hscroot@sqhmc:~> lshmc -n
hostname=sqhmc,domain=austin.ibm.com,"ipaddr=10.0.0.1,9.3.253.230","networkmask
=255.255.255.0,255.255.255.192",gateway=9.3.253.193,"nameserver=9.0.7.1,9.0.6.11
",domainsuffix=,ipaddrlpar=9.3.253.230,networkmasklpar=255.255.255.192,"clients=
10.0.0.254"

Note the last field in the above example: "clients=10.0.0.254" This field of information contains all IP
addresses that have been assigned by the HMC's DHCP server. If multiple systems are attached,
multiple addresses will appear in this field.

Example 2:
hscroot@sqhmc:~> lshmc -n
hostname=sqhmc,domain=austin.ibm.com,"ipaddr=10.0.0.1,9.3.253.230","networkmask
=255.255.255.0,255.255.255.192",gateway=9.3.253.193,"nameserver=9.0.7.1,9.0.6.11
",domainsuffix=,ipaddrlpar=9.3.253.230,networkmasklpar=255.255.255.192,"clients=
10.0.0.251,10.0.0.254,10.0.0.254"


In the example above you can see that there are 2 instances of the same IP address (10.0.0.254) Why?
When the HMC DHCP server assigns an IP address, it creates a DHCP lease. Each instance of an IP
address represents a DHCP lease. All unexpired leases will be displayed in by the lshmc -n command. If
the DHCP server receives additional DHCP requests from the same MAC address, it will create a new
lease for that MAC address despite the fact that an unexpired lease currently exists. This process will be
repeated every time the FSP requests an address. Once the 2 hour lease period expires on any inactive
leases, they will be automatically deleted and will not appear in the lshmc -n command output. Also note
that address 10.0.0.251 appears in the above example. In this case, our HMC is controlling 2 Managed
Systems.
Armed with this information, we can now establish whether or not an FSP and the HMC have established
basic communication.
For example: A Managed System (FSP) is cabled up to an HMC. An object for this managed system does
not appear. Now what? Run the lshmc -n command and check to see if any IP addresses appear in the
"clients" field.
Assuming that the answer to this question is yes, then how do we know if the IP address belongs to the
system we are currently trying to configure or, in the event that multiple systems are attached, how do you make out which IP address belongs to which system?

Make a note of how many IP addresses appear and also how many instances of each IP exists, then
force the FSP to request an address again using one of the methods described in Section 1 above. If the
FSP and the HMC are communicating properly, then another IP address or an additional instance of one
of the previously existing IP addresses should appear. Compare the example below to example 2 above
and note that address 10.0.0.254 now appears a third time as a result of an FSP being reset. If you reset
the FSP again, a 4th instance will appear and so on.


Example 3:
hscroot@sqhmc:~> lshmc -n
hostname=sqhmc,domain=austin.ibm.com,"ipaddr=10.0.0.1,9.3.253.230","networkmask
=255.255.255.0,255.255.255.192",gateway=9.3.253.193,"nameserver=9.0.7.1,9.0.6.11
",domainsuffix=,ipaddrlpar=9.3.253.230,networkmasklpar=255.255.255.192,"clients=
10.0.0.251,10.0.0.254,10.0.0.254,10.0.0.254"
On V4R3 and later HMC code levels, the command lssysconn –r all provides an additional method for
viewing which systems are connected. This command will show all systems and frames managed by the
HMC including those that are not DHCP clients.


Example 4:
hscroot@sqhmc:~> lssysconn -r all
resource_type=sys,type_model_serial_num=9117-570*109C4DD,sp=primary,ipaddr=10.0.0.251
,state=Connected
resource_type=sys,type_model_serial_num=9117-570*100079A,sp=primary,ipaddr=10.0.0.248
,state=Connected


The next example shows the same command run on an HMC with 9119 -595 attached. Note that 4 IP addresses appear, 1 for each of the FSP’s and 1 for each of the Bulk Power Controllers.

Example 5:
hscroot@sqhmc2:~> lssysconn -r all
resource_type=sys,type_model_serial_num=9119-595*02ACC3C,sp=secondary,ipaddr=192
.168.254.255,state=Connected
resource_type=sys,type_model_serial_num=9119-595*026ACC3C,sp=primary,ipaddr=192.1
68.255.252,state=Connected
resource_type=frame,type_model_serial_num=9458-100*9112345,side=a,ipaddr=192.168
.255.253,state=Connected
resource_type=frame,type_model_serial_num=9458-100*9112345,side=b,ipaddr=192.168
.254.254,state=Connected


The above exercise established that the FSP has requested an IP and the HMC has issued one. The next
step is to open a rshterm and perform a ping test to the IP address you obtained in the previous step.
Assuming that the ping test was successful, we have now established that a good communication path
exists between the FSP and the HMC so the problem is most likely NOT with the FSP. Continue with
"Manually Adding a Managed System" below.
What if "clients= " (blank)? This indicates that effective 2 way communications between the FSP and the
HMC have NOT occurred! There are numerous potential causes for this condition including but not limited
to:

• The HMC is not correctly configured as a DHCP server.(The DHCP server is not running or
another HMC problem exists)
• A networking problem exists in the path between the FSP and the HMC:
o Is the Ethernet cable connected to the correct port on the HMC?
o Is there a malfunctioning or misconfigured Switch or Hub in the network path?
o Are all the link lights illuminated?
• The FSP's Ethernet port has been configured with a static IP address To verify, connect to
ASMI and view the Network Configuration settings. (Unless the system has been previously
installed in another environment, this is unlikely)


Manually Adding a Managed System

Now that we know the address of the Managed System which we are trying to configure and verified that
we can ping it, we can use the Add Managed System option to manually add the system. For Systems
configured as DHCP clients we normally we would not have to perform this step however there are few
known HMC problems that will cause the object not to appear automatically. A list of known problems that
produce this behavior are listed in the "Known HMC Connectivity Issues" section.
There is one common cause for this behavior that is by design:

Example: A Managed System which has had it's IP address dynamically assigned by the HMC. It's connection is then removed
using the Reset or Remove connection option. When this occurs, the HMC stores the removed IP address in an "exclusion
file. This prevents the HMC from polling the removed IP address despite the presence of a valid DHCP lease. To resolve this
problem, the Managed system can simply be added manually. On V4R3 and later HMC levels, the command "mksysconn -o
auto" will remove the contents of the exclusion file which will allow the Managed System object to appear automatically.
In the event that the customer does not wish to utilize DHCP to assign IP addresses, the desired
addresses can be assigned manually via the ASMI menus. Once the IP addresses are known, the
follow below steps to add the managed system.

1) Select the "add managed system" option from the Server Management menu:
2) Input the address of the Managed System you are trying to add and click "next". The password is the
HMC’s access password for the FSP. It can be left blank as this time.
3) On the next panel, click "Finish"

----------------------------------------------------------------------------------------------------------------------------------------------------------------

Monday, July 5, 2010

AIX Basics (Software installation and management)



Software and patch management

Software management includes the tasks of installing OS, Installing other programs, Installing TLs, APARs etc.
In an installable CD an OS comes as a number of file sets. These filesets once installed will have a combined OSlevel (version of OS eg: 5.1, 5.2, 5.3 etc) and a Technology level (Patch level eg: 5100-04 means TL4)
Command to check the os and TL level is
#oslevel –r
5300-07 ( means 5.3 TL 7)
Fileset is the smallest part of any installable software program or OS. The standard type of filesets AIX compatible are called bff (Backup file format) filesets. (Apart of bff format rpm packages and normal bin/cpio/tar packages are also installed in AIX like any other UNIX/Linux OS)
Eg: bos.tcp.client.5.2.0.35.bff
The filesets included(installed) in our system can be listed using command
#lslpp –l
The basic packaging structure is as shown below.



Applied state and Committed state of an installed fileset




AIX has a number of software states. When you are installing software for the first time, thesoftware will automatically install to a committed state. This means there is only one levelof that software product installed on your system.When you are installing a fix- or a maintenance- level upgrade to your system, You will be actually installing a higher version of a filesetor set of filesets. Here you have the option of installing the software either in the committed state or the applied state.
The applied state allows you to maintain two levels of the software on your system. Whensoftware is installed in the applied state, the older version is saved on the disk and isdeactivated while the newer version is installed and becomes the active version. The applied state gives you the opportunity to test the newer software before committing to its use. If it works as expected, then you can commit the software which will remove the oldversion from the disk. If the newer version is causing a problem, you can reject it which will remove the newer version and recommit the old version. For example if bos.net.nfs.client version 5.3.0.10 is in committed state in your system and you are installing a higher version of bos.net.nfs.client version 5.3.0.50 on top of it as part TL 5 you can apply the fileset and will keep a copy of the 5.3.0.10 in /usr and will be shown as committed when you do lslpp.
Now if your TL 5 has to be rejected and you want to go back to previous level you can reject all of your newer filesets which are applied.
#smitty maintain_software
Also when you are going to update next level say TL 6 or 7 you will need to commit the now applied filesets of TL 5 and the TL 1 version of filesets will be no more shown in lslpp output. (Their files will be permanently removed from /usr).

Technology Levels and Service Packs

ML (Maintenance Level): It is no more called ML but TL now (Technology level)

Periodic patches and enhancements will be released by IBM for each and every fileset which is part of the OS or subsystem.

Once higher version for all/most of the filesets in an OS level is released they are packaged into a single set called TL. (Eg: for os and TL level 5200-04 will be given an TL level set for 5200-05 after releasing all higher version filesets). A new AIX Technology Level is released roughly every six months or lesser than that.


To see the TL level


#oslevel –r


Service Packs
An AIX Service Pack (SP) is a bundle of PTFs released between Technology Levels. The PTFs/fixes address highly pervasive, critical, or security-related issues. To see the service pack level run

#oslevel -s

6100-02-01-0847 (Means AIX 6.1 TL 02 Service pack 01)


APAR

APAR(Advanced Program Analysis Report). APAR’s are reports generated by IBM Software support for specific problems with specific version of filesets/softwares. For these reports there will be a set of higher version filesets invented and will be downloadable from IBM fixcentral website. The process of downloading and installing these filesets is called applying an APAR. The APARs are given a number starting with IY or IZ normally (eg:IY85674) and the set of filesets for the solution changes depending on our servers oslevel and TL level. The installed APARs in our system can be listed using command
#instfix –i


E-fix

E-fixes are temporary fixes comes as a binary package (Not standard fileset) to provide interim solution to a problem reported by an APAR.

To install an e-fix

#emgr –p packagefilename

To list e-fixes installed

#emgr –P

To remove

#emgr –r efixname


Procedure to Download and install APAR

Once you have the APAR number to be installed first thing to do is to check if the APAR is already installed in your system using instfix command
(eg: #instfix -i grep IY72368)
It is also possible that the APAR number is not shown because you have a much higher level of TL already installed than the TL in which this APAR was included. In that case you need to check in the IBM fixcentral site with which TL the APAR is included. Installing APAR First of all we need to confirm if the APAR is already in your system and the system's current levels. If the APAR number is IZ64635 Run the following commands

#oslevel –r

#instfix –i grep IZ64635

#lslpp –l more

#installp -s (To see any if you have any applied filesets already)

Check for any broken filesets using

#lppchk –v

This command should not give any ouput in ideal case.

List all the applied filesets using

#installp –s

Then commit all applied filesets using

#installp –c all

Confirm with.

#installp –s



You can download all patches/microcodes/firmwares from IBM fixcentral website.




And select systemp, AIX, Your OS version, fix search respectively as below.




Provide APAR number to search




Click on Obtain this Fix






The service pack under which APAR is avilable will be listed. Click on the link to download.




Select the SP and click on continue





Select download option as ftp to ftp the filesets to your local laptop or server itself from internet. You can also select to download through Download director which is a java based tool if ftp is not available.



The site will give an ftp server address and fileset locations and size.






Create a directory or temporary filesystem according to the size of all filesets under /tmp Eg: /tmp/filesets. Connect to the ftp site and download these filesets to /tmp/filesets. Go to /tmp/filesets. #cd /tmp/filesets

Then run
#inutoc .

This will create an index file inside the directory called .toc Run command
#smitty installp

And select menu Update Installed Software to Latest Level (Update All). Select fileset installation files location as /tmp/filesets Select COMMIT as NO Select PREVIEW as YES Then press enter This will do an installation verification and will let you know if the installation would go smooth or not. We can search for “/error” in the smitty screen to find if the installation would create any clash or dependency issue. If there is no error we can go back to previous smitty screen using F3 and select PREVIEW as NO. This will do the actual installation.






OK message will appear after successful installation. Check with the following commands if the APAR is installed properly.


#instfix –i grep IZ64635

#installp –s

#lppchk –v


Do a BLV update to boot disks using bosboot command just in case.

#bosboot -ad /dev/hdisk0

#bosboot -ad /dev/hdisk1

If everything is fine reboot server once using
#shutdown –Fr

To download and install TLs /Service Packs
Downloading and Installing TL/SPs are same as downloading and installing APARs. Go to IBM Go to fixcentral website (http://www-933.ibm.com/support/fixcentral/)

And select systemp, AIX, Your OS version, fix pack respectively (instead of fix search) as below.



The next screen will contain all the TLs /SPs to be downloaded select the desired one and click continue to download.

To remove or reject file sets
#smitty maintain_software
From the menu select filesets to be removed and press enter.

RPM Packages
RPM packages are type of software packages used in Linux.

Originally standing for "Red Hat Package Manager", RPM now stands for "RPM Package Manager", a recursive acronym.
The installable files for programs comes as a single package file with .rpm extension
For (eg: Gzipver7.1.0.rpm is the installable file for gzip utility version 7.1)
To install an rpm package
#rpm –i installablepackagename
To list all rpm packages installed
#rpm –qa
To remove installed package
#rpm –e installedpackagename


-------------------------------------------------------------------------------------------------

AIX Basics (ODM)

ODM

Object Data Manager (ODM) is a set of database files (classes) used to store system configuration such as hardware configuration, software’s installed, network configurations etc.

!! User configuration is not stored in ODM !!

ODM classes

Depending on the configuration data stored ODM has separate database files called ODM classes
For eg: the list of connected devices are stored in ODM class CuDv. And the attributes of these devices are stored in CuAt.
These are stored under directory /etc/objrepos, /usr/lib/objrepos, /usr/share/lib/objrepos

/etc/objrepos : Conatins all Customized (Connected and configured ) device configuration, Software vital product data for / part and other networking/utilities details.
(Do an 'ls' on this directory to see the names of the classes)
CuDv
CuAt
CuDep

History
Inventory
Lpp etc..

/usr/lib/objrepos : Contains all predefined (Supported) Device details and attributes as well as software vital product data for /usr part.

PdDv
PdAt

History
Inventory
Lpp etc

/usr/share/lib/objrepos : Contains software vital product data for /usr part.(Information on filesets installed like their names, history, state, files extracted from each fileset etc.)

History
Inventory
Lpp etc


To display the format (structure) of an ODM class
#odmdisplay

To display the contents of ODM class
#odmget
Eg: odmget CuDv more

To display specific entry

#odmget –q “name=rmt0” CuDv

To delete a specific entry

#odmdelete –o CuDv -q “name=rmt0”

To add or change an entry we need to type all lines in a text file
Eg:

#vi file1
CuAt:
Uniquetype=”/tape/8mm/scsi”
Attribute=”block_size”
Nis_index=”6”

Then for changing
#odmchange -o CuAt –q “Uniquetype=/tape/8mm/scsi” file1

For adding the stanza (entry) newly just
#odmadd file1

AIX Basics (User Management)

User login, attributes, management

No user/security configuration is stored in ODM.
A user is an entity which logs into a unix system with a passwd inorder to use its resources.
Root user
User root has userid ‘0’ and has all the privileges to administer a unix system. There are some default users created with the os for other programs/subsystems. Eg: ftp, mail, bin, etc.
Apart of password a user has the following particularities
Primary group – The default group of a user (A group is a set of users). Eg: for user abc primary group is staff. The default primary group of AIX is staff.
Groups – Other groups which the user is a member of eg: user abc other groups can be apps, admin
User id – Is the number given to identify a user by the system. All permissions are based on userid.
Group id – Similar way for a group
Home directory – Is the default directory user will be placed under after login. By default they can only write in this directory.
Shell – Is the type of shell which a user will be given after login by default it is korn shell (ksh)
Geckoes (Description) – User description given for the admin to identify

This information is stored in /etc/passwd file separated by ‘:’ s.
The seven fields of /etc/passwd are
Username:encryptedpasswd:userid:groupid:userdescription:homedir:shell

In AIX the encrypted passwd is not stored in /etc/passwd. Instead it’s stored in
/etc/security/passwd
Apart of these properties in AIX there are some more attributes for a user like admin, locked/unlocked, login counts, etc.
These properties are stored in /etc/security/user
The similar configuration files for groups are
/etc/group and /etc/security/group.

User profile and root profile

Profiles are configuration files which contains the start up script to set environment for a user like environment variables, PS1, umask etc. Profile is executed at the time of login.
The root profile file is /.profile
The user profile is $HOME/.profile. ($HOME means home dir)
Apart of these files there are few more profiles executed by default are
/etc/profile
/etc/security/environ
/etc/security/environments
/etc/secutiy/limits (Contains usage limits like for memory, disk, cpu for each user)

#umask is the command to see and set the umask value for a user
#set command lists all environment variables set with their value


Managing Users

#mkuser username creates a new user
#lsuser username Command lists all attributes of a user
#chuser “attribute=vale” username Changes the attribute of a user
Eg: #chuser “Home=/home/abc” username Changes home dir of the user
#passwd username To change passwd of a user.
#rmuser username removes a user (Home dir needs to be removed)
#mkgroup groupname to add group
#rmgroup groupname to remove group

Normal user admin activities are creating, deleting a user. Changing password for user. Changing attributes like “account_locked=no”, “unsuccessful_login_count=0” “groups=staff,abc,xyz”
Etc.

Login sequence of a user (What happens while logging in)

Sbin/getty program fetches the login prompt.
When a user types username and passwd it checks with configuration in /etc/login.cfg that how to authenticate the user. If authentication method is given SYSTEM (Default) then it verifies the username and passwd with /etc/passwd and /etc/security/passwd. If not correct it puts an entry
In log file /etc/security/failedlogin and denies login.
If correct the login becomes successful. The system runs environment files /etc/security/environment, etc/security/environ, etc/security/limits, $HOME/.profile and brings the user to home dir with a shell.

Commands related to login

#who command lists all users currently logged in with their terminal, time details
#last command lists details of all previous logins with login time, logout time, terminal details etc.


Log files related to login

/etc/security/failedlogin failed login details
/var/adm/wtmp contains previous and current login details (last command takes data from this)
(Sometimes this file becomes huge and need to copy to another location. Who command is used to read the contents of the copied file)

AIX Basics (OS Installation, Boot up process and rootvg Filesystem structure)

Installation of AIX

Your AIX cd set normally would contain the following parts
AIX OS - The base OS and susbsystems included in OS
Expansion pack - Additional UtilitiesAIX Documentation - The documentation
AIX Toolbox for Linux - The linux utilities which can be installed on AIX as rpm (eg: unzip)
Base os cd/cds contains operating system filesets and filesets for programs included in os.
Eg: bos.nfs.client, bos.mp
Extra utilities cd contains utilities like gzip, webserver etc which may or may not be used.

Installation steps

1) Boot the server and press f1 key (or 1 key) after first beep and before second beep to bring the server
in SMS menu. Go to bootlist menu and select cdrom as first boot device.

2) Insert the cdrom and boot the server

3) Installation program comes up with a menu for installation. There are three types of installations

a) Preservation installation: Keeps files under /home directory if there is already an os exists
b) Migration installation: Keeps all files and programs as it is and this is used only to upgrade a lower version to higher (eg: AIX 5.1 to 5.3)
c) Complete overwrite: Overwrites everything.

The detailed steps are as below.

Step 1. Prepare your system
There must be adequate disk space and memory available. AIX 5L Version 5.2 and AIX 5L Version 5.3 require 128MB of memory and 2.2GB of physical disk space.
Make sure your hardware installation is complete, including all external devices.
If your system needs to communicate with other systems and access their resources, make sure you have the information in the following worksheet before proceeding with the installation:
Network Attribute
Value
Network interface
For example: en0, et0
Host name

IP address

Network mask

Nameserver

Domain name

Gateway

Step 2. Boot from the AIX product CD
Insert the AIX Volume 1 CD into the CD-ROM device.
Make sure all external devices attached to the system, such as CD-ROM drives, tape drives, DVD drives, and terminals, are turned on. Only the CD-ROM drive from which you will install AIX should contain the installation media.
Power on the system.
When the system beeps twice, press F5 on the keyboard or 5 on an ASCII terminal (To boot from CD). If you have a graphics display, you will see the keyboard icon on the screen when the beeps occur. If you have an ASCII terminal, you will see the word keyboard when the beeps occur.
Select the system console and press Enter.

Select the English language for the BOS installation menus by typing a 1 in the Choice field. Press Enter to open the Welcome to Base Operating System Installation and Maintenance screen.
Type 2 to select 2 Change/Show Installation Settings and Install in the Choice field and press Enter.
Welcome to Base Operating System Installation and Maintenance
Type the number of your choice and press Enter.
Choice is indicated by >>>.
1 Start Install Now with Default Settings
2 Change/Show Installation Settings and Install
3 Start Maintenance Mode for System Recovery
88 Help ?
99 Previous Menu>>> Choice [1]: 2

Step 3. Set and verify BOS installation settings
In the Installation and Settings screen, verify that the installation settings are correct by checking the method of installation (new and complete overwrite), the disk or disks you want to install, the primary language environment settings, and the advanced options.If the default choices are correct, type 0 and press Enter to begin the BOS installation. The system automatically reboots after installation is complete. Go to Step 4. Configure the system after installation.Otherwise, go to sub-step 2.
To change the System Settings, which includes the method of installation and disk where you want to install, type 1 in the Choice field and press Enter.
Installation and Settings Either type 0 and press Enter to install with current settings, or type the number of the setting you want to change and press Enter.
1 System Settings: Method of Installation..................New and Complete Overwrite
Disk Where You Want to Install..hdisk0 >>> Choice [0]: 1
Type 1 for New and Complete Overwrite in the Choice field and press Enter. The Change Disk(s) Where You Want to Install screen now displays.
Change Disk(s) Where You Want to Install Type one or more numbers for the disk(s) to be used for installation and press Enter. To cancel a choice, type the corresponding number and Press Enter. At least one bootable disk must be selected. The current choice is indicated by >>>.
Name Location Code Size(MB) VG Status Bootable
1 hdisk0 04-B0-00-2,0 4296 None Yes
2 hdisk1 04-B0-00-5,0 4296 None Yes
3 hdisk2 04-B0-00-6,0 12288 None Yes
>>> 0 Continue with choices indicated above
66 Disks not known to Base Operating System Installation
77 Display More Disk Information 88 Help ?
99 Previous Menu >>> Choice [0]:

In the Change Disk(s) Where You Want to Install screen:
Select hdisk0 by typing a 1 in the Choice field and press Enter. The disk will now be selected as indicated by >>>. To unselect the destination disk, type the number again and press Enter.
To finish selecting disks, type a 0 in the Choice field and press Enter. The Installation and Settings screen now displays with the selected disks listed under System Settings.
Change the Primary Language Environment Settings to English (United States). Use the following steps to change the Cultural Convention, Language, and Keyboard to English.
Type 2 in the Choice field on the Installation and Settings screen to select the Primary Language Environment Settings option.
Type the number corresponding to English (United States) as the Cultural Convention in the Choice field and press Enter.
Select the appropriate keyboard and language options.
You do not need to select the More Options selection, because you are using the default options in this scenario. For more information about the installation options available in AIX 5.2 and greater, see Installation Options in the AIX 5L Version 5.3 Installation Guide and Reference.
Verify that the selections are correct in the Overwrite Installation Summary screen, as follows:
Overwrite Installation Summary
Disks: hdisk0
Cultural Convention: en_US
Language: en_US
Keyboard: en_US64 Bit
Kernel Enabled: No
JFS2 File Systems Created: No
Desktop: CDE
Enable System Backups to install any system: Yes
Optional Software being installed:
>>> 1 Continue with Install
88 Help ? 99 Previous Menu >>> Choice [1]:
Press Enter to begin the BOS installation. The system automatically reboots after installation is complete.

Step 4. Configure the system after installation
After a new and complete overwrite installation, the Configuration Assistant opens on systems with a graphics display. On systems with an ASCII display, the Installation Assistant opens.
Select the Accept Licenses option to accept the electronic licenses for the operating system.
Set the date and time, set the password for the administrator (root user), and configure the network communications (TCP/IP). Use any other options at this time. You can return to the Configuration Assistant or the Installation Assistant by typing configassist or smitty assist at the command line.
Select Exit the Configuration Assistant and select Next. Or, press F10 or ESC+0 to exit the Installation Assistant.
If you are in the Configuration Assistant, select Finish now. Do not start the Configuration Assistant when restarting AIX and select Finish. At this point, the BOS Installation is complete, and the initial configuration of the system is complete.

Structure after installation

After installation there will be a volume group called rootvg and the following lvs on the installation disk selected.
Hd5 - Boot logical volume ( bootimage, Contains copy of kernel and a reduced odm)
Hd1 - /home
Hd2 - /usr
Hd3 - /tmp
Hd4 - /
Hd6 - Paging
Hd7 - Dump
Hd8 - jfs loglv
Hd9var - /var
Hd10opt - /opt (optional)

Hd1(home):It contains the home directories of the users.

Hd2(/usr):Its contains basic os executables and binaries.

Hd3(/tmp):It’s the temporary space used for the system/programs to temporarily keep and delete the files while processing data.

Hd4(/):It’s the root directory. It is the home directory of the root user.
/etc-it contains all configuration files.
/sbin-commands and executables super user binaries.
/bin- contains normal commands and executables.
/dev-all device files.
Hd5(boot logical volume): It keeps kernel image and reduced ODM of the aix operating system. It’s not a file system
Hd6(paging):It’s the logical volume used for the memory paging(swaping).this memory is also called virtual memory.
Hd7(dump):It’s the logical volume used for the system dump.
Hd8(log):Is used for keeping jfslog of all file systems in rootvg.
Hd9var: var file system used for varying files likes log files, spool files.
Hd10opt: It is used for to store extra software or programs.

Boot up, init process and runlevels

Boot up process

When power on happens the firmware program stored in an System board Chip (called Read only Storage [ROS] ) gets loaded into the memory

The firmware Conducts CPU and memory power on self tests, detects all controllers and hardware devices connected to the system and conducts POST for them also

The firmware reads the console path information from an NVRAM called Stable storage (NVRAM stores firmware configuration) and initializes console.

Firmware checks for first boot device in Stable storage and accesses the first 512 bytes of the first boot device (normally disk)

The first 512 bytes contains the boot strap information which locates the hd5 (blv) and kernel from BLV gets loaded to the memory

Kernel creates temporary RAM filesystem containing /, /usr, etc, dev, mnt etc.
These filesystems are virtually mounted in RAM and whatever it contains are what the kernel had read from BLV and nothing else.
Kernel starts init process from blv. Init starts rc.boot script. The rc.boot script runs in three phases to complete the boot process

Rc.boot phase 1

This phase configures base devices from reduced odm in BLV and runs cfgmgr to detect new devices. The device files for new devices will be placed under /dev in RAM.

Rc.boot phase 2

This phase activates rootvg with a special command ipl_varyon and runs fsck on /usr, / and /var filesystems.
Mounts them from disk to replace RAM filesystems.

Activates paging space hd6. Copies all device files from memory to disk. Removes RAMfs.

RC.boot phase3

Init process is replaced with the init script in the disk. This init initializes rc.boot phase3
Mounts /tmp filesystem
Configures all devices other than phase1
Console is configured by the OS and LED display is turned off
( LED display is provided with all servers which displays specific numbers indicating each stage of booting. Looking at this number and manual we can trouble shoot any boot time failures)
After rc.boot script init starts processes in the inittab one by one.
There is an entry in inittab called initdefault to indicate default run level. (Normally 2)



Init

Init (/sbin/init) is responsible for taking the system to a specific run level by starting processes listed in inittab. Init process has process id 1. init is parentprocess for all processes.

The configuration file is /etc/ inittab.

Syntax of inittab
Inittab contains four columns program id, runlevels, action, command to run separated by “:”

Eg - 7ab:1,2,3,4:respawn:/usr/bin/cron

Actions

Boot, bootwait, wait, respawn

Boot : The program is started in the boot time
Bootwait: The program is started in the boot time and to be waited to end before starting next.
Wait: To be waited to end before starting next.
Respawn: This process should never be killed. If we kill the process it will be automatically restarted by init. (eg: cron, getty).

Runlevels

AIX has run levels from 0 to 9, also a and b
0 and 1 are reserved.
2 is the default normal multiuser mode
Run levels from 3 to 9 are defined by administrator.