Probe: ORNL Probe Configuration Details

Clariion FC disk configuration:
===============================

Hardware:

     - Enclosure 0 (bottom)
       - Disks 0-9, 9GB each
     - Enclosure 1 (middle)
       - Disks 0-9, 18GB each
     - Enclosure 2 (top)
       - Disks 0-9, 9GB each
       - NOTE: Disk 0 currently replaced with an 18GB drive!
     - Dual Storage Processors, SPA and SPB

RAID Groups and LUNs (these can be changed whenever we like):

     - RAID Group 0
       - RAID 5
       - Currently bound to LUN 0 on SPA
       - Enclosure 0, disks 0-4
     - RAID Group 1
       - RAID 5
       - Currently bound to LUN 1 on SPB
       - Enclosure 0, disks 5-9
     - RAID Group 2
       - RAID 1/0
       - Currently bound to LUN 2 on SPA
       - Enclosure 1, disks 0-3
     - RAID Group 3
       - RAID 1/0
       - Currently bound to LUN 3 on SPB
       - Enclosure 1, disks 6-9
     - RAID Group 4
       - RAID 3
       - Currently bound to LUN 4 on SPA
       - Enclosure 2, disks 0-4
     - RAID Group 5
       - RAID 3
       - Currently bound to LUN 5 on SPB
       - Enclosure 2, disks 5-9
     - RAID Group 6
       - Hot spare
       - Currently bound to LUN 6
       - Enclosure 1, disk 4
     - RAID Group 7
       - Hot spare
       - Currently bound to LUN 7
       - Enclosure 1, disk 5

SCSI IDs:

     - The FC RAID sets seem to be referenced as pseudo-SCSI devices from
       the Mover hosts.  Anything accessed through SPA has a SCSI ID of 0,
       and anything accessed through SPB has a SCSI ID of 1.  Thus, the
       Storage Processors are sometimes referred to as SP0 and SP1.
     - The above is partly guesswork, so don't take it for the Ultimate
       Truth.

Maverick FC Interfaces:

     - Dual FC adapters:
       - lpfc0     30-60     Emulex LP8000 Fibre Channel Adapter
       - lpfc1     30-70     Emulex LP8000 Fibre Channel Adapter

     - Subsidiary adapter devices (64 of them; I'm not sure what these
	represent, but they are undoubtedly necessary):
       - lpfc0.[01-31]  30-60-[01-31]  Emulex LP8000 Fibre Channel Adapter
       - lpfc1.[01-31]  30-70-[01-31]  Emulex LP8000 Fibre Channel Adapter

     - Storage processors (for some reason, these only show up as connected
       to the first FC adapter):
       - sp0               30-60-00-0        DiskArray Storage Processor
       - sp1               30-60-00-1        DiskArray Storage Processor

     - Disk devices:
       %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
       NOTE:  cfgmgr generates three hdisk devices for each RAID set it can
       see.  But you can (of course) create AIX storage resources on only one
       of the three.

       To figure out which hdisk refers to what, you need to look at the
       "address" of the device (third column in the table below).  The most
       significant part of the address is the last 2 digits separated by a
       comma.  These are the Storage Processor (or the SCSI ID) and the LUN
       of the RAID set.  For example:

	  hdisk4  30-60-00-0,2 DiskArray RAID 1/0 Disk Group
	  hdisk31 30-60-00-0,2 Fibre Channel SCSI FCP Disk Drive
	  hdisk11 30-70-00-0,2 Fibre Channel SCSI FCP Disk Drive

       These are all the same RAID set; Storage Processor 0 (or A), LUN 2.
       I think we're supposed to ignore hdisk4 and all similar disks which
       are listed as "DiskArray RAID" devices (and I don't know why some
       of them show up as "Defined" instead of "Available").  hdisk31 and
       hdisk11 both point to the same LUN, but through different FC
       interfaces.

       So you can create a volume group to use hdisk11 (for example), but
       you can't try to use either of the other two aliases.
       %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
       - hdisk1  Available 30-60-00-0,0 DiskArray RAID 5 Disk Group
       - hdisk3  Available 30-60-00-1,1 DiskArray RAID 5 Disk Group
       - hdisk4  Defined   30-60-00-0,2 DiskArray RAID 1/0 Disk Group
       - hdisk5  Defined   30-60-00-1,3 DiskArray RAID 1/0 Disk Group
       - hdisk6  Defined   30-60-00-0,4 DiskArray RAID 3 Disk Group
       - hdisk7  Defined   30-60-00-1,5 DiskArray RAID 3 Disk Group

       - hdisk21 Available 30-60-00-0,0 Fibre Channel SCSI FCP Disk Drive
       - hdisk23 Available 30-60-00-1,1 Fibre Channel SCSI FCP Disk Drive
       - hdisk31 Available 30-60-00-0,2 Fibre Channel SCSI FCP Disk Drive
       - hdisk24 Available 30-60-00-1,3 Fibre Channel SCSI FCP Disk Drive
       - hdisk22 Available 30-60-00-0,4 Fibre Channel SCSI FCP Disk Drive
       - hdisk25 Available 30-60-00-1,5 Fibre Channel SCSI FCP Disk Drive

       - hdisk26 Available 30-70-00-0,0 Fibre Channel SCSI FCP Disk Drive
       - hdisk28 Available 30-70-00-1,1 Fibre Channel SCSI FCP Disk Drive
       - hdisk11 Available 30-70-00-0,2 Fibre Channel SCSI FCP Disk Drive
       - hdisk29 Available 30-70-00-1,3 Fibre Channel SCSI FCP Disk Drive
       - hdisk27 Available 30-70-00-0,4 Fibre Channel SCSI FCP Disk Drive
       - hdisk30 Available 30-70-00-1,5 Fibre Channel SCSI FCP Disk Drive

     - Usable volumes and/or filesystems:
       - Currently, one VG named "fctest" is created on hdisk11 (LUN 2, a
	RAID 0/1 set).  The VG has one filesystem, /fctest1, which occupies
	the entire RAID set (about 32GB).

Dopey FC Interfaces:

     - Dual FC adapters (known to Tru64 as "busses"):
       - cfc0
       - cfc1

     - FC SCSI controllers:
       - scsi4 (on cfc0)
       - scsi5 (on cfc1)

     - Disk devices in /dev:
       %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
       NOTE:  All these devices (over 2000 of them!) are apparently created
       automatically by the device installation/configuration, but only a
       handful are actually useful.

       Look at "man SCSI" to get a good description of the SCSI disk naming
       scheme.  As an example, device /dev/rrzd34c has these parts:

	  "rrz" = raw SCSI disk device
	  "d"   = LUN 3 (where a = 0)
	  "34"  = SCSI bus 4, bus address 2 (4 x 8 + 2)
	  "c"   = disk partition c

       Currently, all LUNs accessed through cfc0 should have address 32 (for
       SPA) or 33 (for SPB), and all LUNs accessed through cfc1 should have
       address 40 (for SPA) or 41 (for SPB).  Thus /dev/rrze32c and
       /dev/rrze40c both refer to the same RAID set (LUN 4).

       I don't know what the devices like /dev/rrz32c (without the LUN
       letter) refer to, if anything.
       %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
       - /dev/rrz32[a-h] ... /dev/rrz47[a-h]
       - /dev/rrza32[a-h] ... /dev/rrza47[a-h]
       - /dev/rrzb32[a-h] ... /dev/rrza47[a-h]
       - /dev/rrzc32[a-h] ... /dev/rrza47[a-h]
       - /dev/rrzd32[a-h] ... /dev/rrza47[a-h]
       - /dev/rrze32[a-h] ... /dev/rrza47[a-h]
       - /dev/rrzf32[a-h] ... /dev/rrza47[a-h]
       - /dev/rrzg32[a-h] ... /dev/rrza47[a-h]
       - /dev/rrzh32[a-h] ... /dev/rrza47[a-h]
       - /dev/rz32[a-h] ... /dev/rz47[a-h]
       - /dev/rza32[a-h] ... /dev/rza47[a-h]
       - /dev/rzb32[a-h] ... /dev/rza47[a-h]
       - /dev/rzc32[a-h] ... /dev/rza47[a-h]
       - /dev/rzd32[a-h] ... /dev/rza47[a-h]
       - /dev/rze32[a-h] ... /dev/rza47[a-h]
       - /dev/rzf32[a-h] ... /dev/rza47[a-h]
       - /dev/rzg32[a-h] ... /dev/rza47[a-h]
       - /dev/rzh32[a-h] ... /dev/rza47[a-h]

     - Usable volumes and/or filesystems:
       - Currently, device "/dev/rze32c", is mounted on /clariion_lun4
	(LUN 4, a RAID 3 set).  The size is about 32GB.

Sleepy FC Interfaces:

     - Dual FC adapters (the descriptions are my guesses):
       %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
       NOTE:  Solaris seems to have all of their "real" device nodes stored
       in subdirectories of /devices, and symlinks to those device nodes in
       the usual /dev tree.
       %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
       - /devices/pci@1f,2000/fibre-channel@1/clsp@0,0:0
	- SPA accessed through FC adapter 0
	- symlink: /dev/clsp0
       - /devices/pci@1f,2000/fibre-channel@1/clsp@1,0:0
	- SPB accessed through FC adapter 0
	- symlink: /dev/clsp1
       - /devices/pci@1f,4000/fibre-channel@2/clsp@0,0:0
	- SPA accessed through FC adapter 1
	- symlink: /dev/clsp2
       - /devices/pci@1f,4000/fibre-channel@2/clsp@1,0:0
	- SPB accessed through FC adapter 1
	- symlink: /dev/clsp3

     - Disk devices:
       %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
       NOTE:  Each disk device has a "real" name, and two symlink aliases.
       For everyday reference, Solaris seems to prefer the c?t?d?s? format.
       For the FC disks, this breaks down to:

	  c = SCSI controller ID (1 or 2 in sleepy)
	  t = SCSI ID (0 for SPA, 1 for SPB)
	  d = LUN
	  s = partition (0-7 instead of a-h)

       So /dev/dsk/c1t0d0s2, for example, would refer to partition c of
       LUN 0 on SPA.  As usual, most of these devices don't refer to any
       real hardware, so you have to know which units really mean something.
       Also as usual, you can refer to the same RAID array by different
       names, so you have to be careful.
       %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
       - /devices/pci@1f,2000/fibre-channel@1/sd@0,0:[a-h]
	- symlinks: /dev/dsk/c1t0d0s[0-7], /dev/sd81[a-h]
       - /devices/pci@1f,2000/fibre-channel@1/sd@0,0:[a-h],raw
	- symlinks: /dev/rdsk/c1t0d0s[0-7], /dev/rsd81[a-h]
       - /devices/pci@1f,2000/fibre-channel@1/sd@0,2:[a-h]
	- symlinks: /dev/dsk/c1t0d2s[0-7], /dev/sd119[a-h]
       - /devices/pci@1f,2000/fibre-channel@1/sd@0,2:[a-h],raw
	- symlinks: /dev/rdsk/c1t0d2s[0-7], /dev/rsd119[a-h]
       - /devices/pci@1f,2000/fibre-channel@1/sd@0,4:[a-h]
	- symlinks: /dev/dsk/c1t0d4s[0-7], /dev/sd121[a-h]
       - /devices/pci@1f,2000/fibre-channel@1/sd@0,4:[a-h],raw
	- symlinks: /dev/rdsk/c1t0d4s[0-7], /dev/rsd121[a-h]
       - /devices/pci@1f,2000/fibre-channel@1/sd@1,1:[a-h]
	- symlinks: /dev/dsk/c1t1d1s[0-7], /dev/sd126[a-h]
       - /devices/pci@1f,2000/fibre-channel@1/sd@1,1:[a-h],raw
	- symlinks: /dev/rdsk/c1t1d1s[0-7], /dev/rsd126[a-h]
       - /devices/pci@1f,2000/fibre-channel@1/sd@1,3:[a-h]
	- symlinks: /dev/dsk/c1t1d3s[0-7], /dev/sd128[a-h]
       - /devices/pci@1f,2000/fibre-channel@1/sd@1,3:[a-h],raw
	- symlinks: /dev/rdsk/c1t1d3s[0-7], /dev/rsd128[a-h]
       - /devices/pci@1f,2000/fibre-channel@1/sd@1,5:[a-h]
	- symlinks: /dev/dsk/c1t1d5s[0-7], /dev/sd130[a-h]
       - /devices/pci@1f,2000/fibre-channel@1/sd@1,5:[a-h],raw
	- symlinks: /dev/rdsk/c1t1d5s[0-7], /dev/rsd130[a-h]

       - /devices/pci@1f,4000/fibre-channel@2/sd@0,0:[a-h]
	- symlinks: /dev/dsk/c2t0d0s[0-7], /dev/sd60[a-h]
       - /devices/pci@1f,4000/fibre-channel@2/sd@0,0:[a-h],raw
	- symlinks: /dev/rdsk/c2t0d0s[0-7], /dev/rsd60[a-h]
       - /devices/pci@1f,4000/fibre-channel@2/sd@0,2:[a-h]
	- symlinks: /dev/dsk/c2t0d2s[0-7], /dev/sd103[a-h]
       - /devices/pci@1f,4000/fibre-channel@2/sd@0,2:[a-h],raw
	- symlinks: /dev/rdsk/c2t0d2s[0-7], /dev/rsd103[a-h]
       - /devices/pci@1f,4000/fibre-channel@2/sd@0,4:[a-h]
	- symlinks: /dev/dsk/c2t0d4s[0-7], /dev/sd105[a-h]
       - /devices/pci@1f,4000/fibre-channel@2/sd@0,4:[a-h],raw
	- symlinks: /dev/rdsk/c2t0d4s[0-7], /dev/rsd105[a-h]
       - /devices/pci@1f,4000/fibre-channel@2/sd@1,1:[a-h]
	- symlinks: /dev/dsk/c2t1d1s[0-7], /dev/sd110[a-h]
       - /devices/pci@1f,4000/fibre-channel@2/sd@1,1:[a-h],raw
	- symlinks: /dev/rdsk/c2t1d1s[0-7], /dev/rsd110[a-h]
       - /devices/pci@1f,4000/fibre-channel@2/sd@1,3:[a-h]
	- symlinks: /dev/dsk/c2t1d3s[0-7], /dev/sd112[a-h]
       - /devices/pci@1f,4000/fibre-channel@2/sd@1,3:[a-h],raw
	- symlinks: /dev/rdsk/c2t1d3s[0-7], /dev/rsd112[a-h]
       - /devices/pci@1f,4000/fibre-channel@2/sd@1,5:[a-h]
	- symlinks: /dev/dsk/c2t1d5s[0-7], /dev/sd114[a-h]
       - /devices/pci@1f,4000/fibre-channel@2/sd@1,5:[a-h],raw
	- symlinks: /dev/rdsk/c2t1d5s[0-7], /dev/rsd114[a-h]

     - Usable volumes and/or filesystems:
       - Currently, device "/dev/dsk/c1t0d0s2", is mounted on /ClariionLun0
	(LUN 0, a RAID 5 set).  The size is about 32GB.

Grumpy FC Interfaces:

     - Single FC adapter:
       - Can't find a "device name" for the adapter on this machine.
       - LUNs are referenced through SCSI controller 3.

     - Disk devices:
       %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
       NOTE:  The "scsiha -p" command probes the SCSI bus for devices and
       creates entries under /dev and /hw for whatever it finds.  The "hinv"
       command shows the current "hardware inventory".  For the FC hardware,
       the following is displayed:

       Integral SCSI controller 3: Version Fibre Channel QL2200
         Fabric Disk: node 20000060163b5e8a port 0 lun 1 on SCSI controller 3
         Fabric Disk: node 20000060163b5e8a port 0 lun 3 on SCSI controller 3
         Fabric Disk: node 20000060163b5e8a port 0 lun 5 on SCSI controller 3
         Fabric Disk: node 20000060163bc4a8 port 0 lun 0 on SCSI controller 3
         Fabric Disk: node 20000060163bc4a8 port 0 lun 2 on SCSI controller 3
         Fabric Disk: node 20000060163bc4a8 port 0 lun 4 on SCSI controller 3

       The long hex numbers are the FC fabric IDs for SPA and SPB.  I'm not
       sure what the "port 0" refers to.

       Under /dev/rdsk, there is a "volume header" (vh) and a "volume" (vol)
       device created for each LUN found.  For example:

	  /dev/rdsk/20000060163b5e8a/lun5vh/c3p0

       is the "volume header" device for SPB, LUN 5.  The "c3p0" is part of
       all the FC LUN names and simply refers to the SCSI controller 3, port
       0 parts of the address.

       It appears that only when you actually create a filesystem on a LUN,
       you get two more devices; for example:

	  /dev/rdsk/20000060163b5e8a/lun5s0/c3p0
	  /dev/rdsk/20000060163b5e8a/lun5s1/c3p0

       The "lun5s0" and "lun5s1" refer to LUN 5; I believe the "s0" and
       "s1" are "partition numbers".  Dominick used "xfs" to create a
       filesystem, and it apparently created a large xfs partition (s0)
       and a much smaller raw partition (s1) for some reason.

       There are similar filesystem devices created under /dev/dsk.  There
       is a parallel set of nodes created under /hw/disk and /hw/rdisk; not
       sure how IRIX uses those.
       %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
       - /dev/rdsk/20000060163b5e8a/lun1vh/c3p0
       - /dev/rdsk/20000060163b5e8a/lun1vol/c3p0
       - /dev/rdsk/20000060163b5e8a/lun3vh/c3p0
       - /dev/rdsk/20000060163b5e8a/lun3vol/c3p0
       - /dev/rdsk/20000060163b5e8a/lun5s0/c3p0
       - /dev/rdsk/20000060163b5e8a/lun5s1/c3p0
       - /dev/rdsk/20000060163b5e8a/lun5vh/c3p0
       - /dev/rdsk/20000060163b5e8a/lun5vol/c3p0

       - /dev/rdsk/20000060163bc4a8/lun0vh/c3p0
       - /dev/rdsk/20000060163bc4a8/lun0vol/c3p0
       - /dev/rdsk/20000060163bc4a8/lun2vh/c3p0
       - /dev/rdsk/20000060163bc4a8/lun2vol/c3p0
       - /dev/rdsk/20000060163bc4a8/lun4vh/c3p0
       - /dev/rdsk/20000060163bc4a8/lun4vol/c3p0

       - /dev/dsk/20000060163b5e8a/lun5s0/c3p0
       - /dev/dsk/20000060163b5e8a/lun5s1/c3p0

     - Usable volumes and/or filesystems:
       - Currently, device "/dev/dsk/20000060163b5e8a/lun5s0/c3p0", is mounted
	on /clariion_lun5 (LUN 5, a RAID 3 set).  The size is about 32GB.