So, the bright side of Oracle hating the world (as evidenced through their arcane licensing structures) is the chance to get to do some creative technological circus acts and learn a lot in the process.

Here’s my original configuration:
Oracle 10g installed on a bare metal Dell PowerEdge 2850. Single Socket, Single Core, Hyperthreading turned off.

Why? Licensing. Even with Oracle’s ‘generous’ educational discounts, we cannot afford anything more. We license per core rather than per user/connection out of cost considerations. While this isn’t a terrible setup, it allows for no other protections other than backup (to tape, currently). Oracle does not allow you to bring much into the equation of data protection/redundancy/resiliency without, what? Oh yeah, more licensing. Want to replicate? License. Want to virtualize? Nope, gotta license all of your hosts cores. Ridiculous. Now, I’ve read a bunch of blogs and heard a bunch of users talk about ways using cluster tweaks to say they fall within Oracle’s virtualization scheme, but our Oracle rep (who likes to visit us often to check for compliance) has yet to confirm this does in fact fall within compliance.

The big problem: PowerEdge 2850 and 1850 line is pretty old. What happens if when the server croaks? Reinstall and restore from backup. This is not an acceptable SLA for our constituents.

The solution: Boot on SAN

The Setup:
– EqualLogic PS6000XV – primarily used for our virtualization environment
– PowerEdge 2850 Single Socket, Single Core, Hyperthreading turned off.
– QLogic QLA4050C iSCSI HBA

The process (it’s a bit long and I’ll explain at the end why I went the way I did with the RDM):

 Create Volume on EqualLogic and add to vSphere

1. Open up your EqualLogic Group Manager
2. Click on ‘Create Volume’ under the Activities column
3. Give the volume a name (e.g. BootOnSAN) and description (general purpose of volume) and hit Next
4. Give the volume a size (this can be changed later), leave Thin provision box unchecked, hit Next
5. Under the iSCSI Access screen, check the Limit access to iSCSI initiator name box and enter the IQN of an ESX Host that you’ll be building the server on. To get the IQN from the host:
   1. Open vSphere Client, click on a host and to the Configuration tab
   2. Click on Storage Adapters under the Hardware section
   3. Click on the iSCSI Software Adapter, copy the iSCSI Name under the Details section
6. Check the Allow simultaneous connections from initiators with different IQN names box
7. Click Next and then Finish
8. In vSphere client, rescan iSCSI devices for new storage devices under Configuration –> Storage Adapters

Create Virtual Machine with RDM drive

1. Create a new virtual machine from the host that has access to the volume you just created
2. Create a Custom configuration, click Next
3. Give it a name and click Next
4. Select a datastore for the configuration files for the server. This doesn’t matter too much because the server is not going to be virtual. Just don’t choose a local store. Click Next
5. Select Virtual Machine Version: 7 on the Virtual Machine Version page, click Next
6. In the Guest OS page, select the Linux radio and then Red Hat Enterprise Linux 5 (32-bit) (32bit for my ERP because it requires it, 64bit if other application), click Next
7. Choose your number of Processors, amount of RAM and network settings per server needs in the next 3 section.
8. In the SCSI Controller page, select LSI Logic SAS, click Next
9. In the Select a Disk page, select Raw Device Mappings, click Next
10. In the Select and Configure a Raw LUN page, select the volume that was just added under the Storage Adapters section of the Host configuration, click Next
11. Click Next in the Select a Datastore page
12. Select Physical in the Select Compatibility Mode page, click Next
13. Click Next in the Advanced Options page selecting the default Virtual Device Node options
14. Click Finish

Install Red Hat

1. In vSphere Client, edit the virtual machine settings to add the appropriate Red Hat ISO to the CD/DVD Drive, make sure to check the Connected and Connected at Power On boxes.
2. Open a vSphere Client Console to the VM and boot the VM
3. NOTE: This guide assumes a Red Hat EL 5 install.
4. Install in Graphical Mode, skip the media test
5. Click Next on the initial screen and the next 3 screens (Initial setup screen/language/keyboard settings)
6. Skip entering the Installation Number, click OK, click Skip
7. Accept default partition layout (removing all partitions and create default) and click Next
   1. Note: Disk should be labeled ad EQLOGIC
8. Review dialog box and click Yes
9. Edit Network settings
10. Select your time zone  from the time zone drop down, click Next
11. Create the root password, click Next
12. Select the Customize now radio button on the next screen and click Next
13. Make sure you select the following packages for your installation:
    1. Development:
       1. Check Development Libraries
          1. Check libstdc++44
       2. Check Development Tools
          1. Check all gcc packages
    2. Servers
       1. Check Server Configuration Tools
    3. Base System
       1. Base:
          1. Make sure dmraid is checked
          2. Check iscsi-initiator-utils
14. Click Next, then Next to begin installation
15. Run through the First Run Setup making sure to:
    1. Set SELinux to Disabled
    2. If no license is being used, don’t contact RHN

Install QLogic HBA Drivers

Based on: http://filedownloads.qlogic.com/files/Driver/69647/readme_iscsi_k26_5010304.htm

1. Download latest drivers from support.qlogic.com
2. copy tar file (qla4xxx-vx.yy.zz.tar.gz) to /tmp
3. extract files: tar -zxvf qla4xxx-vx.yy.zz.tar.gz
4. cd into the file created (vx.yy.zz)
5. enter the following command to build the driver sources based on the standard SLES9/RHEL4 build environment:
   # ./extras/build.sh
6. Enter the following command to build and install the driver module files:
   # ./extras/build.sh install
   This command does the following:
   1. Builds the driver .ko files.
   2. Copies the .ko files to the appropriate /lib/modules/… directory.
   3. Adds the appropriate directive in the modprobe.conf[.local] file to remove theqla4xxx_conf module when the qla4xxx modules are unloaded.
7. Enter the following command to load the driver for qla4xxx HBAs:
   # /sbin/modprobe -v qla4xxx

Inject Drivers into RAMDISK Image

1. Update the module configuration file as follows:
   – Edit the /etc/modprobe.conf file and add the following entry, if it is not already present:
   alias scsi_hostadapter# qla4xxx (# is the next highest number if one’s there are already)
2. Change to the /boot directory.
   # cd /boot
3. Enter the following command to backup the current RAMDISK image:
   # cp -f initrd-2.6.[kernel_version].img initrd.2.6.[k_v].img.bak
4. Enter the following command to build the RAMDISK image:
   For RedHat :
   # mkinitrd -f initrd-2.6.[kernel_version].img 2.6.[kernel_version]
5. Shutdown the server
   # /sbin/shutdown -h now

Configure iSCSI HBA in Fast!UTIL BIOS

1. Boot physical server with QLogic HBA installed
2. Enter Fast!UTIL with CTRL+Q on server boot
3. Go to Configuration Settings
4. Go to Host Adapter Settings
   1. Spin-up delay: Enable
   2. Initiator IP Address via DHCP: No
   3. Initiator IP Address: aaa.bbb.ccc.ddd (your EqualLogic Group IP address)
   4. Subnet Mask: Appropriate Netmask
   5. Gateway IP Address: your SAN Network Gateway
   6. Initiator Name: leave this as default – enter into text editor
   7. ESC to exit
5. ESC to exit
6. Save Changes
7. Add the Initiator Name to the access list in EQL Group Manager
   1. Open up your EqualLogic Group Manager
   2. Expand Volumes and click on the volume you created in the “Create Volume on EqualLogic” section (e.g. BootOnSAN)
   3. Click on the Access tab and click Add to the right of the Access Control List
   4. Check the box next to Limit access to iSCSI initiator name: and enter the IQN from previous step and click OK
8. Back in Fast!UTIL, go back in to Configuration Settings
9. Go to iSCSI Boot Settings
10. Go to Primary Boot Device Settings
    1. Target IP:  aaa.bbb.ccc.ddd (your EqualLogic Group IP address)
    2. Target Port, Boot LUN and iSCSI Name: leave as default (Port:3260, LUN 0)
    3. ESC to exit
11. ESC to exit
12. Save Changes

• ESC to exit
• Select Scan iSCSI Devices
  1. This should mount the iSCSI LUN and present you with information on it
• ESC
• Back to Configuration Settings
• Select iSCSI Boot Settings
  1. Select the top line (Primary, not Primary Boot Device Settings)
  2. This will scan the iSCSI LUN and present you with a LUN to choose from to mount,
  3. Select the LUN to boot from and hit Enter
  4. This will fill in the iSCSI Name
  5. ESC to exit
  6. Save Changes
• Exit Fast!UTIL
• Reboot System
• Enter BIOS Setup
• Select the HBA from the Hard-Disk Drive Sequence boot list
• Save and restart 

Modify Network Settings

1. log into machine from local console
2. Edit /etc/sysconfig/network-scripts/ifcfg-eth0 to be: (ifcfg-eth0 happens to be the file that grabs the HBA as a regular nic in the OS)
   
   • # QLogic Corp. ISP4022-based Ethernet NIC
   • DEVICE=eth0
   • BOOTPROTO=static
   • ONBOOT=no (you don’t want the OS grabbing the HBA as a nic, since it’s, you know.. busy)
     
   • HWADDR=xxxxxxxxx (leave as whatever it is)
3. Edit /etc/sysconfig/network-scripts/ifcfg-eth1 to be:
   • # Intel Corporation 82541GI Gigabit Ethernet Controller
   • DEVICE=eth1
   • BOOTPROTO=static
   • ONBOOT=yes
   • IPADDR=Ip-address-of-your-choosing
   • HWADDR=xxxxxxxxx (leave as whatever it is)
   • Any other networking setting you require (broadcast, netmask, ,etc.)
     
4. Reboot

Now, when the 2850 dies, I can take that SAME HBA, throw it in another 2850 or 1850, boot, and I’m done.

Why I went with an RDM rather than booting off the CD and installing to the presented LUN? Well, that’s pretty easy. As of Red Hat 5.5, the iSCSI LUN from the HBA was never presented as an installation target destination. Yes, I’m sure I could have spent 2 days getting the QLogic drivers slipstreamed into the Red Hat DVD, but I value my hair while I have it.

PS: anyone have any old PowerEdge 1850’s with single procs
they’d like to donate? :)