Configure InfiniBand Listener on Exadata:

Configure InfiniBand Listener on Exadata:

InfiniBand listener can be used for ETL's, RMAN active duplicate procedure etc.. for fast data transfer using IB network.


Make sure following entries are in sqlnet.ora file on all the database servers, we use soft link from GRID home to DB home.

AUTOMATIC_IPC = ON
names.directory_path = (TNSNAMES,EZCONNECT)
names.default_domain = world
name.default_zone = world
sqlnet.expire_time = 15



1.
Add/Identify 8 new IP addresses on the private Infiniband, which are different from existing "-priv" IPs (Same subnet as the IB network) -- Network Admin Task.

Add Private Virtutal IPs to /etc/hosts for all Exadata compute nodes and client (Informatica Servers or any other):

### Compute Node Listener Interface details
192.168.16.xx atl01db01-ibvip.corp.dwire.com atl01db01-ibvip
192.168.16.xx atl01db02-ibvip.corp.dwire.com atl01db02-ibvip
192.168.16.xx atl01db03-ibvip.corp.dwire.com atl01db03-ibvip
192.168.16.xx atl01db04-ibvip.corp.dwire.com atl01db04-ibvip
192.168.16.xx atl01db05-ibvip.corp.dwire.com atl01db05-ibvip
192.168.16.xx atl01db06-ibvip.corp.dwire.com atl01db06-ibvip
192.168.16.xx atl01db07-ibvip.corp.dwire.com atl01db07-ibvip
192.168.16.xx atl01db08-ibvip.corp.dwire.com atl01db08-ibvip

-- Infiniband Sample config:

cat /etc/sysconfig/network-scripts/ifcfg-bondib0

#### DO NOT REMOVE THESE LINES ####
#### %GENERATED BY CELL% ####
DEVICE=bondib0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.16.1
NETMASK=255.255.252.0
NETWORK=192.168.16.0
BROADCAST=192.168.19.255
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100"
IPV6INIT=no
MTU=65520


2.
As root user, add secondary network to the cluster:

-- Currenly How many networks configured? -- Only one network found
/u01/app/11.2.0.2/grid/bin/crsctl stat res -t | grep net

Output:
$ /u01/app/11.2.0.2/grid/bin/crsctl stat res -t | grep net
ora.net1.network


 -- Add the second network
/u01/app/11.2.0.2/grid/bin/srvctl add network -k 2 -S 192.168.16.0/255.255.252.0/bondib0

-- Help 11.2.0.2
> srvctl add network -help

Adds a network configuration to the Oracle Clusterware.
Usage: srvctl add network [-k net_num] -S subnet/netmask/[if1[if2...]] [-w network_type] [-v]
-k net_num network number (default number is 1)
-S subnet/netmask/[if1[if2...]] NET address spec for network
-w network_type The network type (static, dhcp, mixed)
-h Print usage
-v Verbose output

-- Verify newly added network has been added
/u01/app/11.2.0.2/grid/bin/crsctl stat res -t | grep net

Output:
$ /u01/app/11.2.0.2/grid/bin/crsctl stat res -t | grep net
ora.net1.network
ora.net2.network


3.
-- Register newly created Virtual IP's with the CRS as root user:

cd /u01/app/11.2.0.2/grid/bin
srvctl add vip -n atl01db01 -A atl01db01-ibvip/255.255.252.0/bondib0 -k 2
srvctl add vip -n atl01db02 -A atl01db02-ibvip/255.255.252.0/bondib0 -k 2
srvctl add vip -n atl01db03 -A atl01db03-ibvip/255.255.252.0/bondib0 -k 2
srvctl add vip -n atl01db04 -A atl01db04-ibvip/255.255.252.0/bondib0 -k 2
srvctl add vip -n atl01db05 -A atl01db05-ibvip/255.255.252.0/bondib0 -k 2
srvctl add vip -n atl01db06 -A atl01db06-ibvip/255.255.252.0/bondib0 -k 2
srvctl add vip -n atl01db07 -A atl01db07-ibvip/255.255.252.0/bondib0 -k 2
srvctl add vip -n atl01db08 -A atl01db08-ibvip/255.255.252.0/bondib0 -k 2


4.
As oracle user:
-- Register newly added listener to CRS to make sure listener will come up automatically after the server restart.

srvctl add listener -l LISTENER_IB -k 2 -p TCP:1522
srvctl config listener

Output:
$ srvctl config listener
Name: LISTENER
Network: 1, Owner: oracle
Home: CRS home
End points: TCP:1521
Name: LISTENER_IB
Network: 2, Owner: oracle
Home: CRS home
End points: TCP:1522

-- Help 11.2.0.2
> srvctl add listener -help

Adds a listener configuration to the Oracle Clusterware.

Usage: srvctl add listener [-l lsnr_name] [-s] [-p "[TCP:]port[, ...][/IPC:key][/NMP:pipe_name][/TCPS:s_port] [/SDP:port]"] [-o oracle_home] [-k net_num]

-l lsnr_name Listener name (default name is LISTENER)
-o oracle_home ORACLE_HOME path (default value is CRS_HOME)
-k net_num network number (default number is 1)
-s Skip the checking of ports
-p "[TCP:]port[, ...][/IPC:key][/NMP:pipe_name][/TCPS:s_port] [/SDP:port]" Comma separated tcp ports or listener endpoints
-h Print usage


5.
Update $ORACLE_HOME/network/admin/tnsnames.ora file on all nodes:

Imp:
Each TNSNAMES.ORA file is different for each server depending on the Local and Remote listener settings.

Local Listener,    have information about local node and local IB vip host information.
Remote Listener, have information about other IB vip hosts information except the Local node


Sample entries for node 1 (atl01db01):

Local:
LISTENER_IBLOCAL.WORLD =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db01-ibvip)(PORT = 1522))
)
)

LISTENER_IPLOCAL.WORLD =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db01)(PORT = 1521))
)
)

Remote:
LISTENER_IBREMOTE.WORLD =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db02-ibvip)(PORT = 1522)) -- no node 1 information on node 1!!
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db03-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db04-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db05-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db06-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db07-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db08-ibvip)(PORT = 1522))
)
)

LISTENER_IPREMOTE.WORLD =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = ATL02-SCAN)(PORT = 1521))
)
)

Imp:
Repeat the same for the other nodes, changing Local and Remote values accordingly.


6 .
-- Update the listeners_network DB parameter on all the databases with the listener addresses from both the networks, i.,e network1 and network2

alter system set listener_networks='((NAME=network2) (LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE))','((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both sid='*';

-- Restart the listener and validate the entry.
show parameter listener_networks


-- What do I see with "ps -ef"?
> ps -ef | grep LISTENER
oracle 12874 1 0 May09 ? 00:01:20 /u01/app/11.2.0.2/grid/bin/tnslsnr LISTENER -inherit
oracle 24335 1 0 May12 ? 00:05:22 /u01/app/11.2.0.2/grid/bin/tnslsnr LISTENER_IB -inherit


7.
-- Add TNS entry on client side:

Sample TNS entry:

ODS.WORLD =
(DESCRIPTION =
(LOAD_BALANCE = ON)
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db01-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db02-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db03-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db04-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db05-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db06-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db07-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = atl01db08-ibvip)(PORT = 1522))
)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ODS_INFA_SERVICE.world)
)
)


-- How did you create the Service?:
$ srvctl add service -d ODS -s ODS_INFA_SERVICE -r ODS1,ODS2,ODS3,ODS4 -a ODS5,ODS6,ODS7,ODS8 -q TRUE -m BASIC -e SELECT -z 180 -w 5 -j LONG -k2

-- Service configuration:
$ srvctl config service -d ODS -s ODS_INFA_SERVICE
Service name: ODS_INFA_SERVICE
Service is enabled
Server pool: ODS_ODS_INFA_SERVICE
Cardinality: 4
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: true
Failover type: SELECT
Failover method: BASIC
TAF failover retries: 180
TAF failover delay: 5
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Preferred instances: ODS1,ODS2,ODS3,ODS4
Available instances: ODS5,ODS6,ODS7,ODS8


-- Help 11.2.0.2
> srvctl add service -help

Adds a service configuration to the Oracle Clusterware.

Usage: srvctl add service -d db_unique_name -s service_name {-r "preferred_list" [-a "available_list"] [-P {BASIC | NONE | PRECONNECT}]
-g pool_name [-c {UNIFORM | SINGLETON}] } [-k net_num] [-l [PRIMARY][,PHYSICAL_STANDBY][,LOGICAL_STANDBY][,SNAPSHOT_STANDBY]]
[-y {AUTOMATIC | MANUAL}] [-q {TRUE | FALSE}]
[-x {TRUE | FALSE}] [-j {SHORT | LONG}]
[-B {NONE | SERVICE_TIME | THROUGHPUT}]
[-e {NONE | SESSION | SELECT}] [-m {NONE | BASIC}]
[-z failover_retries] [-w failover_delay] [-t edition] [-f]

-d db_unique_name Unique name for the database

-s service Service name
-r "preferred_list" Comma separated list of preferred instances
-a "available_list" Comma separated list of available instances
-g pool_name Server pool name
-c {UNIFORM | SINGLETON} Service runs on every active server in the server pool hosting this service (UNIFORM) or just one server (SINGLETON)
-k net_num network number (default number is 1)
-P {NONE | BASIC | PRECONNECT} TAF policy specification
-l role Role of the service (primary, physical_standby, logical_standby, snapshot_standby)
-y policy Management policy for the service (AUTOMATIC or MANUAL)
-e Failover type Failover type (NONE, SESSION, or SELECT)
-m Failover method Failover method (NONE or BASIC)
-w integer Failover delay
-z integer Failover retries
-t edition Edition (or "" for empty edition value)
-j clb_goal Connection Load Balancing Goal (SHORT or LONG). Default is LONG.
-B Runtime Load Balancing Goal Runtime Load Balancing Goal (SERVICE_TIME, THROUGHPUT, or NONE)
-x Distributed Transaction Processing Distributed Transaction Processing (TRUE or FALSE)
-q AQ HA notifications AQ HA notifications (TRUE or FALSE)
Usage: srvctl add service -d db_unique_name -s service_name -u {-r "new_pref_inst"
-a "new_avail_inst"} [-f]
-d db_unique_name Unique name for the database
-s service Service name
-u Add a new instance to service configuration
-r new_pref_inst Name of new preferred instance
-a new_avail_inst Name of new available instance
-f Force the add operation even though a listener is not configured for a network
-h Print usage