B
Troubleshooting Oracle Enterprise Manager in Real Application Clusters

Appendix B describes service discovery issues for Oracle Enterprise Manager in Oracle Real Application Clusters environments. Specific topics covered in this appendix are:

• Resolving Service Discovery Failures in Real Application Clusters
• Understanding Discovery in Real Application Clusters
• Discovery Results in Real Application Clusters
• Troubleshooting Discovery in Real Application Clusters

Resolving Service Discovery Failures in Real Application Clusters

Discovery of nodes and managed objects by Oracle Enterprise Manager is robust and rarely fails once a correct configuration is established. Failures typically occur because:

• The Oracle Intelligent Agent is not started on the node
• The Global Services Daemon (GSD) is not running
• The configuration is incorrect

If starting the Oracle Intelligent Agent and the GSD does not resolve the problem, then the discovery failure could be due to a more serious configuration issue.

Understanding Discovery in Real Application Clusters

During discovery, a services.ora file on the managed nodes is created in the $ORACLE_HOME/network/agent directory on UNIX operating systems and %ORACLE_HOME%\network\admin directory on Windows NT or Windows 2000 operating systems. This file contains information about the nodes and their services (databases, instances, and listeners) discovered.

If you used the Database Configuration Assistant to create your database, then the appropriate discovery information will be configured for you automatically. However, if you manually configure your database, then you must manually create the entries for this configuration information as described in the following sections.

The services.ora file is created from the Real Application Clusters configuration information in the shared raw device (or shared cluster file system file maintained by the SRVCTL Utility) and the following sources on the managed nodes:

• The UNIX oratab File and the Windows NT and Windows 2000 Registry
• SRVM Configuration Repository
• The listener.ora File
• The tnsnames.ora File
• The sqlnet.ora File

You must accurately configure each of these components so that discovery succeeds.

The UNIX oratab File and the Windows NT and Windows 2000 Registry

Discovery first determines the Real Application Clusters database name and the nodes associated with the database. How the discovery process accomplishes this is operating-system dependent.

• UNIX and oratab
• Windows NT and Windows 2000

UNIX and oratab

On UNIX operating systems, discovery uses information in the oratab entry for the name of the cluster database. The location of the file for the oratab entry is platform specific and is found in either the /etc/oratab or /var/opt/oracle/oratab directories. This file contains entries of the form:

db_name:$ORACLE_HOME:N

Where db_name is the database name and $ORACLE_HOME is the Oracle home given to your database. From this entry, the database name is acquired.

Next, discovery executes the following command to retrieve the instances-to-node mappings of the Real Application Clusters database:

srvctl config database -d db_name

Where db_name was retrieved from oratab.

Windows NT and Windows 2000

Obtain the value of instance SID from the following Registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\HOMEID\ORACLE_SID

SRVM Configuration Repository

Discovery requires that the configuration for each instance be recorded in the SRVM repository. To determine whether the configuration for an instance is recorded, enter the following command:

srvctl config database -p db_name

If the output from this command reveals that entries are missing from the repository, then manually add the configuration for the affected instances and databases as described in "Task 3: Create the Real Application Clusters Configuration with SRVCTL".

The listener.ora File

Discovery locates the listener and instance names for a node from the listener.ora file located in $ORACLE_HOME/network/admin on UNIX operating systems and %ORACLE_HOME%\network\admin on Windows NT or Windows 2000 operating systems on the discovered nodes.

Discovery requires the following entries:

• The listener address must contain a TCP/IP address that specifies the HOST value as the host name of the discovered node. This ensures that the listener actually resides on the node.

  (description=
   (address=(protocol=tcp)(host=db1-server)(port=1521)))
  
  

• For each listener that runs on the node, the sid_list_listener_name entry is name):

  sid_list_listener=
   (sid_list=
    (sid_desc=
      (sid_name=db1)))
  
  

The listener.ora file created after installation typically contains the configuration for discovery.

How SRVCTL Operations Associate Listeners with Instances

When SRVCTL operations attempt to start instances and their associated listeners, the listener that SRVCTL will start is determined from the static configuration information set for the SID_NAME parameter in the SID_LIST_LISTENER entry in the listener.ora file. For example, the SID_NAME is set for the instance in the listener.ora file as follows:

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (ORACLE_HOME = /private/system/db/
      (SID_NAME = db1)
    )
  )

For this example, when SRVCTL starts instance db1, SRVCTL will also attempt to start the listener named LISTENER.

The tnsnames.ora File

The discovery process determines the service names and connect descriptors information for the database and its instances using the entries in the tnsnames.ora file. This file is located in $ORACLE_HOME/network/admin on UNIX operating systems and %ORACLE_HOME%\network\admin on Windows NT or Windows 2000 on the discovered nodes, is read by the discovery process to determine names and address information for the database and instances on a node.

Discovery requires the following entries:

• Each instance must have an entry in the tnsnames.ora file. For example:

  db1.us.acme.com= 
   (description= 
    (address=(protocol=tcp)(host=db1-server)(port=1521))
    (connect_data= 
      (service_name=db.us.acme.com)
      (instance_name=db1)))
  

• The Real Application Clusters database should have an entry. For example:

  db.us.acme.com= 
   (description= 
    (load_balance=on)
    (failover=on)
   (address=(protocol=tcp)(host=db1-server)(port=1521))
     (address=(protocol=tcp)(host=db2-server)(port=1521)) 
    (connect_data=
       (service_name=db.us.acme.com)))
  
  

The sqlnet.ora File

The sqlnet.ora file is automatically configured with:

• The computer's domain which is automatically appended to any unqualified net service name. For example, if the default domain is set to us.acme.com, Oracle resolves db in the connect string CONNECT scott/tiger@db as: db.us.acme.com.
• A naming method the server can use to resolve a name to a connect descriptor.

  The order of naming methods is as follows: directory naming (for the Customized installations only), tnsnames.ora file, Oracle Names server, and host naming.

The following is a sample SQLNET.ORA file created during a preconfigured database configuration install:

names.default_domain=us.acme.com
names.directory_path=(tnsnames, onames,hostname)

Discovery Results in Real Application Clusters

Discovery results in the creation of:

• Discovered nodes and services listed in services.ora file in the
  $ORACLE_HOME/network/agent directory on UNIX operating systems and %ORACLE_HOME%\network\admin directory on Windows NT or Windows 2000.

  The services.ora file should contain an ops_database entry for the Real Application Clusters database, that lists the node, database address, and name of the database. The following example shows a database named db.us.acme.com running on node db1-server and db2-server. The database address comes from the db.us.acme.com net service name in the tnsnames.ora file.

  db.us.acme.com=(ops_database, db1-server, (description=(load_
  balance=on)(failover=on) (address=(protocol=tcp)(host=db1-server) 
  (port=1521))(address=(protocol=tcp)(host=db2-server)(port=1521))(connect_
  data=(service_name=db.us.acme.com)))), db.us.acme.com)
  
  

  The services.ora file should also contain an OPS_INSTANCE entry for the instance that runs on the node. This entry identifies the:

  • Name of the node
  • Net service entry for the instance from the tnsnames.ora file
  • Real Application Clusters database name to which the instance belongs
  • Listener name

  The following example shows the instance db1 runs on node db1-server, and is listened for by listener_db1-server:

  db1.us.acme.com=(ops_instance, db1-server, 
  (description=(address=(protocol=tcp)(host=db1-server)(port=1521))(connect_
  data= (service_name=db.us.acme.com)(instance_name=db1))), db.us.acme.com, 
  (PROP=(LSNRS=listener_db1-server)(ORACLE_HOME=/private/system/db:n)(ORACLE_
  SID=db1))))
  
  

• Oracle records errors in the nmiconf.log file in the $ORACLE_HOME/network/log directory on UNIX operating systems and in the %ORACLE_HOME%\network\log directory on Windows NT or Windows 2000.

Troubleshooting Discovery in Real Application Clusters

If the services.ora file contains an ORACLE_DATABASE entry instead of the ops_database and ops_instance entries, then discovery has failed. To resolve this:

1. Check that the database is configured correctly:

   On UNIX:

   • Verify that oratab file is configured correctly.

   On Windows NT or Windows 2000:

   • Check the registry entries associated with the database.
2. Run the following command to verify the configuration:

   SRVCTL config database -d db_name
   
   
   

   SRVCTL displays the name of the node and the instance for the node. The following example shows a node named db1-server running an instance named db1 with a listener named LISTENER_NODE1. If you execute the following command:

   srvctl config database -d db
   
   

   The output should appear similar to:

   db1-server db1 /private/system/db
   db2-server db2 /private/system/db
   
   

3. Inspect the listener.ora and tnsnames.ora file entries to ensure that the required entries are present.

   See Also: Oracle9i Real Application Clusters Administration for information on troubleshooting with Oracle Enterprise Manager Thread Trace Files, and how to contact Oracle Support Services