Could not start database ORA-01092: Oracle instance terminated. Disconnection forced

Oracle11g CRS with an Oracle10g database.

When testing our new RAC installation we disconnected our NetApp storage brutally from the cluster by disconnecting the network cables.
Connected it back again and restarted the both nodes of the cluster. The crs came upp fine but the database failed to start.
The log shiws the following errors:

ORA-00604: error occured at recursive SQL level 1
ORA-00376: file 2 cannot be read at this time
ORA-01110: data file 2: '/oradata/myDb/dbfile/undotbs01.dbf'



$ . oraenv
myDb1

$ sqlplus / as sysdba

SQL> startup mount

SQL> select name, status, file# from gv$datafile;

NAME                                          STATUS                FILE#
----------------------------------   ------------------  -----
/oradata/myDb/dbfile/undotbs01.dbf   OFFLINE             2
/oradata/myDb/dbfile/undotbs02.dbf   ONLINE               3
/oradata/myDb/dbfile/system01.dbf     ONLINE                1

The undo-tablespace has suddenly gone offline!

I tried the following:

SQL> alter database recover automatic datafile 2;

SQL> alter database datafile 2 online;

SQL> alter database open;

database opened


This could also be done from RMAN:


$ rman target / nocatalog

RMAN> startup mount

RMAN> recover datafile 2;

RMAN> sql 'alter database datafile 2 online';

RMAN> alter database open;

database opened

All well again!

PRKR-1005 : adding of cluster database testdb configuration failed, PROC-5: User does not have permission to perform a cluster registry operation on this key

PRKR-1005 : adding of cluster database myDb configuration failed, PROC-5: User does not have permission to perform a cluster registry operation on this key

I was struggling trying to get a failed databas installation, Oracle 10.2.0.5 2 nodes RAC, working. The database instances could be started manually with:

Logged in as oracle

$ sqlplus / as sysdba

SQL> startup


But the registration with the crs, in this case an Oracle 11.2.0.3, did not work and I tryed everything to unregister the instances and the database from the cluster:

$ srvctl remove instance -d myDb -i myDb1

$ srvctl remove database -d myDb

I restarted the cluster and crs_stat showed no database registered with crs! But...

$ srvctl add database -d myDb -o /oracle/10.2.0.5

Just wouldn't work! The databas was already registered!

Run the following as root

This is strange but can bve fixed. First check that you have a backup of the crs config:

$ ocrconfig -showbackup

Then stop all cluster services:

$ crsctl stop cluster -all

Export the settings to a temporary file:

$ ocrconfig -export /tmp/config

Edit the file /tmp/config and remove the lines containing myDb:

DATABASE.LOG.myDb

DATABASE.LOG.myDb1.INSTANCE

DATABASE.LOG.myDb2.INSTANCE

Import the setting back again:

$ ocrconfig -import /tmp/config

Start the cluster services:

$ crsctl start cluster -all

Now you can try to register again as oracle

$ srvctl add database -d myDb /oracle/10.2.0.5