Could not start database ORA-01092: Oracle instance terminated. Disconnection forced

Oracle11g CRS with an Oracle10g database.

When testing our new RAC installation we disconnected our NetApp storage brutally from the cluster by disconnecting the network cables.
Connected it back again and restarted the both nodes of the cluster. The crs came upp fine but the database failed to start.
The log shiws the following errors:

ORA-00604: error occured at recursive SQL level 1
ORA-00376: file 2 cannot be read at this time
ORA-01110: data file 2: '/oradata/myDb/dbfile/undotbs01.dbf'



$ . oraenv
myDb1

$ sqlplus / as sysdba

SQL> startup mount

SQL> select name, status, file# from gv$datafile;

NAME                                          STATUS                FILE#
----------------------------------   ------------------  -----
/oradata/myDb/dbfile/undotbs01.dbf   OFFLINE             2
/oradata/myDb/dbfile/undotbs02.dbf   ONLINE               3
/oradata/myDb/dbfile/system01.dbf     ONLINE                1

The undo-tablespace has suddenly gone offline!

I tried the following:

SQL> alter database recover automatic datafile 2;

SQL> alter database datafile 2 online;

SQL> alter database open;

database opened


This could also be done from RMAN:


$ rman target / nocatalog

RMAN> startup mount

RMAN> recover datafile 2;

RMAN> sql 'alter database datafile 2 online';

RMAN> alter database open;

database opened

All well again!

PRKR-1005 : adding of cluster database testdb configuration failed, PROC-5: User does not have permission to perform a cluster registry operation on this key

PRKR-1005 : adding of cluster database myDb configuration failed, PROC-5: User does not have permission to perform a cluster registry operation on this key

I was struggling trying to get a failed databas installation, Oracle 10.2.0.5 2 nodes RAC, working. The database instances could be started manually with:

Logged in as oracle

$ sqlplus / as sysdba

SQL> startup


But the registration with the crs, in this case an Oracle 11.2.0.3, did not work and I tryed everything to unregister the instances and the database from the cluster:

$ srvctl remove instance -d myDb -i myDb1

$ srvctl remove database -d myDb

I restarted the cluster and crs_stat showed no database registered with crs! But...

$ srvctl add database -d myDb -o /oracle/10.2.0.5

Just wouldn't work! The databas was already registered!

Run the following as root

This is strange but can bve fixed. First check that you have a backup of the crs config:

$ ocrconfig -showbackup

Then stop all cluster services:

$ crsctl stop cluster -all

Export the settings to a temporary file:

$ ocrconfig -export /tmp/config

Edit the file /tmp/config and remove the lines containing myDb:

DATABASE.LOG.myDb

DATABASE.LOG.myDb1.INSTANCE

DATABASE.LOG.myDb2.INSTANCE

Import the setting back again:

$ ocrconfig -import /tmp/config

Start the cluster services:

$ crsctl start cluster -all

Now you can try to register again as oracle

$ srvctl add database -d myDb /oracle/10.2.0.5

ORA-14086: a partitioned index may not be rebuilt as a whole

SQL> ALTER INDEX REBUILD ONLINE NOLOGGING compress 2 storage (initial 8k next 8k);
            *
ERROR at line 1:
ORA-14086: a partitioned index may not be rebuilt as a whole

 
To compress (use key compression on) a partitioned index you first have to create the index with the compress keyword. Drop it and recreate!
 
If you try to alter the index:

SQL> ALTER INDEX  compress;

ORA-02243: ogiltigt argument för ALTER INDEX eller ALTER MATERIALIZED VIEW


After recreation (drop and create) you can compress each partition on the index if you like:


SQL> alter index rebuild
    partition compress 2;