scheduler ORA-01001: invalid cursor ORA-06512: at "SYS.DBMS_AQ_INV" ORA-08102: index key not found, obj

I enabled the event JOB_OVER_MAX_DUR. for our scheduled jobs and all by a sudden the database jammed! It was dumping lika an horse!

from the trace-files:

----- Error Stack Dump -----

ORA-01001: invalid cursor

ORA-06512: at "SYS.DBMS_AQ_INV", line 208

ORA-08102: index key not found, obj# 12482, file 1, block 28009 (2)

ORA-06512: at "SYS.DBMS_AQADM", line 1471

ORA-06512: at line 1

----- Current SQL Statement for this session (sql_id=14ur5x3gfybjy) -----

 delete from "SYS"."AQ$_SCHEDULER$_EVENT_QTAB_H" where msgid = :1 and subscriber# = :2 and  name = '0' and address# = 0

From the alert.log:

ORA-00600: internal error code, arguments: [kwqitnmphe:ltbagi], [1], [0], [], [], [], [], [], [], [], [], []

ORA-01001: invalid cursor
ORA-06512: at "SYS.DBMS_AQ_INV", line 208
ORA-08102: index key not found, obj# 12482, file 1, block 28009 (2)
ORA-06512: at "SYS.DBMS_AQADM", line 1471
ORA-06512: at line 1
ORA-01403: no data found
ORA-06512: at "SYS.DBMS_AQ_INV", line 1085
ORA-06512: at line 1
ORA-01403: no data found
ORA-06512: at "SYS.DBMS_AQ_INV", line 1085
ORA-06512: at line 1
Dumping diagnostic data in directory=[cdmp_20130918123540], requested by (instance=1, osid=13808 (J001)), summary=[abnormal process termination].

First stop the scheduler:

SQL> exec dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED', 'TRUE');

I tried to rebuild the index:

SQL> Alter index sys.scheduler$_event_qtab_hist rebuild online;

No difference!

I tried to rebuild the table:

SQL> alter TABLE SYS.AQ$_SCHEDULER$_EVENT_QTAB_H move online;

alter TABLE SYS.AQ$_SCHEDULER$_EVENT_QTAB_H move online

                *

ERROR at line 1:

ORA-24005: Inappropriate utilities used to perform DDL on AQ table

SYS.AQ$_SCHEDULER$_EVENT_QTAB_H

No luck there!

But wait!!! An index on an index-organized table!?! Where did that one come from?

SQL> drop index SYS.SCHEDULER$_EVENT_QTAB_HIST;

SQL> exec dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED', 'FALSE');

All works great again!

The script to reproduce the index is as follows:

CREATE INDEX SYS.SCHEDULER$_EVENT_QTAB_HIST ON SYS.AQ$_SCHEDULER$_EVENT_QTAB_H

(SUBSCRIBER#, NAME, ADDRESS#)

LOGGING

TABLESPACE SYSTEM

PCTFREE    10

INITRANS   2

MAXTRANS   255

STORAGE    (

            INITIAL          64K

            NEXT             1M

            MAXSIZE          UNLIMITED

            MINEXTENTS       1

            MAXEXTENTS       UNLIMITED

            PCTINCREASE      0

            FREELISTS        1

            FREELIST GROUPS  1

            BUFFER_POOL      DEFAULT

            FLASH_CACHE      DEFAULT

            CELL_FLASH_CACHE DEFAULT

           )

NOPARALLEL;