I enabled the event JOB_OVER_MAX_DUR. for our scheduled jobs and all by a sudden the database jammed! It was dumping lika an horse!
from the trace-files:
----- Error Stack Dump -----
ORA-01001: invalid cursor
ORA-06512: at
"SYS.DBMS_AQ_INV", line 208
ORA-08102: index key
not found, obj# 12482, file 1, block 28009 (2)
ORA-06512: at
"SYS.DBMS_AQADM", line 1471
ORA-06512: at line 1
----- Current SQL Statement for this session (sql_id=14ur5x3gfybjy) -----
delete from "SYS"."AQ$_SCHEDULER$_EVENT_QTAB_H" where msgid = :1 and subscriber# = :2 and name = '0' and address# = 0
From the alert.log:
ORA-00600: internal error code, arguments: [kwqitnmphe:ltbagi], [1], [0], [], [], [], [], [], [], [], [], []
ORA-01001: invalid cursor
ORA-06512: at "SYS.DBMS_AQ_INV", line 208
ORA-08102: index key not found, obj# 12482, file 1, block 28009 (2)
ORA-06512: at "SYS.DBMS_AQADM", line 1471
ORA-06512: at line 1
ORA-01403: no data found
ORA-06512: at "SYS.DBMS_AQ_INV", line 1085
ORA-06512: at line 1
ORA-01403: no data found
ORA-06512: at "SYS.DBMS_AQ_INV", line 1085
ORA-06512: at line 1
Dumping diagnostic data in directory=[cdmp_20130918123540], requested by (instance=1, osid=13808 (J001)), summary=[abnormal process termination].
First stop the scheduler:
SQL> exec
dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED', 'TRUE');
I tried to rebuild the index:
SQL> Alter
index sys.scheduler$_event_qtab_hist rebuild online;
No difference!
I tried to rebuild the table:
SQL> alter TABLE SYS.AQ$_SCHEDULER$_EVENT_QTAB_H move online;
alter TABLE SYS.AQ$_SCHEDULER$_EVENT_QTAB_H move online
*
ERROR at line 1:
ORA-24005: Inappropriate utilities used to perform DDL on AQ table
SYS.AQ$_SCHEDULER$_EVENT_QTAB_H
No luck there!
But wait!!! An index on an index-organized table!?! Where did that one come from?
SQL> drop index SYS.SCHEDULER$_EVENT_QTAB_HIST;
SQL> exec
dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED', 'FALSE');
All works great again!
The script to reproduce the index is as follows:
CREATE INDEX SYS.SCHEDULER$_EVENT_QTAB_HIST ON SYS.AQ$_SCHEDULER$_EVENT_QTAB_H
(SUBSCRIBER#, NAME, ADDRESS#)
LOGGING
TABLESPACE SYSTEM
PCTFREE 10
INITRANS 2
MAXTRANS 255
STORAGE (
INITIAL 64K
NEXT 1M
MAXSIZE UNLIMITED
MINEXTENTS 1
MAXEXTENTS UNLIMITED
PCTINCREASE 0
FREELISTS 1
FREELIST GROUPS
1
BUFFER_POOL DEFAULT
FLASH_CACHE DEFAULT
CELL_FLASH_CACHE DEFAULT
)
NOPARALLEL;