[Upgrade with DBUA + REMOTE_LISTENER with unresolvable SCAN => Fail] & [crsctl “-unsupported” supports you!]

I played with some troubleshooting tasks during database upgrade and stumbled upon some – how can I describe it? – a new findings for me.

I have following environment:

• 2-node RAC environment with Grid Infrastructure version 12.1.0.2
• 1 RAC database still running on 11GR2 (11.2.0.3)
• Note: the parameter REMOTE_LISTENER is empty

Scenario:

• That database should be upgraded to the version 12.1.0.2 using DBUA

I run ORAchk Upgrade Readiness Assessment & CVU to Validate Readiness for Oracle Clusterware Upgrades before upgrade and everything was fine.

Then I executed PreUprade tasks and started the DBUA tool from the 12c home. Upgrade went smoothly until 53% and then displayed this error:

The reason of this failure was the unresolvable SCAN name. So, the init parameter REMOTE_LISTENER was empty before upgrade and DBUA set it automatically to the SCAN name, but if the SCAN name for some reason is not resolvable during the upgrade, then the upgrade will fail. I tried the workaround, adding REMOTE_LISTENER=” to the initDUM11G1.ora and initDUM11G2.ora:

SPFILE='+DATA/dum11g/spfiledum11g.ora'
REMOTE_LISTENER=''

but it didn’t helped. I aborted the upgrade and DBUA restored the database to the before upgrade status.

Later, I tried the upgrade again, but this time two DUM11G databases were standing in the list! I checked the /etc/oratab and found only one DUM11G record! How could that happen!?

I checked the clusterware resources and found this:

--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.DUM11G.DUM11G1.inst
      1        OFFLINE OFFLINE                               STABLE
ora.DUM11G.DUM11G2.inst
      1        OFFLINE OFFLINE                               STABLE
ora.DUM11G.db
      1        OFFLINE OFFLINE                               STABLE
ora.dum11g.db
      1        ONLINE  OFFLINE                               Instance Shutdown,ST
                                                             ABLE
      2        ONLINE  OFFLINE                               Instance Shutdown,ST
                                                             ABLE

Very interessting, isn’t it!?

Because this is a dummy database, I wanted to drop the database to have a clean environment:

[oracle@dummy ~]$ dbca -silent -deleteDatabase -sourceDB DUM11G -sysDBAUserName SYS -sysDBAPassword dummypass

Connecting to database
9% complete
14% complete
19% complete
23% complete
28% complete
33% complete
38% complete
47% complete
Updating network configuration files
48% complete
52% complete
Deleting instances and datafiles
66% complete
80% complete
95% complete
100% complete
Look at the log file "/opt/oracle/cfgtoollogs/dbca/DUM11G.log" for further details.

but the output from the “crsctl stat res -t” still shows some resources of the DUM11G database/instances:

[oracle@dummy ~]$ crsctl stat res

...
NAME=ora.DUM11G.DUM11G1.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.DUM11G.DUM11G2.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.DUM11G.db
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
...

[oracle@dummy ~]$ crsctl stat res -t -w "TYPE = application"
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.DUM11G.DUM11G1.inst
      1        OFFLINE OFFLINE                               STABLE
ora.DUM11G.DUM11G2.inst
      1        OFFLINE OFFLINE                               STABLE
ora.DUM11G.db
      1        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------

Hmmm, very strange. Ok, I had to clean up that mess.

[oracle@dummy ~]$ crsctl delete resource ora.DUM11G.DUM11G1.inst
CRS-4995:  The command 'Delete  resource' is invalid in crsctl. Use srvctl for this command.

Whoooops! It doesn’t work …

But, the “-unsupported” flag supports you!

[oracle@dummy~]$ crsctl delete resource ora.DUM11G.DUM11G1.inst -unsupported

[oracle@dummy~]$ crsctl delete resource ora.DUM11G.DUM11G2.inst -unsupported

[oracle@dummy~]$ crsctl delete resource ora.DUM11G.db -unsupported

Finally, all DUM11G resources were removed from the clusterware.

Now I have to analyse this issue with the aborted upgrade and “zombie” clusterware resources, and currently I don’t know if this is some bug or “works as designed”.

Stay tuned.

UPDATE on 10.03.2016:

Well, it seems, that DBUA ignores the value you set for the REMOTE_LISTENER parameter and set it during the upgrade to the SCAN name. I tried 3 test cases:

• REMOTE_LISTENER parameter is empty before upgrade
• REMOTE_LISTENER parameter is set to (ADRESS=…(HOST=…)(PORT=…))
• REMOTE_LISTENER parameter is set to use an alias defined in the tnsnames.ora

All of them failed, when the SCAN is unresolvable during the upgrade. Ok, it could happen, that SCAN for some reason is unresolvable, but why DBUA sets REMOTE_LISTENER explicitly to the SCAN name overwriting the value I specified before upgrade!?

Still searching for the answer…