Blackout Thinks It’s Still Active

I had a blackout running for a few hours in OEM due to a hardware move. Ended blackout in OEM all hosts report as being available but a few database and a listener target for one of the hosts all stay at pending state.

Restarted agent. Restarted OMS but still no joy.

Checked emoms.log find errors.

2016-06-27 20:33:17,274 [pool-42-thread-1] WARN client.RepositoryUtils logp.251 - getting unrestricted (private/repository owner) connection 2016-06-27 20:33:17,288 [pool-42-thread-1] WARN client.RepositoryUtils logp.251 - getting unrestricted (private/repository owner) connection 2016-06-27 20:33:17,452 [[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR browser.InstanceBrowserViewBean logp.251 - RemoteAccessException occured during refreshConfiguration oracle.sysman.eml.ecm.track.RemoteAccessException: Encountered the following problem(s) during collection and loading process: [null: the target is currently blacked out, null: the target is currently blacked out, null: No metric collection named host_san for host,mybox.local, null: the target is currently blacked out] at oracle.sysman.eml.ecm.track.TrackUtilities.doCollection(TrackUtilities.java:677) at oracle.sysman.eml.ecm.track.SnapshotFactory.refreshTargetConfig(SnapshotFactory.java:550) at oracle.sysman.eml.ecm.track.SnapshotFactory.refreshTargetConfig(SnapshotFactory.java:404) at oracle.sysman.eml.ecm.track.SnapshotFactory.refreshTargetConfig(SnapshotFactory.java:387)

Strange as OEM console shows the blackout as ended.

[oracle@myoembox bin]$ emcli get_blackout_details -name="Blackout-27-Jun-2016 18:24:46" -createdby="MATT" Status Status ID Run Jobs Next Start Duration Reason Frequency Repeat Days Months Start Time End Time TZ Region TZ Offset Type Stopped 10 yes none 01:00 none none none none 2016-06-27 18:27:51 none Europe/London +01:00 Blackout

Check the local agent.

[oracle@mybox bin]$ ./emctl status blackout Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. Blackoutname = Blackout-27-Jun-2016 18:24:46 Targets = (mybox.local:host,) Time = ({2016-06-27|18:27:54,|} ) Expired = False

So the local OEM agent thinks a blackout is still in place.

[oracle@mybox emd]$ pwd /n01/app/oracle/agent/agent_inst/sysman/emd [oracle@mybox emd]$ cat blackouts.xml <Blackouts> <Blackout NAME="Blackout-27-Jun-2016 18:24:46" UNIQ_NAME="Blackout-27-Jun-2016 18:24:46" GUID="E951242013B5E24E215EE615D9384C63" CREATED_USING="CONSOLE" NODE_LEVEL="TRUE"> <Schedule START_DATE="2016-06-27" START_TIME="18:27:54"/> <Target NAME="mybox.local" TYPE="host"/> </Blackout> </Blackouts> [oracle@mybox emd]$

Lets remove the local blackout files.

./emctl stop agent mv blackouts.xml blackouts.lck /tmp ./emctl start agent

Checked OEM console now all targets show as available.

Originally published at blog.mattdevdba.tech on June 27, 2016.