DynCfg override removal should restore the masked discovered job #23302

Problem

Removing a DynCfg override for a collector job deletes the active graph resource, but it does not immediately reactivate the unchanged lower-priority user, discovered, or stock candidate for the same job identity.

This is generic Job Manager/DynCfg behavior, not discoverer-specific. It affects SNMP and every other discovery provider.

@ilyam8, this is in the shared Go Job Manager/DynCfg lifecycle and likely needs a generic reconciliation path rather than collector-specific workarounds.

Current behavior

Configuration precedence is:

DynCfg > user > discovered > stock

The override flow works:

1. Discovery installs a job.
2. Editing that job through DynCfg preserves its module and name and adopts it into DynCfg ownership.
3. The DynCfg owner masks lower-priority candidates.

But on remove:

1. prepareRemove() deletes the DynCfg graph record/resource.
2. The discovery DecisionIndex is not notified.
3. It still has the unchanged lower-priority candidate in acknowledged.
4. Its same-UID fast path returns without reinstalling the candidate.

As a result, “remove override / return to discovered defaults” leaves the job absent until a plugin restart, candidate content change, or disappear/reappear transition. An identical periodic discovery scan does not restore it.

Evidence

• Source precedence and current limitation: src/go/plugin/agent/jobmgr/ARCHITECTURE.md:266-276
• Discovery state and acknowledged candidates: src/go/plugin/agent/jobmgr/discovery/decision.go:27-41
• Same-UID early return: src/go/plugin/agent/jobmgr/discovery/decision.go:215-223
• DynCfg removal only removes the current graph owner: src/go/plugin/agent/jobmgr/joboutput/dyncfg_prepare.go:432-480
• Generic discovery fallback test only covers changes within discovery: src/go/plugin/agent/jobmgr/discovery/decision_test.go:160-186
• Higher-priority owner protections: src/go/plugin/agent/jobmgr/joboutput/dyncfg_jobs_test.go:306-418

SNMP has no special workaround. All discoverers feed the same DecisionIndex and cannot observe collector-job DynCfg removal.

Expected behavior

After a successful DynCfg override removal, Job Manager should immediately re-arbitrate the same full job name and install the highest-priority remaining candidate.

If no candidate remains, the job should stay removed.

Suggested design direction

Add a generic “higher-priority owner released” reconciliation event for FullName:

1. Trigger only after a successful DynCfg removal transaction.
2. Deliver it to DecisionIndex through the composition/discovery owner, serialized on the same queue/goroutine as normal discovery batches.
3. Force reconciliation for that name without treating the unchanged acknowledged UID as currently installed.
4. Install the best retained user/discovered/stock candidate through the normal job transaction path.

Avoid:

• discoverer-specific re-emission;
• target hash churn;
• collector-specific fallback logic;
• synchronous kernel re-entry from an AfterApply hook.

Acceptance criteria

• Removing a running DynCfg override immediately restores an unchanged discovered candidate.
• The same works for masked user and stock candidates.
• No remaining candidate leaves the job removed.
• Failed, rejected, or no-op removals do not trigger reconciliation.
• Candidate change/removal racing with override removal is deterministic.
• No kernel re-entrancy or discovery-state data race.
• Behavior survives restart/replay ordering.
• Tests cover running, accepted/disabled, and failed job states.