Check and remove orphaned Runbooks

To keep System Center Orchestrator up and running, and of course healthy, you need to take care of orphaned Runbooks. Sometime, or maybe more often, Orchestrator produces orphaned Runbooks. As you maybe know, Orchestrator acts very special with his Database, so always make sure the Database is attached as best as possible.

What are orphaned Runbooks

Orphaned Runbooks are instances of Runbooks, which Orchestrator, or the Database, thinks that they are running, but they are not. This does not sound very bad, but it can be, trust me. If Problems exists, which are causing orphaned Runbooks, there will be a lot of orphaned Runbooks in a short period of time.

How does orphaned runbooks occur?

Mostly they occur, cause the SQL is not as good as possible attached, with other words, the connection between SQL and Orchestrator is bad. At the other Hand, the Orchestrator Server itself is not equipped as it should be, in case of RAM or CPU, and even Disk Performance. Also the Structure of your Runbooks, and how you built them, can be the reason for orphaned Runbooks.

At least, and of course the point we can not solve, Orchestrator itself can and will produce orphaned Runbooks. In this case only Microsoft can help, but….

How can I check, if I have orphaned Runbooks?

The easiest way is to use System Center Orchestrator Health Checker. By the Way, i would really recommend to use the Health Checker at each Orchestrator Installation, see my Article for Details: https://www.techguy.at/system-center-orchestrator-health-checker/

Take a look at this Screenshot

2019-02-26 11_05_53-CitrixDesktop ClientManagement - Citrix Workspace

At the Column “Running Instances”  should usually show only one instance, except you have configured your Runbooks for more than one instance.

But in our case, we haven’t, but as you can see, Orchestrator thinks, that there are more than on instance running in some Runbooks. if you take a look at the Runbook in the Runbook Designer, you should see something like this. A List of running instances, but no Runbook is active, or only one.

How to fix it

First, you need to make sure, your Orchestrator Server is healthy, has enough Power and RAM, and your SQL DB is clean, and is providing the correct Indexes.

If you still experience orphaned Runbooks, my Team and i created a PowerShell Script, which we scheduled once a day, at our Customers.

The Script will query all, running instances, also the orphaned, and check each Runbook Server if there is an active Process with this instance. If there is not, we will kill this Instance.

The Script

$Databaseserver='Server-SCO01';
$Orchestratordatabase='Orchestrator';
$Orchestratorservers="Server-SCO01;Server2"
$User="domain\scoservice";
$UserPassword="mygoodpassword";

[array]$Actionservers=$Orchestratorservers -split ";"

foreach ($Actionserver in $Actionservers)
    {
    if ($user -ne "")
        {
        $password = ConvertTo-SecureString  -AsPlainText $UserPassword -Force;
        $Credential = New-Object System.Management.Automation.PsCredential($user,$password);
        $session=New-PSSession -ComputerName $Actionserver -Credential $Credential;
        }
    else
        {
        $session=New-PSSession -ComputerName $Actionserver;
        }

    $updatesqlQuery=Invoke-Command -Session $session -scriptblock {
        $Databaseserver=$Args[0];
        $Orchestratordatabase=$Args[1];
        $Actionserver=$Args[2];
        $updatesqlQuery="";
        $SQLQueryrunningpolicies="SELECT Policies.[UniqueID],
                Policies.[ProcessID],
                POLICIES.JobId,
                Servertable.computer,
                policies.TimeStarted,
                Policies.Status,
                Policies.TimeEnded
                FROM [$Orchestratordatabase].[dbo].[POLICYINSTANCES] as Policies
                inner join [$Orchestratordatabase].[dbo].[ACTIONSERVERS] as Servertable on policies.ActionServer = Servertable.UniqueID
                where TimeEnded is null and Servertable.computer = '$Actionserver'
                order by Processid";
        $runningpolicies=Invoke-Sqlcmd -ServerInstance $Databaseserver -Database $Orchestratordatabase -Query $SQLQueryrunningpolicies;
        foreach ($policy in $runningpolicies)
            {
            Try
                {
                $process=Get-Process -id $policy.processid -ErrorAction Stop
                }
            Catch
                {
                $uniqueid=$policy.UniqueID;
                $jobid=$policy.JobId;
                $updatesqlQuery+="Update [$Orchestratordatabase].[dbo].[POLICYINSTANCES]
                    Set TimeEnded=GetDate(), Status='failed'
                    where Jobid='$jobid' and UniqueID='$uniqueid'"    +"`n";
                }
            }
        if ($updatesqlQuery -ne "")
            {
            Invoke-Sqlcmd -ServerInstance $Databaseserver -Database $Orchestratordatabase -Query $updatesqlQuery;
            }
        Return $updatesqlQuery
        } -ArgumentList $Databaseserver, $Orchestratordatabase, $Actionserver;
    Remove-PSSession $session;
    }

Please note, this Script does not cure all your Problems, take car of y healthy and running Orchestrator Server first, before using this Script

Michael Seidl aka Techguy

4 thoughts on “Check and remove orphaned Runbooks”

  1. Hi Michael

    Danke für Deinen Artikel.

    Wir haben das Script in leicht abgeänderter Form ohne Remoting implementiert und es funktioniert sehr gut!

    Danke und Gruess
    Raphael

  2. Andre Grootveld

    Hi Michael,

    I think I am having the same issue over here and want to go ahead with the script but I have two concerns:

    1) $process=Get-Process -id $policy.processid -ErrorAction Stop
    You stop the process based on processid assuming that it is a orphaned runbook. But can it be that the runbook process is already stopped and another process has that id? So you stop another process?

    2) For security matter I don’t want to have scripts with username/password in it..

Leave a Comment

Your email address will not be published. Required fields are marked *

*