Threshold based selective maintenance of series-parallel systems

Abstract

A condition of an asset including one or more subsystems connected in series is monitored. Each subsystem includes one or more components connected in parallel. The asset has one or more jobs. A probability of the asset surviving a predetermined amount of time is determined based on the monitoring and one or more shared resources. The one or more shared resources are configured to be shared between the subsystems. A model is established using a threshold based heuristic maintenance policy. The model is configured to maximize a number of successful jobs that the asset is able to complete based on the determined probability. The one or more shared resources are allocated to the one or more subsystems based on the model.

Description

SUMMARY

Embodiments described herein involve a method comprising monitoring a condition of an asset comprising one or more subsystems connected in series. Each subsystem comprises one or more components connected in parallel. The asset has one or more jobs. A probability of the asset surviving a predetermined amount of time is determined based on the monitoring and one or more shared resources. The one or more shared resources are configured to be shared between the subsystems. A model is established using a threshold based heuristic maintenance policy. The model is configured to maximize a number of successful jobs that the asset is able to complete based on the determined probability. The one or more shared resources are allocated to the one or more subsystems based on the model.

Embodiments involve a system comprising a processor and a memory storing computer program instructions which when executed by the processor cause the processor to perform operations. The operations comprise monitoring a condition of an asset comprising one or more subsystems connected in series. Each subsystem comprises one or more components connected in parallel. The asset has one or more jobs. A probability of the asset surviving a predetermined amount of time is determined based on the monitoring and one or more shared resources. The one or more shared resources are configured to be shared between the subsystems. A model is established using a threshold based heuristic maintenance policy. The model is configured to maximize a number of successful jobs that the asset is able to complete based on the determined probability. The one or more shared resources are allocated to the one or more subsystems based on the model.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows a process for allocating resources using a threshold based heuristic system in accordance with embodiments described herein;

FIG. 1B. illustrates another process for allocating resources using a threshold based heuristic system in accordance with embodiments described herein;

FIG. 2 shows a block diagram of a system capable of implementing embodiments described herein;

FIG. 3 shows a more an optimal maintenance resource allocation policy of threshold versus time slots in accordance with embodiments described herein; and

FIG. 4 illustrates a maintenance resource allocation policy of optimal allocation versus number of maintenance resources in accordance with embodiments described herein;

The figures are not necessarily to scale. Like numbers used in the figures refer to like components. However, it will be understood that the use of a number to refer to a component in a given figure is not intended to limit the component in another figure labeled with the same number.

DETAILED DESCRIPTION

Industrial and mission critical assets such as factory assembly lines, aircraft engines and military equipment are composed of series-parallel systems that require periodic maintenance and replacement of faulty components. The assets typically operate over a finite time window and are brought in (or shut down) for service between consecutive operational windows (e.g., missions). To prevent excessive down times and costly breakdown, a selective maintenance policy that judiciously selects the sub-systems and components for repair and prescribes allocation of resources may be used.

Embodiments described herein involve a threshold based system and method that provides near optimal maintenance policy especially when resources (man power, budget, spare parts) are sparse and have to be shared across the entire planning horizon. The exact planning problem is computationally intensive and our heuristic method will produce near optimal maintenance schedules in reasonable time suitable for practical applications. The threshold value is expressed as the long-term expected maximum reward yielded by holding a single resource in reserve. The heuristic policy described herein determines threshold values that are a function of the resource inventory levels and state of the system. If the immediate reward yielded by allocating a resource exceeds the long-term expected marginal yield (e.g., the threshold value), the policy dictates that the resource be allocated. In this regard, we perform a tradeoff between the immediate reward and the long-term expected reward yielded by holding the resource in reserve (i.e., for allocation at a future time). If a plurality of resources are available, the policy dictates that the resource with the maximum marginal reward be allocated first. Also, if a plurality of sub-systems/components are in need of repair, the policy dictates that those sub-systems/components whose repair will yield the maximum asset reliability be repaired first. Embodiments described herein can be used in any type of system. For example, the techniques described herein can be used in aircraft engine maintenance, assembly line selective maintenance (e.g., in printer ink facilities), industrial equipment, military asset management, and/or weapons allocation (e.g., assignment of weapons to targets).

In some cases, finite resources are typically allocated for each service period (or break). Embodiments described herein specifically include allocation of scarce resources across the planning horizon. In general, consumable resources, (e.g., man hours, parts) can be shared and therefore optimally assigned depending on the condition of the asset and the remaining number of missions (time horizon). The selective maintenance scheduling problem for series-parallel systems is computationally intensive and optimal schedules may take hours to compute on current day edge devices. Embodiments described herein exploit the structure in the problem to derive an intuitive threshold based heuristic policy that is amenable to fast real time implementation. The heuristic policy employs a scalable linear backward recursion for computing the threshold values as opposed to solving a non-linear Bellman recursion (which is typically not scalable for a system with large number of sub-systems and components) for computing the optimal allocation.

FIG. 1A shows a process for allocating resources using a heuristic threshold-based system in accordance with embodiments described herein. A condition of an asset comprising one or more subsystems connected in series is monitored 110. For example, the monitoring 110 may involve determining if all subsystems are functioning properly. Each subsystem has one or more components connected in parallel. One or more of the components may be redundant within each subsystem. In some cases, all of the one or more components are redundant within each subsystem. According to various embodiments, the asset has one or more jobs. According to various embodiments the one or more jobs comprise one or more identical jobs.

A probability of the asset surviving a predetermined amount of time is determined 120 based on the monitoring 110 and one or more shared resources configured to be shared between the subsystems. The shared resources may include one or both of replenishable resources and consumable resources. Specifically, the one or more shared resources include one or more of man hours, budget, parts, and equipment used to perform maintenance.

A model is established 130 using a threshold based heuristic maintenance policy. The model may be configured to maximize a number of successful jobs that the asset is able to complete based on the determined probability. According to various embodiments, the model is configured to minimize a shared resource cost. The resource cost may be calculated using a Maximal Marginal Reward (MMR) algorithm that is configured to maximize an expected payout or minimize an expected cost.

The one or more shared resources are allocated 140 to the one or more subsystems based on the model. A maintenance schedule for the asset may be determined based on the model. In some cases, the one or more components have a known failure rate and the known failure rates are used to establish the model.

FIG. 1B shows a more detailed flow diagram for allocating resources using a heuristic threshold-based system in accordance with embodiments described herein. The process starts 145 and It is determined 155 whether it is the final decision stage. This may occur when enough information is available about the condition of the asset and the number of available resources to build a model.

If it is not determined 155 that it is the final decision stage, a failure probability for all components of the asset is determined 150 until the next decision epoch. The next decision epoch may be a current task that one or more subsystems of the asset is performing and/or a current mission of the asset, for example.

Components and/or subsystems are prioritized for repair and/or replacement 160. The prioritization may be based on a time to failure of the component and/or subsystem, for example. In some cases, the prioritization may be based on a cost of the replacement and/or repair.

A threshold value is computed 170 for the next available resource. It is determined 180 whether the immediate resource allocation reward is greater than the threshold. If it is determined 180 that the immediate resource allocation reward is not greater than the threshold, the process again computes 170 the threshold value for the next available resource.

If it is determined 180 that the immediate resource allocation reward is greater than the threshold, resources are allocated 190. The remaining resources and components selected for repair are iterated through. The process then advances 195 to the next decision epoch and the process returns to determining 155 if it is the final decision stage and the process continues.

If it is determined 155 has been reached, an algorithm is used 165 to allocate all remaining resources. According to various configurations, the algorithm is the MMR algorithm. The process then ends 175.

The methods described herein can be implemented on a computer using well-known computer processors, memory units, storage devices, computer software, and other components. A high-level block diagram of such a computer is illustrated in FIG. 2. Computer 200 contains a processor 210, which controls the overall operation of the computer 200 by executing computer program instructions which define such operation. It is to be understood that the processor 210 can include any type of device capable of executing instructions. For example, the processor 210 may include one or more of a central processing unit (CPU), a graphical processing unit (GPU), a field-programmable gate array (FPGA), and an application-specific integrated circuit (ASIC). The computer program instructions may be stored in a storage device 220 and loaded into memory 230 when execution of the computer program instructions is desired. Thus, the steps of the methods described herein may be defined by the computer program instructions stored in the memory 230 and controlled by the processor 210 executing the computer program instructions. The computer 200 may include one or more network interfaces 250 for communicating with other devices via a network. The computer 200 also includes a user interface 260 that enable user interaction with the computer 200. The user interface 260 may include I/O devices 262 (e.g., keyboard, mouse, speakers, buttons, etc.) to allow the user to interact with the computer. The user interface may include a display 264. The computer may also include a receiver 215 configured to receive data from the user interface 260 and/or from the storage device 220. According to various embodiments, FIG. 2 is a high-level representation of possible components of a computer for illustrative purposes and the computer may contain other components.

Establishing the model using a threshold based heuristic maintenance policy is described in further detail below.

Markov Task Sets

The decision maker (DM) has at his disposal M homogenous resources that are to be sequentially allocated to incoming tasks. Decisions are made at discrete epochs, i.e., t=0, 1, . . . . Each task i has a window of opportunity, [t_sⁱ, t_fⁱ], within which it is active and therefore, the DM can assign resources towards completing the task. A single resource assigned to a task gets the job done at or before the next decision epoch (i.e., task is completed) with probability p<1. So, if k resources are allocated to an incomplete task i at decision epoch t∈[t_sⁱ, t_fⁱ), the task will be completed with probability 1−(1−p)^k at the next decision epoch t+1. We assume “complete information” in that the task completion status is known to the DM at the beginning of each decision epoch. Upon successful completion of task i, the DM earns the positive reward r_i. The DM also incurs a cost c>0 for each resource assigned. Suppose at time 0, the DM knows that N tasks shall arrive with a priori known windows of opportunity. Without loss of generality, we shall assume that at least one of the tasks starts at time 1 i.e., min_it_sⁱ=1. Furthermore, let T=max_it_fⁱ be the time horizon of interest. We wish to compute the optimal allocation of resources to tasks so that the DM accrues the maximal cumulative reward over the time window, t∈[1, T].

Let the task completion status (state) variable: z(i)=0,1 indicate that task i is incomplete and complete respectively. At decision epoch t, let V (z,k,t) indicate the optimal cumulative reward (value function) that can be achieved thereafter where k indicates the number of resources left and the vector z encapsulates the status of each task.

A task i is active at the current time if it is incomplete, i.e., z(i)=0 and if the current time t∈[t_sⁱ, t_fⁱ]. Accordingly, we define the set of active tasks, A(z,t)={i: t∈[t_sⁱ, t_fⁱ] and z(i)=0}.

The decision variable, u is a vector wherein u_i indicates the number of resources to be assigned to task i. The feasible allocation set is shown in (1).
U(z,k,t)={u:Σ_i∈A(z,t)u_i≤k,u_i≥0,∀i∈A(z,t),u_i=0,∀i∉A(z,t)}.  (1)
It follows that the value function V (z,k,t) satisfies the Bellman recursion:

$\begin{array}{cc}V\left(z,k,t\right)=\underset{u\in U\left(z,k,t\right)}{\max }\left\{\sum _{i\in A\left(z,t\right)}\left[-c{u}_i+\left(1-q^{u_i}\right)r_i\right]+\sum _{\overline{z}}\mathrm{Prob}\left(\stackrel{\_}{z}\right)V\left(\overline{z},k-\sum _{i\in A\left(z,t\right)}{u}_i,t+1\right)\right\},& \left(2\right)\end{array}$ $\forall z,k=1,\dots ,M,\forall t\in \left[1,T-1\right],$
where q=1−p. In effect, we assign Σ_i∈A(z,t) u_i resources in total. In the next decision epoch, the status of each task changes according to:

$\begin{array}{cc}\mathrm{Prob}\left(\stackrel{\_}{z}\left(i\right)=z\left(i\right)\right)=\{\begin{array}{c}{q}^{u_i},\forall i\in A\left(z,t\right)\\ 1,\forall i\notin A\left(z,t\right)\end{array}& \left(3\right)\end{array}$
Since the evolution of each task is a completely independent Markov process, the transition probabilities in (3) are given by:
Prob(z)=π_{i=1, . . . ,N} Prob(z(i)).  (4)
The boundary conditions for the value function are given by:

$\begin{array}{cc}V\left(z,k,T\right)=\underset{u\in U\left(z,k,T\right)}{\max }{\sum }_{i\in A\left(z,T\right)}\left[-c{u}_i+\left(1-q^{u_i}\right)r_i\right],\forall z,& \left(5\right)\end{array}$ $k=1,\dots ,M,$ $\begin{array}{cc}V\left(z,0,t\right)=0,\forall z,t.& \left(6\right)\end{array}$
We shall address the much simpler single task scenario first.

Single Task Allocation

Suppose we are only interested in the optimal allocation of resources to a single active task i. For ease of exposition, let us assume that 1=t_sⁱ<t_fⁱ=T_i. The corresponding task specific value function is given by:

$\begin{array}{cc}{V}_i\left(k,t\right)=\underset{u_i\in \left[0,k\right]}{\max }\left\{\left[-c{u}_i+\left(1-q^{u_i}\right)r_i\right]+q^{u_i}{V}_i\left(k-u_i,t+1\right)\right\},& \left(7\right)\end{array}$ $k=1,\dots ,M,\forall t\in \left[1,T_i\right],$
with the boundary conditions:

$\begin{array}{cc}{V}_i\left(k,T_i\right)=\underset{u_i\in \left[0,k\right]}{\max }\left[-c{u}_i+\left(1-q^{u_i}\right)r_i\right],k=1,\dots ,M,& \left(8\right)\end{array}$ $\begin{array}{cc}{V}_i\left(0,t\right)=0,\forall t\in \left[1,T_i\right].& \left(9\right)\end{array}$
The optimal policy is given by:
μ_i(k,t)=arg max_ui∈[0,k]{[−cu_i+(1−q^ui)r_i]+q^uiV_i(k−u_i,t+1)}, k=1, . . . ,M,∀t ∈[1,T_i].  (10)
Suppose the DM has u+1 resources left at the terminal time T_i. We denote the marginal reward yielded by assigning 1 additional resource over and above u resources to the active task i as:
Δ_Tⁱ(u)=−c(u+1)+(1−q^u+1)r_i−[−cu+(1−q^u)r_i]=−c+pq^ur_i.  (11)
Since q<1 it follows that Δ_Tⁱ(u) is monotonic decreasing in u. Let κ_i(T) be the least non-negative integer at which the marginal reward becomes negative i.e., κ_i(T_i)= s.t. r_i<c. It follows that the optimal policy at the terminal time is given by:

$\begin{array}{cc}{\mu }_i\left(k,T_i\right)=\arg {\max }_{u_i\in \left[0,k\right]}\left[-c{u}_i+\left(1-q^{u_i}\right)r_i\right]=\{\begin{array}{cc}k,& k<{\kappa }_i\left(T_i\right),\\ {\kappa }_i\left(T_i\right),& k\ge {\kappa }_i\left(T_i\right).\end{array}& \left(12\right)\end{array}$ $\begin{array}{cc}\Rightarrow V_i\left(k,T_i\right)=-c{\mu }_i\left(k,T_i\right)+\left[1-q^{{\mu }_i\left(k,T_i\right)}\right]r_i,\forall k.& \left(13\right)\end{array}$
In other words, we assign as many resources as is available but no more than the threshold. We will show likewise that at each decision epoch, there is an upper bound to the number of resources that the DM shall assign (optimally) for the remainder of the time horizon. Having computed the terminal value function, we are now in a position to compute the value function at the previous decision epoch. Indeed, we have from (7):

$\begin{array}{cc}{V}_i\left(k,T_i-1\right)=\underset{u_i\in \left[0,k\right]}{\max }\left\{\left[-c{u}_i+\left(1-q^{u_i}\right)r_i\right]+q^{u_i}{V}_i\left(k-u_i,T_i\right)\right\},& \left(14\right)\end{array}$ $k=1,\dots ,M.$
We can show that the value function V_i(k,T_i−1) has a threshold based analytical form similar to V_i(k,T_i). Let the function inside the max operator in (14) be given by,

$\begin{array}{cc}W\left(k,u,T_i\right)=\left[-\mathrm{cu}+\left(1-q^u\right)r_i\right]+q^{u_i}{V}_i\left(k-u,T_i\right)=\{\begin{array}{cc}-\mathrm{cu}+r_i-q^u{\stackrel{\_}{r}}_i,& u<k-{\kappa }_i\left(T_i\right),\\ -\mathrm{cu}+r_i-q^{u}c\left(k-u\right)-q^k{r}_i,& u\ge k-{\kappa }_i\left(T_i\right),\end{array}& \left(15\right)\end{array}$
where, r_i=cκ_i(T_i)+q^κi(Ti)r_i. Note that we substitute for V_i(k−u, T_i) from (13) to get (15). As before, the marginal reward at time T_i−1 is defined according to:

$\begin{array}{cc}{\Delta }_{T-1}^i\left(k,u\right)=W\left(k,u+1,T_i\right)-W\left(k,u,T_i\right)=\{\begin{array}{cc}-c+{\mathrm{pq}}^u{r}_i,& u<k-{\kappa }_i\left(T_i\right)\\ -c+c{q}^u{f}_{T_i-1}\left(k-u\right),& u\ge k-{\kappa }_i\left(T_i\right),\end{array}& \left(16\right)\end{array}$
where, f_Ti−1(w)=pw+q. Note however that, unlike Δ_Tⁱ(u), the marginal reward at time T_i−1 is also a function of k, the number of resources left with the DM. The optimal allocation is determined by the threshold value at which the marginal reward becomes negative.

Let κ_i(T_i−1)= s.t. r_i<c. We have the following results.

Lemma 1 κ_i(T_i−1)≤κ_i(T_i).

Proof. If κ_i(T_i)=0, r_i=r_i and it follows from the definition of κ_i(T_i−1) that it will also equal 0. Suppose κ_i(T_i)>0. We have:

$\begin{array}{cc}\begin{array}{c}{r}_i-{\stackrel{\_}{r}}_i=-c{\kappa }_i\left(T_i\right)+\left[1-q^{{\kappa }_i\left(T_i\right)}\right]r_i\\ =-c{\kappa }_i\left(T_i\right)+p{\sum }_{\ell =0}^{{\kappa }_i\left(T_i\right)-1}{q}^{\ell }{r}_i\\ ={\sum }_{\ell =0}^{{\kappa }_i\left(T_i\right)-1}\left[{\mathrm{pq}}^{\ell }{r}_i-c\right]>0\end{array},& \left(17\right)\end{array}$
where (17) follows from the definition of κ_i(T_i). In other words, r_i>c for all ∈ [0, κ_i(T_i)). It immediately follows from the definition of κ_i(T_i−1) that κ_i(T_i−1)≤κ_i(T_i).

From the definition of the marginal reward (16), we note that determining when it switches sign depends in part on f_Ti−1(k−u). Clearly, f_Ti−1(k−u)=p(k−u)+q is a decreasing function of u. Since, q⁰f_Ti−1(k)=pk+q≥1 and q^kf_Ti−1(0)=q^k+1<1, ∃u*(k)=min_u∈[0,k]u s.t. q^uf_Ti−1(k−u)≤1.

Lemma 2

$\begin{array}{cc}{u}^{*}\left(k-1\right)=\{\begin{array}{cc}{u}^{*}\left(k\right),& \mathrm{if}{q}^{u^{*}\left(k\right)}{f}_{T_i-1}\left(k-u^{*}\left(k\right)\right)<q,\\ u^{*}\left(k\right)-1,& \mathrm{otherwise}.\end{array}& \left(18\right)\end{array}$

Proof. By definition we have: q^u*(k)f_Ti−1(k−u*(k))≤1 and q^u*(k)−1f_Ti−1(k−u*(k)+1)>1. In addition, we can write:

$\begin{array}{cc}{q}^{u^{*}\left(k\right)}{f}_{T_i-1}\left(k-1-u^{*}\left(k\right)\right)=q^{u^{*}\left(k\right)}{f}_{T_i-1}\left(k-u^{*}\left(k\right)\right)-{\mathrm{pq}}^{u^{*}\left(k\right)}<1.& \left(19\right)\end{array}$ $q^{u^{*}\left(k\right)-2}{f}_{T_i-1}\left(k-1-u^{*}\left(k\right)+2\right)=q^{u^{*}\left(k\right)-2}{f}_{T_i-1}\left(k-u^{*}\left(k\right)+1\right)>1.$ $q^{u^{*}\left(k\right)-1}{f}_{T_i-1}\left(k-1,u^{*}\left(k\right)-1\right)=q^{u^{*}\left(k\right)-1}{f}_{T_i-1}\left(k-u^{*}\left(k\right)\right)u^{*}\left(k-1\right)=\{\begin{array}{cc}{u}^{*}\left(k\right)-1,& \mathrm{if}{q}^{u^{*}\left(k\right)}{f}_{T_i-1}\left(k-u^{*}\left(k\right)\right)\le q,\\ u^{*}\left(k\right),& \mathrm{otherwise}.\end{array}$

Theorem 1 The optimal allocation policy at T_i−1 is dictated by:

$\begin{array}{cc}\begin{array}{c}{\mu }_i\left(k,T_i-1\right)=\{\begin{array}{cc}{u}^{*}\left(k\right),& k\le {\kappa }_i\left(T_i\right)+{\kappa }_i\left(T_i-1\right),\\ {\kappa }_i\left(T_i-1\right),& k>{\kappa }_i\left(T_i\right)+{\kappa }_i\left(T_i-1\right),\end{array}\end{array}& \left(20\right)\end{array}$ $\mathrm{where}{u}^{*}\left({\kappa }_i\left(T_i\right)+{\kappa }_i\left(T_i-1\right)\right)={\kappa }_i\left(T_i-1\right)\mathrm{and}\mathrm{for}1<k\le {\kappa }_i\left(T_i\right)+{\kappa }_i\left(T_i-1\right),$ $\begin{array}{cc}{u}^{*}\left(k-1\right)=\{\begin{array}{cc}{u}^{*}\left(k\right)-1,& \mathrm{if}{q}^{u^{*}\left(k\right)}{f}_{T_i-1}\left(k-u^{*}\left(k\right)\right)\le q,\\ u^{*}\left(k\right),& \mathrm{otherwise}.\end{array}& \left(21\right)\end{array}$

Proof. First, we note that the marginal reward function Δ_T−1ⁱ(k,u) defined earlier (16) is monotonic decreasing in u. Indeed, for u∈[0, k−κ_i(T_i)), Δ_T−1ⁱ(k,u)=−c+pq^ur_i and since q<1 it is decreasing in u. For u≥k−κ_i(T_i), Δ_T−1ⁱ(k,u)=−c(1−q^u+1)+cpq^u(k−u) which is clearly also a decreasing function of u. So, we only need to show that:
Δ_T−1ⁱ(k,k−κ_i(T_i)−1)>Δ_T−1ⁱ(k,k−κ_i(T_i)).
From the definition of κ_i(T_i), we have: pq^κi(Ti)−1r_i>c. So, we can write:

$\begin{array}{cc}\begin{array}{c}{\Delta }_{T-1}^i\left(k,k-{\kappa }_i\left(T_i\right)\right)=-c+q^{k-{\kappa }_i\left(T_i\right)}c\left[p{\kappa }_i\left(T_i\right)+q\right]\\ <-c+q^{k-{\kappa }_i\left(T_i\right)-1}\mathrm{cp}{\kappa }_i\left(T_i\right)+q^{k-{\kappa }_i\left(T_i\right)}c\\ <-c+q^{k-{\kappa }_i\left(T_i\right)-1}\mathrm{cp}{\kappa }_i\left(T_i\right)+q^{k-{\kappa }_i\left(T_i\right)}\left[{\mathrm{pq}}^{{\kappa }_i\left(T_i\right)-1}r\right]\\ =-c+{\mathrm{pq}}^{k-{\kappa }_i\left(T_i\right)-1}{\stackrel{\_}{r}}_i\\ ={\Delta }_{T-1}^i\left(k,k-{\kappa }_i\left(T_i\right)-1\right).\end{array}& \left(22\right)\end{array}$
As before, It follows that the optimal allocation u* is the threshold value at which the marginal reward becomes negative. However, the added complexity here is the dependence on k. So, we deal with the three possible scenarios that lead to different threshold values:

• • k>κ_i(T_i)+κ_i(T_i−1): For this case, κ_i(T_i−1)<k−κ_i(T_i). We have already shown that Δ_T−1ⁱ(k,u) decreases for u ∈ [0, k−κ_i(T_i)) and so, the threshold occurs at u*=κ_i(T_i−1). Indeed, this follows from the definition of κ_i(T_i−1).
  • k=κ_i(T_i)+κ_i(T_i−1): For this case, k−κ_i(T_i)=κ_i(T_i−1). As such, Δ_T−1ⁱ(k,u)>0 for u ∈ [0, k−κ_i(T_i)). From the definition of κ_i(T_i), we have: pq^κi(Ti)−1r_i>c⇒pq^κi(Ti)r_i>cq. The marginal reward at κ_i(T_i−1) is given by:

$\begin{array}{cc}\begin{array}{c}{\Delta }_{T-1}^i\left(k,{\kappa }_i\left(T_i-1\right)\right)=c\left[p{\kappa }_i\left(T_i\right)+q\right]q^{{\kappa }_i\left(T_i-1\right)}-c\\ =\left[\mathrm{pc}{\kappa }_i\left(T_i\right)+\mathrm{cq}\right]q^{{\kappa }_i\left(T_i-1\right)}-c\\ <\left[c{\kappa }_i\left(T_i\right)+q^{{\kappa }_i\left(T_i\right)}{r}_i\right]{\mathrm{pq}}^{{\kappa }_i\left(T_i-1\right)}-c\\ ={\mathrm{pq}}^{{\kappa }_i\left(T_i-1\right)}{\stackrel{\_}{r}}_i-c\le 0,\end{array}& \left(23\right)\end{array}$
where the last inequality follows from the definition of κ_i(T_i−1). So, it follows that the threshold value at which the marginal reward turns negative is given by u*=κ_i(T_i−1).

• • k<κ_i(T_i)+κ_i(T_i−1): For this case, k−κ_i(T_i)<κ_i(T_i−1). As such, Δ_T−1ⁱ(k,u)>0 for u ∈ [0, k−κ_i(T_i)). So, the threshold must occur at some u ∈ [k−κ_i(T_i), k] at which q^uf_Ti−1(k−u) dips below unity. From Lemma 2 it follows that: for

$\begin{array}{cc}k\le {\kappa }_i\left(T_i\right)+{\kappa }_i\left(T_i-1\right),u^{*}\left(k-1\right)=\{\begin{array}{cc}{u}^{*}\left(k\right)-1,& \mathrm{if}{q}^{u^{*}\left(k\right)}{f}_{T_i-1}\left(k-u^{*}\left(k\right)\right)\le q,\\ u^{*}\left(k\right),& \mathrm{otherwise}.\end{array}& \left(24\right)\end{array}$
with the boundary condition u*(κ_i(T_i)+κ_i(T_i−1))=κ_i(T_i−1), which was established in 2).

In summary, we have:

$\begin{array}{cc}{\mu }_i\left(k,T_i-1\right)=\{\begin{array}{cc}{\kappa }_i\left(T_i-1\right),& k\ge {\kappa }_i\left(T_i\right)+{\kappa }_i\left(T_i-1\right),\\ u^{*}\left(k\right),& k<{\kappa }_i\left(T_i\right)+{\kappa }_i\left(T_i-1\right),\end{array}& \left(25\right)\end{array}$
where u*(k) is computed via the backward recursion (24).

It follows from (15) that the optimal value function at time T_i−1 is given by:

$\begin{array}{cc}{V}_i\left(k,T_i-1\right)=\{\begin{array}{cc}-c{\kappa }_i\left(T_i-1\right)+r_i-q^{{\kappa }_i\left(T_i-1\right)}{\stackrel{\_}{r}}_i,& k\ge {\kappa }_i\left(T_i\right)+{\kappa }_i\left(T_i-1\right),\\ -{\mathrm{cu}}^{*}\left(k\right)+\left[1-q^k\right]r_i-q^{u^{*}\left(k\right)}c\left(k-u^{*}\left(k\right)\right),& \mathrm{otherwise}.\end{array}& \left(26\right)\end{array}$

Corollary 1 The optimal allocation at time T_i−1 is such that the number of resources left at the last stage T_i is no more than the optimal threshold κ_i(T_i). In other words,
k−μ_i(k,T_i−1)≤κ_i(T_i), k<κ_i(T_i)+κ_i(T_i−1).  (27)

Proof. This follows from (24). As k decreases by a value of 1, u*(k) either remains the same or goes down by 1. So, it cannot decrease fast enough that k−u*(k) exceeds the threshold κ_i(T_i). In other words, when the total number of resources is less than the sum of thresholds κ_i(T_i)+κ_i(T_i−1), no resources will be left unused at the final time T_i.

Threshold Based Optimal Policy and Recursion

In light of Theorem 1, one can easily compute the optimal thresholds i.e., number of resources to be allocated at each decision epoch when resources are aplenty. In particular, we have the result: for any t∈[1, T_i],

$\begin{array}{cc}{\mu }_i\left(k,t\right)={\kappa }_i\left(t\right),k\ge {\kappa }_i\left(T_i-\right),& \left(28\right)\end{array}$
where the thresholds can be computed via a backward recursion. For any t=T_i, . . . , 1:

$\begin{array}{cc}\begin{array}{c}{\kappa }_i\left(t\right)=\underset{\ell \ge 0}{\min }\ell s.t.{\mathrm{pq}}^{\ell }{R}_{T_i-t}\le c,\\ R_{T_i-t+1}=c{\kappa }_i\left(t\right)+q^{{\kappa }_i\left(t\right)}{R}_{T_i-t},\end{array}& \left(29\right)\end{array}$
with the boundary condition, R₀=r_i. The optimal value function is given by:

$\begin{array}{cc}{V}_i\left(k,t\right)=-c{\kappa }_i\left(t\right)+r_i-q^{{\kappa }_i\left(t\right)}{R}_{T_i-t},k\ge {\kappa }_i\left(T_i-\right).& \left(30\right)\end{array}$

In other words, one can quickly compute the thresholds κ_i(t) over the entire time window without resorting to the non-linear Bellman recursion. At decision time t, the DM allocates exactly the threshold amount so long as the number of available resources is no less than the sum of thresholds over the remainder of the time horizon.

Remarks

This also tells us that for any task i, the DM requires at most a maximum of

${\kappa }_i\left(T_i-\right)$
resources. Furthermore, since the rewards are strictly decreasing with (see (17) in Lemma 1), it follows that κ_i(T_i−) must go to zero as increases. Indeed, there exists n(i) ∈ [0, ∞) such that κ_i(T(i)−n(i))=0. So, the DM need never allocate more than

${\kappa }_i\left(T_i-\right)$
resources to task i and furthermore, the window of opportunity for target i needs to be no longer than n(i). If the window is any longer, it is optimal for the DM to simply hold off without assigning any resources until the remaining time window shrinks to n(i).

Numerical Example

We use as an example the allocation of maintenance personnel to a piece of industrial equipment prone to failure to illustrate our method and results. The example problem involves the allocation of maintenance personnel to a single equipment over a window of opportunity, T=10. The reward, r=90, the probability of resolving the fault, p=0.25 and the cost of allocating a maintainer, c=1. According to various configurations, since the cost of allocating one maintainer is c=1, minimizing this cost is equivalent to minimizing the number of maintainers. Using the backward recursion (29), we compute the threshold values, κ(j), j=1, . . . , 10 as shown in FIG. 3. In particular, κ(10)=11 and κ(9)=5. So, we compute the optimal allocation at time t=9 for all possible k<κ(10)+κ(9) as shown in FIG. 4. As per Lemma 2, the optimal allocation starts at κ(9) and either remains the same or decreases by one with decreasing k.

Single Task Optimal Allocation at All Stages

In the previous section, we have completely characterized the optimal allocation at stages T_i and T_i−1. Note that the optimal allocation at stage T_i is linear up to the threshold κ_i(T_i) and a constant thereafter. The optimal allocation at stage T_i−1 is far more interesting in that it is piecewise constant and monotonic non-decreasing up to the threshold κ_i(T_i−1). Arguably, the optimal allocation at earlier stages will also exhibit a piecewise constant and monotonic non-decreasing behavior. The difficulty is in computing the switching points where the allocation increases by 1. When resources are abundant, we have the complete solution. The interesting case is when resources are scarce where it is not obvious how to distribute them among the different stages knowing that future stages may not come into play. Nonetheless, one can still generalize some of the results that have been shown to be true for stage T_i−1. Indeed, the optimal value function for earlier stages generalizes (26) and has the form: for >0 and k≤

$\begin{array}{cc}{\sum }_{h=0}^{\ell }{\kappa }_i\left(T_i-h\right),& \left(31\right)\end{array}$ $V_i\left(k,T_i-\ell \right)=-c\mu \left(k,T_i-\ell \right)+\left[1-q^{\mu \left(k,T_i-\ell \right)}\right]r_i+q^{\mu \left(k,T_i-\ell \right)}\left\{-c\mu \left({\tilde{k}}_1,T_i-\ell +1\right)+\left[1-q^{\mu \left({\tilde{k}}_1,T_i-\ell +1\right)}\right]r_i+q^{\mu \left({\tilde{k}}_1,T_i-\ell +1\right)}[\dots ]\right\}=-c\mu \left(k,T_i-\ell \right)+\left[1-q^k\right]r_i-c\sum _{j=1}^{\ell }{q}^{{\sum }_{n=0}^{j-1}\mu \left({\tilde{k}}_n,T_i-\ell +n\right)}\mu \left({\tilde{k}}_j,T_i-\ell +j\right),$ ${\tilde{k}}_j={\tilde{k}}_{j-1}-\mu \left({\tilde{k}}_{j-1},T_i-\ell +j-1\right),j=1,\dots ,\ell ,\mathrm{and}{\tilde{k}}_0=k.$
The above value function reflects the fact that the DM incurs a cost of c×μ({tilde over (k)}_j, T_i−+j) only if the resources allocated at all previous stages T_i−+n, n ∈ [0, j−1] are unsuccessful in completing the task. Moreover, the optimal allocation at stage T_i−+j is a function of the number of resources left at the stage, , which, by definition, is the number of resources allocated at previous stages deducted from the initial inventory of k resources. Since all k resources are allocated, the expected reward equals [1−q^k]r_i. The exact distribution of the k resources amongst the different stages requires solving the Bellman recursion. For the special case of k=κ_i(T_i−h), the result is immediate and the optimal allocation, μ({tilde over (k)}_j, T_i−+j)=κ_i(T_i−+j), j ∈ [0, ].

From (31), we can also generalize the marginal reward function. Suppose u resources are allocated at stage T_i− and optimal allocations are made thereafter. We can write:

$\begin{array}{cc}{W}_i\left(k,u,T_i-\ell \right)=-\mathrm{cu}+\left[1-q^k\right]r_i-{\mathrm{cq}}^u\left[\mu \left({\tilde{k}}_1,T_i-\ell +1\right)+\sum _{j=2}^{\ell }{q}^{{\sum }_{n=1}^{j-1}\mu \left({\tilde{k}}_n,T_i-\ell +n\right)}\mu \left({\tilde{k}}_j,T_i-\ell +j\right)\right],& \left(32\right)\end{array}$ ${\tilde{k}}_j={\tilde{k}}_{j-1}-\mu \left({\tilde{k}}_{j-1},T_i-\ell +j-1\right),j=2,\dots ,\ell ,\mathrm{and}{\tilde{k}}_1=k-u.$
It follows that the marginal reward at stage T_i− is given by:

$\begin{array}{cc}\begin{array}{c}{\Delta }_{T_i-\ell }^i\left(k,u\right)=W_i\left(k,u+1,T_i-\ell \right)-W_i\left(k,u,T_i-\ell \right)\\ =-c+{\mathrm{cq}}^u\{\mu \left(k-u,T_i-\ell +1\right)-q\mu \left(k-u-1,T_i-\ell +1\right)\\ +{\sum }_{j=2}^{\ell }{q}^{{\sum }_{n=1}^{j-1}\mu \left({\tilde{k}}_n,T_i-\ell +n\right)}\mu \left({\tilde{k}}_j,T_i-\ell +j\right)\\ -q{\sum }_{j=2}^{\ell }{q}^{{\sum }_{n=1}^{j-1}\mu \left({\stackrel{\_}{k}}_n,T_i-\ell +n\right)}\mu \left({\stackrel{\_}{k}}_j,T_i-\ell +j\right)\},\end{array}& \left(33\right)\end{array}$ $\begin{array}{cc}{\stackrel{\_}{k}}_j={\stackrel{\_}{k}}_{j-1}-\mu \left({\stackrel{\_}{k}}_{j-1},T_i-\ell +j-1\right),j=2,\dots ,\ell ,\mathrm{and}{\stackrel{\_}{k}}_1=k-u-1.& \left(34\right)\end{array}$
Let (k,u)=−c+cq^u(k−u), where (k−u) represents the quantity inside the curly brackets in (33). Assuming that the unit incremental property of the optimal allocation (see Lemma 2) holds for all stages, we have either μ({tilde over (k)}₁, T_i−+1)=μ({tilde over (k)}₁−1, T_i−+1) or μ({tilde over (k)}₁, T_i−+1)=μ({tilde over (k)}₁−1, T_i−+1)+1. Accordingly,

• • 1) if μ({tilde over (k)}₁, T_i−+1)=μ({tilde over (k)}₁−1, T_i−+1), we have:

$\begin{array}{cc}{f}_{T_i-\ell }\left(k-u\right)=p\mu \left({\tilde{k}}_1,T_i-\ell +1\right)& \left(35\right)\\ +q^{\mu \left({\tilde{k}}_1,T_i-\ell +1\right)}[\sum _{j=2}^{\ell }{q}^{{\sum }_{n=2}^{j-1}\mu \left({\tilde{k}}_n,T_i-\ell +n\right)}\mu \left({\tilde{k}}_j,T_i-\ell +j\right)& \\ -q{\sum }_{j=2}^{\ell }{q}^{{\sum }_{n=2}^{j-1}\mu \left({\stackrel{\_}{k}}_n,T_i-\ell +n\right)}\mu \left({\stackrel{\_}{k}}_j,T_i-\ell +j\right)]& \\ =p\mu \left({\tilde{k}}_1,T_i-\ell +1\right)+q^{\mu \left({\tilde{k}}_1,T_i-\ell +1\right)}{f}_{T_i-\ell +1}\left({\tilde{k}}_1-\mu \left({\tilde{k}}_1,T_i-\ell +1\right)\right),& \left(36\right)\end{array}$
where we recognize that the expression inside the square brackets in (35) is, by definition, f_Ti1 ({tilde over (k)}₁−μ({tilde over (k)}₁, T_i−+1)).

• • if μ({tilde over (k)}₁, T_i−+1)=μ({tilde over (k)}₁−1, T_i−+1)+1, we have: k₁={tilde over (k)}₁−1 and k_j={tilde over (k)}_j, ∀j≥2. It follows that,
    (k−u)=pμ({tilde over (k)}₁,T_i−+1)+q.  (37)

In addition to the unit incremental property, further suppose (as in Lemma 2)), that:
μ({tilde over (k)}₁−1,T_i−+1)=μ({tilde over (k)}₁,T_i−+1)−2) if ({tilde over (k)}₁,μ({tilde over (k)}₁−T_i−+1))≤q.  (38)
Combining (36), (37) and (38), we have:
(w)=pμ(w,T₁−+1)+max{q(w−μ(w,T_i−+1))}.  (39)
Recall that the optimal allocation at the last stage, μ(k,T_i)=k, k≤κ_i(T_i). So, we have the initial condition for the update given by:
f_Ti−1(w)=pμ(w,T_i)+q=pw+q,  (40)
which matches with our earlier definition for f_Ti−1(w). Finally, we can extend the optimal allocation result to all stages by generalizing Theorem 1.

Theorem 2 The optimal allocation policy at T_i− for any >0 is dictated by:

$\begin{array}{cc}{\mu }_i\left(k,T_i-\ell \right)=\{\begin{array}{cc}{u}^{*}\left(k\right),& k\le {\sum }_{h=0}^{\ell }{\kappa }_i\left(T_i-h\right),\\ {\kappa }_i\left(T_i-\ell \right),& k>{\sum }_{h=0}^{\ell }{\kappa }_i\left(T_i-h\right),\end{array}& \left(41\right)\end{array}$ $\mathrm{where}{u}^{*}\left({\sum }_{h=0}^{\ell }{\kappa }_i\left(T_i-h\right)\right)={\kappa }_i\left(T_i-\ell \right)\mathrm{and}\mathrm{for}1<k\le {\sum }_{h=0}^{\ell }{\kappa }_i\left(T_i-h\right),$ $\begin{array}{cc}{u}^{*}\left(k-1\right)=\{\begin{array}{cc}{u}^{*}\left(k\right)-1,& \mathrm{if}{q}^{u^{*}\left(k\right)}{f}_{T_i-\ell }\left(k-u^{*}\left(k\right)\right)\le q,\\ u^{*}\left(k\right),& \mathrm{otherwise}.\end{array}& \left(42\right)\end{array}$
The update function for (.) is given by (39) with the initial condition (40).

To prove the above result, we have to generalize the results shown for f_Ti−1(.) to the general case of (.) for any >1. Recall the optimal policy at the terminal time is given by:

$\begin{array}{cc}{\mu }_i\left(k,T_i\right)=\{\begin{array}{cc}k,& k<{\kappa }_i\left(T_i\right),\\ {\kappa }_i\left(T_i\right),& k\ge {\kappa }_i\left(T_i\right).\end{array}& \left(43\right)\end{array}$

Multiple Temporally Disjoint Tasks

We return our attention to the multiple task arrivals. The ensuing analysis is greatly simplified when the arriving tasks are temporally disjoint. In other words, suppose the time windows [t_sⁱ,t_fⁱ] for different tasks are completely disjoint. Without loss of generality, we can assume that the N tasks are ordered temporally as follows: 1=t_s¹<t_f¹<t_s²<t_f²< . . . <t_s^N<t_f^N=T. Furthermore, we can also assume that t_sⁱ⁺¹=t_fⁱ+1 for all i<N since the DM cannot allocate anything between two successive disjoint windows of opportunity.

Concurrent Tasks

This case deals with the scenario wherein there are two or more tasks such that the intersection of their windows of opportunity is non empty. The curse of dimensionality may render the dynamic programming recursion intractable. However, we can use the notion of marginal reward from the single task case to establish heuristic policies. The idea would be to assign resources greedily to tasks that yield the maximal marginal reward.

Random Task Arrivals

This would be the most complex scenario, wherein the windows of opportunity are no longer fixed. In other words, suppose the length of each task window T_i is known, but the start time is random. The original recursion has to be modified to account for random arrivals. For example, we could assume that each time step task of type i would appear with probability α_i. Furthermore, we could assume that once a task arrives, no other task can arrive until either the current task is completed or its window of opportunity expires.

The allocation of resources is described in more detail in the following paragraphs. According to various implementations, an asset performs a sequence of identical missions at time, t=1, 2, . . . , T and is brought into service in between missions. The asset is comprised of m independent sub-systems connected in series with each sub-system i comprising n_i independent, identical constant failure rate (CFR) components connected in parallel. At any point in time, a component is either functioning or has failed. A sub-system is also either functioning (if at least one of its components is functioning) or has failed. The asset is functioning and can perform a mission successfully only if all of its sub-systems are functioning. A faulty component can be repaired during the break between two consecutive missions. We assume that each component in sub-system i has reliability r_i i.e., a functioning component at the start of a mission fails during the mission with probability q_i=1−r_i. In this example, each component requires exactly 1 man hour to be repaired. The decision maker (DM) has at his disposal a total budget of M man hours over the entire planning horizon. There is a unit cost c>0 borne by the DM for each man hour expended. We assume “complete information” in that each component's status is known to the DM at the end of a mission (or beginning of a break). We wish to maximize the asset reliability while minimizing cumulative service cost over the remainder of the planning horizon. At the end of mission t, suppose s_i≤n_i components are found to be faulty in sub-system i and the DM allocates u_i≤s_i man-hours for repairing subsystem i. The reliability of subsystem i for mission t+1 is given by: 1−q_i^ni−si+ui. In other words, this is the probability that not all functioning components in subsystem i fail during the (t+1)^th mission. Correspondingly, the reliability of the asset for mission t+1 is given by:

${\Pi }_{i=1}^m\left[1-q_i^{n_i-s_i+u_i}\right].$

Single Mission Allocation

Suppose we have a budget of k man-hours and a single mission for which we wish to maximize asset reliability and minimize resource use (cost). As before, let s_i≤n_i denote the number of faulty components before the start of the mission. Let us allocate u_i≤s_i man hours towards repairing components in sub-system i. The optimization problem becomes:

$\begin{array}{cc}\underset{u}{\max }\left\{{\prod }_{i=1}^m\left[1-q_i^{n_i-s_i+u_i}\right]-c{\sum }_{i=1}^m{u}_i\right\}& \left(44\right)\end{array}$ $\mathrm{subject}\mathrm{to}$ ${\sum }_{i=1}^m{u}_i\le k.$
For this single stage optimization problem, it can be shown that the optimal allocation is given by the Maximal Marginal Reward (MMR) algorithm. Indeed, we assign one man-hour at a time to the sub-system that yields the highest marginal reward for the additional allocation. Suppose we have already assigned u_i man-hours to sub-system i. The marginal reward i.e., increase in asset reliability yielded by assigning an additional man hour to sub-system i minus the additional cost is given by:

$\begin{array}{cc}{\Delta }_i\left(u\right)=r_i{q}_i^{n_i-s_i+u_i}{\Pi }_{j=1;j\ne i}^m\left[1-q_j^{n_j-s_j+u_j}\right]-c.& \left(45\right)\end{array}$
We assign the next available man hour to the sub-system with the highest payoff: arg max_i[Δ_i(u)]. We stop assigning resources when this quantity becomes negative indicating that any additional allocation yields a negative (marginal) reward.

Multi-Mission Allocation

Suppose at decision time t, the DM is left with k man-hours and the number of faulty components in sub-system i is given by s_i and the state vector, s=(s₁, . . . , s_m). Let the DM allocate u_i≤s_i man-hours towards repairing components in sub-system i. The corresponding expected future payoff or value function is given by:

$\begin{array}{cc}V\left(k,s,t\right)=\underset{u_i;{\sum }_{i=1}^m{u}_i\le k}{\max }\left\{\prod _{i=1}^m\left[1-q_i^{n_i-s_i+u_i}\right]-c\sum _{i=1}^m{u}_i+\sum _{\bar{s}}\mathrm{Prob}\left(\stackrel{\_}{s}\right)V\left(k-\sum _{i=1}^m{u}_i,\stackrel{\_}{s},t+1\right)\right\},& \left(46\right)\end{array}$ $s_i=1,\dots ,n_i,\forall i,$
where the number of failed components in sub-system i at the end of mission t+1 is given by s_i=s_i−u_i+z_i. Note that z_i is a binomial random variable with n_i−s_i+u_i individual and identical Bernoulli trials having probability of success q_i, i.e., z_i˜B(n_i−s_i+u_i, q_i). Suppose we wish to assign an additional resource (man-hour) to sub-system i. As before, the immediate marginal reward yielded by this allocation is given by:

$\begin{array}{cc}{\Delta }_i\left(u\right)=r_i{q}_i^{n_i-s_i+u_i}{\Pi }_{j=1;j\ne i}^m\left[1-q_j^{n_j-s_j+u_j}\right]-c.& \left(47\right)\end{array}$
On the other hand, if the additional resource is kept in reserve for future stages (breaks), the expected marginal future reward is given by:

$\begin{array}{cc}W\left(k-{\sum }_{i=1}^m{u}_i,t+1\right)-W\left(k-{\sum }_{i=1}^m{u}_i-1,t+1\right),& \left(48\right)\end{array}$
where:
W(k,t+1)=Σ_sProb(s)V(k,s,t+1).  (49)

Single Sub-System Multi-Mission Allocation

Suppose we are only interested in the optimal allocation of resources to a single sub-system i. Recall decisions are made at t=1, . . . , T before the start of the t^th mission. We assume that all components are healthy before the start of the 1^st mission. Suppose at decision time t, the DM is left with k man-hours and the number of faulty components is given by s. We wish to minimize the cost of labor and maximize the number of successful missions. The unit cost c>0 can be appropriately chosen to trade off the cost of labor with mission success probability. The corresponding sub-system specific value function is given by:

$\begin{array}{cc}{V}_i\left(k,s,t\right)=\underset{u\in \left[0,\min \left(k,s\right)\right]}{\max }\left\{R_i\left(s,u\right)+\underset{\overline{s}=s-u}{\sum ^{n_i}}\mathrm{Prob}\left(\overline{s}\right)V_i\left(k-u,\overline{s},t+1\right)\right\},& \left(50\right)\end{array}$ $k=1,\dots ,M,s=1,\dots ,n_i,\forall t\in \left[1,T-1\right],$
where where the immediate reward associated with allocation u is given by R_i(s,u)=1−q_i^ni−s+u−cu and the number of failed components at the end of mission t+1 is given by s=s−u+z_i. Note that z_i is a binomial random variable with n_i−s+u individual and identical Bernoulli trials having probability of success q_i, i.e., z_i˜B(n_i−s+u,q_i). Let the binomial mass distribution function be denoted by

$b\left(y,\ell ,q_i\right)=\left(\begin{array}{c}y\\ \ell \end{array}\right)q_i^{\ell }{r}_i^{y-}.$
So, the Bellman recursion becomes:

$\begin{array}{cc}{V}_i\left(k,s,t\right)=\underset{u\in \left[0,\min \left(k,s\right)\right]}{\max }\left\{R_i\left(s,u\right)+\sum _{\ell =0}^{n_i-s+u}b\left(n_i-s+u,\ell ,q_i\right)V_i\left(k-u,s-u+,t+1\right)\right\},& \left(51\right)\end{array}$ $k=1,\dots ,M,s=1,\dots ,n_i,\forall t\in \left[1,T-1\right].$
The optimal policy is given by:

$\begin{array}{cc}{\mu }_i\left(k,s,t\right)=\arg {\max }_{u\in \left[0,\min \left(k,s\right)\right]}\left\{R_i\left(s,u\right)+\sum _{\ell =0}^{n_i-s+u}b\left(n_i-s+u,\ell ,q_i\right)V_i\left(k-u,s-u+,t+1\right)\right\},& \left(52\right)\end{array}$ $k=1,\dots ,M,s=1,\dots ,n_i,\forall t\in \left[1,T-1\right].$
At the terminal decision epoch, we have the boundary condition:

$\begin{array}{cc}{V}_i\left(k,s,T\right)=\underset{u\in \left[0,\min \left(k,s\right)\right]}{\max }{R}_i\left(s,u\right),k=1,\dots ,M,s=1,\dots ,n_i.& \left(53\right)\end{array}$
When there are no man hours left, the system evolves autonomously and we have:

$\begin{array}{cc}{V}_i\left(0,s,t\right)=1-q_i^{n_i-s}+b\left(n_i-s,,q_i\right)V_i\left(0,s+,t+1\right),s=1,\dots ,n_i,t\in \left[1,T-1\right].& \left(54\right)\end{array}$

Lemma 1

$\begin{array}{cc}{V}_i\left(0,s,T-h+1\right)=h-{\sum }_{j=1}^h{\left(1-r_i^j\right)}^{n_i-s},s=1,\dots ,n_i,\forall h\in \left[1,T\right].& \left(55\right)\end{array}$

Proof. From the boundary condition (53), we have:
V_i(0,s,T)=1−q_i^ni−s, s=1, . . . ,n_i.  (56)
We make the induction assumption that:

$\begin{array}{cc}{V}_i\left(0,s,T-h+1\right)=h-{\sum }_{j=1}^h{\left(1-r_i^j\right)}^{n_i-s},s=1,\dots ,n_i.& \left(57\right)\end{array}$
for some h>1. It follows that:

$\begin{array}{cc}{V}_i\left(0,s,T-h\right)=1-q_i^{n_i-s}+\sum _{\ell =0}^{n_i-s}b\left(n_i-s,\ell ,q_i\right)V_i\left(0,s+,T-h+1\right)& \\ =1-q_i^{n_i-s}+{\sum }_{\ell =0}^{n_i-s}\left(\begin{array}{c}{n}_i-s\\ \ell \end{array}\right)q_i^{\ell }{r}_i^{n_i-s-\ell }\left[h-{\sum }_{j=1}^h{\left(1-r_i^j\right)}^{n_i-s-\ell }\right]& \left(58\right)\\ =h+1-q_i^{n_i-s}-q_i^{n_i-s}{\sum }_{\ell =0}^{n_i-s}\left(\begin{array}{c}{n}_i-s\\ \ell \end{array}\right)r_i^{n_i-s-\ell }{\sum }_{j=1}^h{\left(\frac{1-r_i^j}{q_i}\right)}^{n_i-s-\ell }& \\ =h+1-q_i^{n_i-s}-q_i^{n_i-s}{\sum }_{\ell =0}^{n_i-s}\left(\begin{array}{c}{n}_i-s\\ \ell \end{array}\right)r_i^{n_i-s-\ell }{\sum }_{j=1}^h{\left({\sum }_{a=0}^{j-1}{r}_i^a\right)}^{n_i-s-\ell }& \\ =h+1-q_i^{n_i-s}\left[1+{\sum }_{j=1}^h{\sum }_{\ell =0}^{n_i-s}\left(\begin{array}{c}{n}_i-s\\ \end{array}\right){\left({\sum }_{a=1}^j{r}_i^a\right)}^{n_i-s-\ell }\right]& \left(59\right)\\ =h+1-q_i^{n_i-s}\left[1+{\sum }_{j=1}^h{\left(1+{\sum }_{a=1}^j{r}_i^a\right)}^{n_i-s}\right]& \\ =h+1-q_i^{n_i-s}\left[1+{\sum }_{j=1}^h{\left({\sum }_{a=0}^j{r}_i^a\right)}^{n_i-s}\right]& \\ =h+1-{\sum }_{j=1}^{h+1}{\left(1-r_i^j\right)}^{n_i-s}.& \left(60\right)\end{array}$
We have used the induction assumption in (58) and repeatedly used

$q_i=1-r_i\mathrm{and}1-r_i^b=q_i{\Sigma }_{j=0}^{b-1}{r}_i$
to arrive at (60).

At the terminal decision epoch T, if there are s faulty components, we will at most need only s man hours to repair them. So, the optimal allocation for k>s is the same as the optimal allocation for k=s. With this in mind, recall the boundary condition:

$\begin{array}{cc}{V}_i\left(k,s,T\right)=\underset{u\in \left[0,\min \left(k,s\right)\right]}{\max }{R}_i\left(s,u\right),s=1,\dots ,n_i.& \left(61\right)\end{array}$

Suppose the DM has u+1 resources left at the terminal time T. We denote the marginal reward yielded by assigning 1 additional resource over and above u resources to the active task i as:
Δ_Tⁱ(u)=R_i(s,u+1)−R_i(s,u)=−c+q_i^ni−s+ur_i.  (62)
Since q_i<1 it follows that Δ_Tⁱ(u) is monotonic decreasing in u. Let κ_i(T) be the least non-negative integer at which the marginal reward becomes negative i.e., κ_i(T)=s.t. r_i<c. It follows that the optimal policy at the terminal decision epoch is given by: for k≤s,

$\begin{array}{cc}{\mu }_i\left(k,s,T\right)=\arg {\max }_{u\in \left[0,k\right]}{R}_i\left(s,u\right)=\{\begin{array}{cc}0,& n_i-s\ge {\kappa }_i\left(T\right),\\ k,& n_i-s+k\le {\kappa }_i\left(T\right),\\ {\kappa }_i\left(T\right)-n_i+s,& \mathrm{othe}\mathrm{rwise}.\end{array}& \left(63\right)\end{array}$
For k>s, the optimal assignment is given by:

$\begin{array}{cc}{\mu }_i\left(k,s,T\right)={\mu }_i\left(s,s,T\right)=\{\begin{array}{cc}0,& n_i-s\ge {\kappa }_i\left(T\right),\\ s,& n_i\le {\kappa }_i\left(T\right),\\ {\kappa }_i\left(T\right)-n_i+s,& \mathrm{othe}\mathrm{rwise}.\end{array}& \left(64\right)\end{array}$
In other words, we assign as many resources as possible so that n_i−s+u gets close to the threshold κ_i(T) without exceeding it. The optimal terminal value is given by:
V_i(k,s,T)=R_i(s,μ_i(k,s,T))=1−q_i^{ni−s+μi(k,s,T)}−cμ_i(k,s,T).  (65)
We will show likewise that at each decision epoch, there is an upper bound to the number of resources that the DM shall assign (optimally) for the remainder of the time horizon. Having computed the terminal value function, we are now in a position to compute the value function at the previous decision epoch. Indeed, we have from (50):

$\begin{array}{cc}{V}_i\left(k,s,T-1\right)=\underset{u\in \left[0,\min \left(k,s\right)\right]}{\max }\left\{R\left(s,u\right)+Q\left(k,s,u,T\right)\right\},& \left(66\right)\end{array}$ $k=1,\dots ,M,s=1,\dots ,n_i,$
where the future expected reward associated with allocation u is given by:

$\begin{array}{cc}Q\left(k,s,u,T\right)=b\left(n_i-s+u,,q_i\right)V_i\left(k-u,s-u+,T\right)& \left(67\right)\end{array}$
We wish to show that the value function V_i(k,s,T−1) has a threshold based analytical form similar to V_i(k,s,T).

Case 1: u<κ_i(T)−n_i+s and k−u≥κ_i(T). It follows that:
n−s+u−<κ_i(T) and n−s+k−≥κ_i(T)∀.  (68)
⇒μ_i(k−u,s−u+,T)=κ_i(T)−n_i+s−u+.  (69)
So, we have:

$\begin{array}{cc}\begin{array}{cc}\begin{array}{c}Q\left(k,s,u,T\right)=\sum _{\ell =0}^{n_i-s+u}b\left(n_i-s+u,\ell ,q_i\right)V_i\left(k-u,s-u+\ell ,T\right)\\ =\sum _{\ell =0}^{n_i-s+u}b\left(n_i-s+u,\ell ,q_i\right)\left[1-c\left({\kappa }_i\left(T\right)-n_i+s-u+\ell \right)-q_i^{{\kappa }_i\left(T\right)}\right]\\ =1-c{\kappa }_i\left(T\right)-q_i^{{\kappa }_i\left(T\right)}+c\sum _{\ell =0}^{n_i-s+u}b\left(n_i-s+u,\ell ,q_i\right)\left(n_i-s+u-\ell \right)\\ =1-c{\kappa }_i\left(T\right)-q_i^{{\kappa }_i\left(T\right)}+c\sum _{\ell =0}^{n_i-s+u}\left(\begin{array}{c}{n}_i-s+u\\ \ell \end{array}\right)q_i^{\ell }{r}_i^{n_i-s+u-\ell }\left(n_i-s+u-\ell \right)\\ =1-c{\kappa }_i\left(T\right)-q_i^{{\kappa }_i\left(T\right)}\mathrm{cr}\left(n_i-s+u\right)\sum _{\ell =0}^{n_i-s+u-1}\left(\begin{array}{c}{n}_i-s+u-1\\ \ell \end{array}\right)q_i^{\ell }{r}_i^{n_i-s+u-1-\ell }\\ =1-c{\kappa }_i\left(T\right)-q_i^{{\kappa }_i\left(T\right)}+cr\left(n_i-s+u\right)\end{array}.& \end{array}& \left(70\right)\end{array}$

Case 2: n−s+k≤κ_i(T). It follows that:
n−s+k−≤κ_i(T)∀.  (71)
⇒μ_i(k−u,s−u+,T)=k−u.  (72)
So, we have:

$\begin{array}{cc}\begin{array}{c}Q\left(k,s,u,T\right)={\sum }_{\ell =0}^{n_i-s+u}b\left(n_i-s+u,,q_i\right)V_i\left(k-u,s-u+\ell ,T\right)\\ ={\sum }_{\ell =0}^{n_i-s+u}b\left(n_i-s+u,\ell ,q_i\right)\left[1-c\left(k-u\right)-q_i^{n_i-s+k-\ell }\right]\\ =1-c\left(k-u\right)-{\sum }_{\ell =0}^{n_i-s+u}b\left(n_i-s+u,\ell ,q_i\right)q_i^{n_i-s+k-\ell }\\ =1-c\left(k-u\right)-{\sum }_{\ell =0}^{n_i-s+u}\left(\begin{array}{c}{n}_i-s+u\\ \ell \end{array}\right)q_i^{\ell }{r}_i^{n_i-s+u-\ell }{q}_i^{n_i-s+k-\ell }\\ =1-c\left(k-u\right)-q_i^{n_i-s+k}{\sum }_{\ell =0}^{n_i-s+u}\left(\begin{array}{c}{n}_i-s+u\\ \ell \end{array}\right)r_i^{n_i-s+u-\ell }\\ =1-c\left(k-u\right)-{q_i^{n_i-s+k}\left(1+r\right)}^{n_i-s+u}\end{array}.& \left(73\right)\end{array}$
It follows that:
R(s,u)+Q(k,s,u,T)=2−ck−q_i^ni−s+u−q_i^ni−s+k(1+r)^ni−s+u.⇒Δ_T−1ⁱ(k,u)=rq_i^ni−s+u[1−q^k−u(1+r)^ni−s+u].  (74)

Unless otherwise indicated, all numbers expressing feature sizes, amounts, and physical properties used in the specification and claims are to be understood as being modified in all instances by the term “about.” Accordingly, unless indicated to the contrary, the numerical parameters set forth in the foregoing specification and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by those skilled in the art utilizing the teachings disclosed herein. The use of numerical ranges by endpoints includes all numbers within that range (e.g. 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.80, 4, and 5) and any range within that range.

The various embodiments described above may be implemented using circuitry and/or software modules that interact to provide particular results. One of skill in the computing arts can readily implement such described functionality, either at a modular level or as a whole, using knowledge generally known in the art. For example, the flowcharts illustrated herein may be used to create computer-readable instructions/code for execution by a processor. Such instructions may be stored on a computer-readable medium and transferred to the processor for execution as is known in the art. The structures and procedures shown above are only a representative example of embodiments that can be used to facilitate ink jet ejector diagnostics as described above.

The foregoing description of the example embodiments have been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the inventive concepts to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. Any or all features of the disclosed embodiments can be applied individually or in any combination, not meant to be limiting but purely illustrative. It is intended that the scope be limited by the claims appended herein and not with the detailed description.

Claims

1. A method comprising:

monitoring a condition of a physical asset comprising one or more subsystems connected in series, each subsystem comprising one or more components connected in parallel, the physical asset having one or more jobs and experiencing one or more shutdowns between consecutive operational periods, wherein monitoring the condition of the physical asset involves determining that the one or more subsystems are functioning properly;

determining a probability of the physical asset surviving a predetermined amount of time based on the monitoring and one or more shared resources, the one or more shared resources configured to be shared between the subsystems;

establishing a model using a threshold based heuristic maintenance policy and known failure rates of the one or more components, the model configured to maximize a number of successful jobs that the physical asset is able to complete based on the determined probability; and

mitigating downtime and a cost associated with shutdowns of the physical asset, including the one or more subsystems and the one or more components of each subsystem, by allocating the one or more shared resources to the one or more subsystems based on the model.

2. The method of claim 1, wherein the one or more jobs comprise one or more substantially identical jobs.

3. The method of claim 1, wherein the model comprises minimizing a shared resource cost.

4. The method of claim 3, wherein the shared resource cost is calculated using a Maximal Marginal Reward (MMR) algorithm that is configured to maximize an expected payout.

5. The method of claim 1, further comprising determining a maintenance schedule based on the model.

6. The method of claim 1, wherein monitoring the condition of the physical asset comprises determining if all subsystems are functioning.

7. The method of claim 1, wherein the one or more components are redundant within each subsystem.

8. The method of claim 1, wherein the shared resources comprise one or more of man hours and parts used to perform maintenance.

9. The method of claim 1, wherein the one or more shared resources comprise one or both of consumable resources and replenishable resources.

10. A system, comprising:

a processor; and

a memory storing computer program instructions which when executed by the processor cause the processor to perform operations comprising:

monitoring a condition of a physical asset comprising one or more subsystems connected in series, each subsystem comprising one or more components connected in parallel, the physical asset having one or more jobs and experiencing one or more shutdowns between consecutive operational periods, wherein monitoring the condition of the physical asset involves determining that the one or more subsystems are functioning properly;

establishing one or more consumable resources that are shared among the one or more subsystems;

determining a probability of the physical asset surviving a predetermined amount of time based on the monitoring and the one or more consumable resources;

establishing a model using a threshold based heuristic maintenance policy and known failure rates of the one or more components, the model configured to maximize a number of successful jobs that the physical asset is able to complete based on the determined probability; and

mitigating downtime and a cost associated with shutdowns of the physical asset, including the one or more subsystems and the one or more components of each subsystem, by allocating the one or more consumable resources to the one or more subsystems based on the model.

11. The system of claim 10, wherein the one or more jobs comprise one or more substantially identical jobs.

12. The system of claim 10, wherein the model comprises minimizing a shared resource cost.

13. The system of claim 12, wherein the shared resource cost is calculated using a Maximal Marginal Reward (MMR) algorithm that is configured to maximize an expected payout.

14. The system of claim 10, wherein the processor is configured to determine a maintenance schedule based on the model.

15. The system of claim 10, wherein monitoring the condition of the physical asset comprises determining if all subsystems are functioning.

16. The system of claim 10, wherein the one or more components are redundant within each subsystem.

17. The system of claim 10, wherein the one or more consumable resources comprise one or more of man hours and parts used to perform maintenance.

18. The system of claim 10, wherein the one or more consumable resources comprise replenishable resources.