({{, (RefLink[ContinuousMarkovProcess, paclet:ref/ContinuousMarkovProcess])[(Subscript[i, 0]),q] 
represents a continuous-time finite-state Markov process with transition rate matrix q and initial state (Subscript[i, 0]).}, {, (RefLink[ContinuousMarkovProcess, paclet:ref/ContinuousMarkovProcess])[(Subscript[p, 0]),q]
represents a Markov process with initial state probability vector (Subscript[p, 0]).}, {, (RefLink[ContinuousMarkovProcess, paclet:ref/ContinuousMarkovProcess])[…,m,μ]
represents a Markov process with transition matrix m and transition rates μ.}, {, (RefLink[ContinuousMarkovProcess, paclet:ref/ContinuousMarkovProcess])[…,g]
represents a Markov process transition rate matrix from the graph g. }})

OpenersNotesSection(Spacer1)Details

(RefLink[ContinuousMarkovProcess, paclet:ref/ContinuousMarkovProcess]) is also known as a continuous-time Markov chain.

(RefLink[ContinuousMarkovProcess, paclet:ref/ContinuousMarkovProcess]) is a continuous-time and discrete-state random process.

The states of (RefLink[ContinuousMarkovProcess, paclet:ref/ContinuousMarkovProcess]) are integers between 1 and n, where n is the length of transition rate matrix q.

For infinitesimal time dt, q[[i,j]]dt gives the probability that the process transitions from state i to state j over the next dt units of time q[[i,j]]dt==(RefLink[Probability, paclet:ref/Probability])[x[t+dt]==j\[Conditioned]x[t]==i].

The time the process stays in state i before transitioning follows (RefLink[ExponentialDistribution, paclet:ref/ExponentialDistribution])[-(Subscript[q, ii])].

The transition matrix m specifies conditional transition probabilities m[[i,j]]==(RefLink[Probability, paclet:ref/Probability])[x[(Subscript[t, k+1])]==j\[Conditioned]x[(Subscript[t, k])]==i], where x[(Subscript[t, k])] is the state at time (Subscript[t, k]), and the transition rate (Subscript[μ, i]) specifies that the time between events in state i follows (RefLink[ExponentialDistribution, paclet:ref/ExponentialDistribution])[(Subscript[μ, i])].

(RefLink[EstimatedProcess, paclet:ref/EstimatedProcess])[data,ContinuousMarkovProcess[n]] indicates that process with n states should be estimated.

The transition matrix in the case of a graph g is constructed to give equal probability of transitioning to each incident vertex with unit transition rates.

(RefLink[ContinuousMarkovProcess, paclet:ref/ContinuousMarkovProcess]) allows q to be an n×n matrix where (Subscript[q, ii])≤0 and (Subscript[q, ij])≥0 for i≠j with rows that sum to 0, (Subscript[i, 0]) can be an integer between 1 and n, (Subscript[p, 0]) is a vector of length n of non-negative elements that sum to 1, m is an n×n matrix with non-negative elements and rows that sum to 1, and μ is a vector of length n with positive elements.

(RefLink[ContinuousMarkovProcess, paclet:ref/ContinuousMarkovProcess]) can be used with such functions as (RefLink[MarkovProcessProperties, paclet:ref/MarkovProcessProperties]), (RefLink[PDF, paclet:ref/PDF]), (RefLink[Probability, paclet:ref/Probability]), and (RefLink[RandomFunction, paclet:ref/RandomFunction]).

OpenersPrimaryExamplesSection(Spacer1)Examples  (28)

OpenersExampleSection0(Spacer1)Basic Examples  (2)

Define a continuous-time Markov process:

𝒫=ContinuousMarkovProcess[{1,0,0},(({{-2, 1, 1}, {1, -2, 1}, {1, 1, -2}}))];

Simulate it:

data=RandomFunction[𝒫,{0,10}]

ListStepPlot[data,PlotRange->{0,3},Ticks->{Automatic,{1,2,3}}]

Find the PDF for the state at time t:

𝒫=ContinuousMarkovProcess[{1,0,0},(({{-2, 1, 1}, {1, -2, 1}, {1, 1, -2}}))];

PDF[𝒫[t],k]//PiecewiseExpand

Find the long-run proportion of time the process is in state 2:

PDF[𝒫[∞],2]

OpenersExampleSection1(Spacer1)Scope  (12)

Process properties:

MarkovProcessProperties[ContinuousMarkovProcess[1,{{-3,1,2,0},{3,-6,2,1},{4,2,-9,3},{0,0,0,0}}]]

Visualize the process as a graph, with the transition rates given as tooltips:

Graph[ContinuousMarkovProcess[1,{{-3,1,2,0},{3,-6,2,1},{4,2,-9,3},{0,0,0,0}}]]

Convert the graph representation back to a continuous Markov process:

ContinuousMarkovProcess[1,%]

Find the recurrent and transient classes for a continuous Markov process:

𝒫=ContinuousMarkovProcess[1,{{-2,1,1,0},{2,-6,2,2},{3,3,-9,3},{0,0,0,0}}];

MarkovProcessProperties[𝒫,"RecurrentClasses"]

MarkovProcessProperties[𝒫,"TransientClasses"]

Generate a sample path with default machine precision:

RandomFunction[ContinuousMarkovProcess[1,{{-3,3},{2,-2}}],{0,2}]//Normal

Generate a sample path with higher precision:

RandomFunction[ContinuousMarkovProcess[1,{{-3,3},{2,-2}}],{0,2},WorkingPrecision-> 20]//Normal

Visualize a sample path:

𝒫=With[{λ=3,μ=2,m=10},ContinuousMarkovProcess[1,SparseArray[{{1,1}->-λ,{m,m}->-μ,Band[{1,2}]->λ,Band[{2,1}]->μ,Band[{2,2}]->-λ-μ},{m,m}]]];

BlockRandom[SeedRandom[1234];data=RandomFunction[𝒫,{0,25}]];

ListPlot[data,Filling->Axis,FillingStyle-> LightGray]

Estimate the process parameters from sample data:

𝒫=ContinuousMarkovProcess[{1,0,0,0},{{-3,1,2,0},{3,-6,2,1},{4,2,-9,3},{0,0,0,0}}];

data=RandomFunction[𝒫,{0,10^2},10^3];

estproc=EstimatedProcess[data,ContinuousMarkovProcess[4],WorkingPrecision->MachinePrecision]//Normal

Slice distribution:

𝒫=With[{λ=3,μ=2,m=10},ContinuousMarkovProcess[1,SparseArray[{{1,1}->-λ,{m,m}->-μ,Band[{1,2}]->λ,Band[{2,1}]->μ,Band[{2,2}]->-λ-μ},{m,m}]]];

PDF[𝒫[4.],3]

DiscretePlot[PDF[𝒫[4.],x],{x,0,10}]

Find the distribution for the time to absorption:

𝒫=ContinuousMarkovProcess[1,{{-2,1,1,0},{1,-6,2,3},{3,2,-8,3},{0,0,0,0}}];

𝒟=FirstPassageTimeDistribution[𝒫,4];

Plot[PDF[𝒟,t],{t,0,10},Evaluated->True,PlotRange->All]

Mean of the distribution:

Mean[𝒟]

N[%]

Compare with the value obtained from process simulation:

paths=RandomFunction[𝒫,{0,20},10^4]["Paths"];

Mean[Part[#,LengthWhile[#,Last[#]≠4&]+1,1]&/@paths]

Characteristic function:

CharacteristicFunction[𝒟,s]

Compute the probability of an event:

Probability[time>Mean[𝒟],time\[Distributed]𝒟]

N[%]

Find the stationary distribution:

StationaryDistribution[ContinuousMarkovProcess[1,{{-a,a},{b,-b}}]]

Perform operations on the stationary distribution:

Mean[%]//Simplify

Find the conditional mean number of total transitions starting in state 1 and ending in state 4:

𝒫=ContinuousMarkovProcess[{1,0,0,0},{{-3,1,2,0},{3,-6,2,1},{4,2,-9,3},{0,0,0,0}}];

n=30.;

MarkovProcessProperties[𝒫,"TransitionCount"][[4]][n]/Probability[x[n]==4,x\[Distributed]𝒫]

Compare with the results from simulation:

data=RandomFunction[𝒫,{0,n},4*10^4,Method->{Automatic,"StoppingFunction"->(#3=!=4&)}]["ValueList"];

N[Mean[Length[#]-1&/@data]]

Find the conditional mean number of transitions from state 2 to state 3:

𝒫=ContinuousMarkovProcess[{1,0,0,0},{{-3,1,2,0},{3,-6,2,1},{4,2,-9,3},{0,0,0,0}}];

MarkovProcessProperties[𝒫,{"TransitionCount",{{2,3}}}][[4]][n]/Probability[x[n]==4,x\[Distributed]𝒫]

Compare with the results from simulation:

data=RandomFunction[𝒫,{0,n},10^4,Method->{Automatic,"StoppingFunction"->(#3=!=4&)}]["ValueList"];

N[Mean[Count[Partition[data[[#]],2,1],{2,3}]&/@Range[Length[data]]]]

Calculate probabilities:

𝒫=ContinuousMarkovProcess[{1,0,0},{{-2,1,1},{1,-2,1},{1,1,-2}}];

Probability[x^2+x<6,x\[Distributed]𝒫[t]]//Simplify

N[%/.t->3]

NProbability[x^2+x<6,x\[Distributed]𝒫[3]]

Moments and generating functions:

𝒫=ContinuousMarkovProcess[{1,0,0},{{-2,1,1},{1,-2,1},{1,1,-2}}];

Moment[𝒫[t],r]//Simplify

CharacteristicFunction[𝒫[t],w]//Simplify

CentralMomentGeneratingFunction[𝒫[t],w]//Simplify

OpenersExampleSection2(Spacer1)Generalizations & Extensions  (2)

Represent a continuous Markov process as a graph:

proc=ContinuousMarkovProcess[{1,0,0,0},{{-3,1,2,0},{3,-6,2,1},{4,2,-9,3},{0,0,0,0}}];

Graph[proc]

Convert a graph to a continuous Markov process:

gr=Graph[{1\[DirectedEdge]2,2\[DirectedEdge]1},Properties->{1->{"InitialProbability"->0.6},2->{"InitialProbability"->0.4},1\[DirectedEdge]2->{"TransitionRate"->3},2\[DirectedEdge]1->{"TransitionRate"->2}}];

MatrixForm/@ContinuousMarkovProcess[gr]

Use a discrete Markov process graph to construct a corresponding continuous Markov process:

gr=Graph[DiscreteMarkovProcess[3,{{1/2,1/2,0,0},{1/2,1/2,0,0},{1/4,1/4,1/4,1/4},{0,0,0,1}}]]

ContinuousMarkovProcess[3,gr,{2,3,5,7}]

OpenersExampleSection4(Spacer1)Applications  (9)

Boston-to-Cambridge migration rate is estimated to be six people per year:

migrationProcess=ContinuousMarkovProcess[1,{{-6,6},{0,0}}];

Find the percentage of people who moved in a period of two months:

Probability[x[t+2/12]==2\[Conditioned]x[t]==1,x\[Distributed]migrationProcess]

N[%]

A hospital owns two identical and independent power generators. The time to breakdown for each is exponential with parameter λ, and the time for repair of a malfunctioning generator is exponential with parameter μ. Assuming both generators were in order at t==0, find the probability that both generators remain functional at time t>0. The number of working generators can be modeled with a continuous-time Markov chain, with the state number representing 1 plus the number of working generators:

proc=ContinuousMarkovProcess[3,{{-2μ,2μ,0},{λ,-λ-μ,μ},{0,2λ,-2λ}}];

Probability[workinggenerators[t]==3,workinggenerators\[Distributed]proc]//FullSimplify

A flammable product is stored in a special tank at a filling station. Customers asking for the product arrive according to a Poisson process with rate λ. Each customer asks for one unit of the product. Any demand that occurs when the tank is out of stock is lost. Opportunities to replenish the stock in the tank occur according to a Poisson process with rate μ. The two Poisson processes are assumed to be independent of each other. For reasons of security, the stock can only be replenished when the tank is empty. At those opportunities, the stock is replenished with n units:

proc=With[{n=5,λ=4,μ=2},ContinuousMarkovProcess[n+1,SparseArray[{{1,1}->-μ,{1,n+1}->μ,{i_,i_}->-λ,{i_,j_}/;i>1&&j==i-1->λ},{n+1,n+1}]]];

proc[Infinity]

Long-run average stock in the tank:

PDF[proc[Infinity],Range[6]].Range[0,5]

Long-run fraction of demand that is lost:

PDF[proc[Infinity],1]

A router receives packets from a group of users and transmits them over a single transmission line. Suppose packets arrive according to a Poisson process at a rate of one packet every 4 milliseconds and packet transmission times are exponentially distributed with mean 3 milliseconds. Approximate an infinite router capacity:

qm=With[{λ=1/4,μ=1/3,m=100},SparseArray[Join[Table[{i,i+1}->λ,{i,m}],Table[{i+1,i}->μ,{i,m}],Table[{i,i}->-λ-μ,{i,2,m}],{{1,1}->-λ,{m+1,m+1}->-μ}]]];

dist=StationaryDistribution[ContinuousMarkovProcess[1,qm]];

Mean delay:

With[{λ=1/4},(Mean[dist]-1)/(λ*(1-PDF[dist,100]))]//N

Probability that there are more than five packets waiting for transmission in the router:

NProbability[k>6,k\[Distributed]dist]

The number of flaws appearing on a polished mirror surface is a Poisson random variable. For a mirror with an area of 8.54` (cm^2), the probability of no flaws is 0.91. Find the probability of no flaws on the surface of another mirror with an area of 17.5` (cm^2), fabricated using the same process:

proc=ContinuousMarkovProcess[1,With[{vec={-λ,λ,0,0}},Append[RotateRight[vec,#]&/@Range[0,Length[vec]-2],ConstantArray[0,Length[vec]]]]];

proc=proc/.First@Quiet@NSolve[Chop@Probability[flaws[8.54]==1,flaws\[Distributed]proc]==0.91,λ,Reals];

NProbability[flaws[17.5]==1,flaws\[Distributed]proc]

The arrival pattern of cars to an oil change center follows a Poisson process at the rate of four per hour. There is a single available mechanic, and the time taken to perform an oil change is exponentially distributed and requires an average of 12 minutes to carry out. Approximate an unlimited car space:

𝒫=With[{λ=4,μ=60/12,m=10^2},ContinuousMarkovProcess[1,SparseArray[Join[Table[{i,i+1}->λ,{i,m}],Table[{i+1,i}->μ,{i,m}],Table[{i,i}->-λ-μ,{i,2,m}],{{1,1}->-λ,{m+1,m+1}->-μ}]]]];

Simulate the number of cars in the queue:

BlockRandom[SeedRandom[1234];data=RandomFunction[𝒫,{0,25}]];

ListPlot[data,Filling->Axis,FillingStyle-> LightGray]

𝒟=StationaryDistribution[𝒫];

The probability that there are more than three cars in the queue:

NProbability[x>4, x\[Distributed]𝒟]

NProbability[x>3, x\[Distributed]StationaryDistribution[QueueingProcess[4,5]]]

Mean and variance for the number of cars in the system:

{Mean[𝒟]-1,Variance[𝒟]}//N

With[{𝒟=StationaryDistribution[QueueingProcess[4,5]]},{Mean[𝒟],Variance[𝒟]}]

Distribution for the number of cars in the system:

DiscretePlot[PDF[𝒟,k],{k,1,20},ExtentSize->0.5,PlotRange->All]

Vehicles in a certain country are required to be assessed every year for roadworthiness. At one vehicle assessment center, drivers wait for an average of 15 minutes before the roadworthiness assessment of their vehicle commences. The assessment takes on average 20 minutes to complete. Following the assessment, 80% of vehicles are passed as roadworthy, allowing the driver to drive home. A further 15% of vehicles are categorized as a "minor fail"; these vehicles require on average 30 minutes of repair work before the driver is allowed to drive home. The remaining 5% of vehicles are categorized as a "significant fail"; these vehicles require on average three hours of repair work before the driver can go home. Model the operation of the vehicle assessment center using a continuous-time Markov model, with states W (waiting for assessment), A (assessment taking place), M (minor repair taking place), S (significant repair taking place), and H (traveling home):

states={"W","A","M","S","H"};

Transition rates:

transrates={statetransition["W","A"]->1/15,Sequence@@Thread[statetransition["A",#]&/@{"M","S","H"}->1/20*{15/100,5/100,80/100}],statetransition["M","H"]->1/30,statetransition["S","H"]->1/(3*60)};

Make the row sums zero by setting diagonals to the negative sum of off-diagonal elements:

qm=SparseArray[Join[transrates,statetransition[#,#]->-Total[Cases[transrates,(statetransition[#,_]->r_):>r]]&/@states]/.Thread[states->Range[5]]/.statetransition->List,{5,5}];

Process for modeling assessment and repair:

proc=ContinuousMarkovProcess[1,qm];

Probability that a vehicle is undergoing significant repair at time t==2.5:

Probability[x[5/2]==4,x\[Distributed]proc]//Simplify

N[%]

Average time in minutes spent by vehicles in assessment and repair:

Mean[FirstPassageTimeDistribution[proc,5]]//N

Consider a system with exponential interarrival and service times. There is one server and room for two customers: the one in service and another that is waiting. Let the number of customers in the room at time t be X(t). This is an M/M/1/2 queue; model it as a birth-death process with the following transition rate matrix, where the states are indexed as X(t)+1:

qm={{-λ,λ,0},{μ,-λ-μ,λ},{0,μ,-μ}};

proc=ContinuousMarkovProcess[1,qm];

Graph[proc,GraphLayout->Automatic,ImageSize->Small]

The transition rate matrix is irreducible:

MarkovProcessProperties[proc,"Irreducible"]

Hence the stationary distribution exists uniquely:

dist=StationaryDistribution[proc]

This agrees with the stationary distribution for (RefLink[QueueingProcess, paclet:ref/QueueingProcess]):

PDF[dist,Range[3]]==PDF[StationaryDistribution[QueueingProcess[λ,μ,1,2]],Range[0,2]]//Simplify

It is a truncated geometric distribution:

PDF[dist,Range[3]]-Table[(1-λ/μ)/(1-(λ/μ)^(n+1))(λ/μ)^j,{j,0,2}]/.n->2//Simplify

Fraction of time the server is busy:

utilfactor=1-PDF[dist,1]//Together

This agrees with the result from (RefLink[QueueProperties, paclet:ref/QueueProperties]):

QueueProperties[QueueingProcess[λ,μ,1,2],"UtilizationFactor"]

Average number of customers in the room in the long run:

avcust=Mean[dist]-1//Simplify

QueueProperties[QueueingProcess[λ,μ,1,2],"MeanSystemSize"]//Simplify

Average size of the queue:

queuesize=PDF[dist,3]//Simplify

QueueProperties[QueueingProcess[λ,μ,1,2],"MeanQueueSize"]//Simplify

avcust-queuesize-utilfactor//Simplify

Proportion of potential customers that enter the room in the long run:

CDF[dist,2]//Simplify

QueueProperties[QueueingProcess[λ,μ,1,2],"Throughput"]/λ//Simplify

Average waiting time:

avcust/(λ*(1-PDF[dist,3]))//Together

QueueProperties[QueueingProcess[λ,μ,1,2],"MeanSystemTime"]//Simplify

Increase in number of customers if the server works twice as fast:

(Mean[StationaryDistribution[ContinuousMarkovProcess[1,qm/.μ->2μ]]]-1)/avcust//Simplify

The Hubble space telescope carries six gyroscopes, with a minimum of three required for full accuracy. The operating times of the gyroscopes are independent and exponentially distributed with failure rate λ. If a fourth gyroscope fails, the telescope goes into sleep mode, in which further observations are suspended. It requires an exponential time with mean 1/μ to put the telescope into sleep mode, after which the base station on Earth receives a sleep signal, and a shuttle mission is prepared. It takes an exponential time with mean 1/η before the repair crew arrives at the telescope and repairs the stabilizing unit with the gyroscopes. In the meantime, the other two gyroscopes may fail. If the last gyroscope fails, the telescope will crash. Suppose λ==0.1, μ==100, and η==5, all with units of inverse years:

proc=ContinuousMarkovProcess[1,{{-6λ,6λ,0,0,0,0,0,0,0},{0,-5λ,5λ,0,0,0,0,0,0},{0,0,-4λ,4λ,0,0,0,0,0},{0,0,0,-3λ,3λ,0,0,0,0},{0,0,0,0,-2λ-μ,2λ,μ,0,0},{0,0,0,0,0,-λ-μ,0,μ,λ},{η,0,0,0,0,0,-2λ-η,2λ,0},{η,0,0,0,0,0,0,-λ-η,λ},{ν,0,0,0,0,0,0,0,-ν}}]/.{λ->1/10,μ->100,η->5};

Visualize the process, with integers representing the number of operational gyroscopes:

Graph[MapThread[Style, {{"6","5","4","3","2","1","Sleep2","Sleep1","Crash"}, {Green,Green,Green,Green,Orange,Orange,LightBlue,LightBlue,Red}}],proc/.{ν->0},GraphLayout->{"SpringElectricalEmbedding","Rotation"->Pi},ImageSize->Medium]

Find the probability that the telescope will crash in the next 10 years:

CDF[FirstPassageTimeDistribution[proc,9],10.]

Find the probability that sleep mode is not reached (no shuttle mission is required) in 10 years:

SurvivalFunction[FirstPassageTimeDistribution[proc,{7,8}],10]/.ν->1.

Plot[SurvivalFunction[FirstPassageTimeDistribution[proc/. ν->1,{7,8}],x],{x,0,20},Filling->Axis]

OpenersExampleSection5(Spacer1)Properties & Relations  (1)

Define a continuous Markov process using the transition rate matrix:

𝒫=ContinuousMarkovProcess[{1,0,0,0},{{-3,1,2,0},{3,-6,2,1},{4,2,-9,3},{0,0,0,0}}];

Obtain the transition rate vector:

μ=MarkovProcessProperties[𝒫,"TransitionRateVector"]

Transition matrix for the embedded discrete Markov process:

m=MarkovProcessProperties[𝒫,"TransitionMatrix"]

Recover the original Markov process using m and μ:

ContinuousMarkovProcess[{1,0,0,0},m,μ]

%===𝒫

OpenersExampleSection6(Spacer1)Possible Issues  (2)

If the transition matrix rows do not sum to zero, the matrix diagonal is adjusted accordingly:

ContinuousMarkovProcess[1,{{-1,2},{4,-3}}]

If the initial probabilities do not sum to 1, they are normalized:

ContinuousMarkovProcess[{1,2},{{-2,2},{3,-3}}]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[DiscreteMarkovProcess, paclet:ref/DiscreteMarkovProcess])  ▪  (RefLink[Graph, paclet:ref/Graph])  ▪  (RefLink[MarkovProcessProperties, paclet:ref/MarkovProcessProperties])  ▪  (RefLink[FirstPassageTimeDistribution, paclet:ref/FirstPassageTimeDistribution])  ▪  (RefLink[QueueingProcess, paclet:ref/QueueingProcess])  ▪  (RefLink[PoissonProcess, paclet:ref/PoissonProcess])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Finite Markov Processes, paclet:guide/MarkovProcesses])}, {▪(RefLinkPlain[Random Processes, paclet:guide/RandomProcesses])}, {▪(RefLinkPlain[Queueing Processes, paclet:guide/QueueingProcesses])}})}})