Transient Exception Handling Circuit Breaker Pattern and Polly

Circuit Breaker pattern is a rather interesting pattern which was discussed in the Release It book. This pattern provides a mechanism to wrap “unpredictable code” i.e. calls to external services within a fabric that manages the execution of the actual code and takes certain actions if a certain pre-defined failure rate has been reached or a particular condition is met (for e.g. DividByZeroException happens)

The circuit breaker pattern has three states

1. Closed State – When the pattern is in closed state each call to the underlying resource is allowed or in other words the risky code is executed as it normally would. However on each failure an internal counter is incremembted and once the count hits a certain threshold the pattern moves to the Open state.
2. Open State – In this state the pattern sets an internal timer to elapse at a configured value. Until such time the timeout is reached no calls to the underlying resource are allowed. In some implementations of the pattern and as it was originally discussed dring this time the pattern throws an exception to indiciate that it is not allowing calls at this moment. I personally do not see much merit in this approach. Once the timeout is elapsed the pattern moves to Half Open state.
3. Half Open State – In this state the pattern allows one call to the underlying resource or the risky code. If this call succeeds the pattern immediately switches back to Closed state however if the call fails the pattern resets the timer and moves to Open state.

An open source project titled Polly brings this pattern beautifully to the .NET world along with a really structured way to allow developers to express transient exception handling policies such as Retry, Retry Forver, Circuit Breaker etc . Polly has complete async, await support. In a nutshell the approach constitutes of setting up “Policies” in the calling code where the code configures the Polly framework on how the developer wants to handle a particular situation if it happens. Next an instance of this policy is created and it is used to execute the code. Here are some details (taken from the Github page directly)

Step 1 : Specify the type of exceptions you want the policy to handle

 // Single exception type
 Policy
 .Handle<DivideByZeroException>()
// Single exception type with condition
 Policy
 .Handle<SqlException>(ex => ex.Number == 1205)
// Multiple exception types
 Policy
 .Handle<DivideByZeroException>()
 .Or<ArgumentException>()
// Multiple exception types with condition
Policy
.Handle<SqlException>(ex => ex.Number == 1205)
.Or<ArgumentException>(ex => x.ParamName == "example")

Step 2 : Specify how the policy should handle those exceptions

Retry

// Retry once
Policy
.Handle<DivideByZeroException>()
.Retry()
// Retry multiple times
Policy
.Handle<DivideByZeroException>()
.Retry(3)

// Retry multiple times, calling an action on each retry
// with the current exception and retry count
Policy
.Handle<DivideByZeroException>()
.Retry(3, (exception, retryCount) =>
{
// do something
});

// Retry multiple times, calling an action on each retry
// with the current exception, retry count and context
// provided to Execute()
Policy
.Handle<DivideByZeroException>()
.Retry(3, (exception, retryCount, context) =>
{
// do something
});

Retry forever

// Retry forever
Policy
.Handle<DivideByZeroException>()
.RetryForever()

// Retry forever, calling an action on each retry with the
// current exception
Policy
.Handle<DivideByZeroException>()
.RetryForever(exception =>
{
// do something
});

// Retry forever, calling an action on each retry with the
// current exception and context provided to Execute()
Policy
.Handle<DivideByZeroException>()
.RetryForever((exception, context) =>
{
// do something
});

Retry and Wait

// Retry, waiting a specified duration between each retry
Policy
.Handle<DivideByZeroException>()
.WaitAndRetry(new[]
{
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(3)
});

// Retry, waiting a specified duration between each retry,
// calling an action on each retry with the current exception
// and duration
Policy
.Handle<DivideByZeroException>()
.WaitAndRetry(new[]
{
1.Seconds(),
2.Seconds(),
3.Seconds()
}, (exception, timeSpan) => {
// do something
});

// Retry, waiting a specified duration between each retry,
// calling an action on each retry with the current exception,
// duration and context provided to Execute()
Policy
.Handle<DivideByZeroException>()
.WaitAndRetry(new[]
{
1.Seconds(),
2.Seconds(),
3.Seconds()
}, (exception, timeSpan, context) => {
// do something
});

// Retry a specified number of times, using a function to
// calculate the duration to wait between retries based on
// the current retry attempt (allows for exponential backoff)
// In this case will wait for
// 2 ^ 1 = 2 seconds then
// 2 ^ 2 = 4 seconds then
// 2 ^ 3 = 8 seconds then
// 2 ^ 4 = 16 seconds then
// 2 ^ 5 = 32 seconds
Policy
.Handle<DivideByZeroException>()
.WaitAndRetry(5, retryAttempt =>
TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))
);

// Retry a specified number of times, using a function to
// calculate the duration to wait between retries based on
// the current retry attempt, calling an action on each retry
// with the current exception, duration and context provided
// to Execute()
Policy
.Handle<DivideByZeroException>()
.WaitAndRetry(
5,
retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
(exception, timeSpan, context) => {
// do something
}
);

Circuit Breaker

// Break the circuit after the specified number of exceptions
// and keep circuit broken for the specified duration
Policy
.Handle<DivideByZeroException>()
.CircuitBreaker(2, TimeSpan.FromMinutes(1));

Step 3 : Execute the policy

// Execute an action
var policy = Policy
.Handle<DivideByZeroException>()
.Retry();

policy.Execute(() => DoSomething());

// Execute an action passing arbitrary context data
var policy = Policy
.Handle<DivideByZeroException>()
.Retry(3, (exception, retryCount, context) =>
{
var methodThatRaisedException = context["methodName"];
Log(exception, methodThatRaisedException);
});

policy.Execute(
() => DoSomething(),
new Dictionary<string, object>() {{ "methodName", "some method" }}
);

// Execute a function returning a result
var policy = Policy
.Handle<DivideByZeroException>()
.Retry();

var result = policy.Execute(() => DoSomething());

// Execute a function returning a result passing arbitrary context data
var policy = Policy
.Handle<DivideByZeroException>()
.Retry(3, (exception, retryCount, context) =>
{
object methodThatRaisedException = context["methodName"];
Log(exception, methodThatRaisedException)
});

var result = policy.Execute(
() => DoSomething(),
new Dictionary<string, object>() {{ "methodName", "some method" }}
);

// You can of course chain it all together
Policy
.Handle<SqlException>(ex => ex.Number == 1205)
.Or<ArgumentException>(ex => ex.ParamName == "example")
.Retry()
.Execute(() => DoSomething());>

Polly – https://github.com/App-vNext/Polly

 

3 Comments

 Add your comment
  1. Hey Nik are you using the Transient Fault Handling Block from Enterprise Libs?

    • Hey Darren Thanks for visiting. No I am not using the enterprise block. This is a library called Polly. You can get this from its GitHub or Nuget page.

  2. Thank you for this article, it helps a lot.

Leave a Comment

Your email address will not be published.