Circuit Breaker pattern is a rather interesting pattern which was discussed in the Release It book. This pattern provides a mechanism to wrap “unpredictable code” i.e. calls to external services within a fabric that manages the execution of the actual code and takes certain actions if a certain pre-defined failure rate has been reached or a particular condition is met (for e.g. DividByZeroException happens)
The circuit breaker pattern has three states
1. Closed State – When the pattern is in closed state each call to the underlying resource is allowed or in other words the risky code is executed as it normally would. However on each failure an internal counter is incremembted and once the count hits a certain threshold the pattern moves to the Open state.
2. Open State – In this state the pattern sets an internal timer to elapse at a configured value. Until such time the timeout is reached no calls to the underlying resource are allowed. In some implementations of the pattern and as it was originally discussed dring this time the pattern throws an exception to indiciate that it is not allowing calls at this moment. I personally do not see much merit in this approach. Once the timeout is elapsed the pattern moves to Half Open state.
3. Half Open State – In this state the pattern allows one call to the underlying resource or the risky code. If this call succeeds the pattern immediately switches back to Closed state however if the call fails the pattern resets the timer and moves to Open state.
An open source project titled Polly brings this pattern beautifully to the .NET world along with a really structured way to allow developers to express transient exception handling policies such as Retry, Retry Forver, Circuit Breaker etc . Polly has complete async, await support. In a nutshell the approach constitutes of setting up “Policies” in the calling code where the code configures the Polly framework on how the developer wants to handle a particular situation if it happens. Next an instance of this policy is created and it is used to execute the code. Here are some details (taken from the Github page directly)
Step 1 : Specify the type of exceptions you want the policy to handle
// Single exception type Policy .Handle<DivideByZeroException>() // Single exception type with condition Policy .Handle<SqlException>(ex => ex.Number == 1205) // Multiple exception types Policy .Handle<DivideByZeroException>() .Or<ArgumentException>() // Multiple exception types with condition Policy .Handle<SqlException>(ex => ex.Number == 1205) .Or<ArgumentException>(ex => x.ParamName == "example")
Step 2 : Specify how the policy should handle those exceptions
Retry
// Retry once Policy .Handle<DivideByZeroException>() .Retry() // Retry multiple times Policy .Handle<DivideByZeroException>() .Retry(3) // Retry multiple times, calling an action on each retry // with the current exception and retry count Policy .Handle<DivideByZeroException>() .Retry(3, (exception, retryCount) => { // do something }); // Retry multiple times, calling an action on each retry // with the current exception, retry count and context // provided to Execute() Policy .Handle<DivideByZeroException>() .Retry(3, (exception, retryCount, context) => { // do something });
Retry forever
// Retry forever Policy .Handle<DivideByZeroException>() .RetryForever() // Retry forever, calling an action on each retry with the // current exception Policy .Handle<DivideByZeroException>() .RetryForever(exception => { // do something }); // Retry forever, calling an action on each retry with the // current exception and context provided to Execute() Policy .Handle<DivideByZeroException>() .RetryForever((exception, context) => { // do something });
Retry and Wait
// Retry, waiting a specified duration between each retry Policy .Handle<DivideByZeroException>() .WaitAndRetry(new[] { TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(3) }); // Retry, waiting a specified duration between each retry, // calling an action on each retry with the current exception // and duration Policy .Handle<DivideByZeroException>() .WaitAndRetry(new[] { 1.Seconds(), 2.Seconds(), 3.Seconds() }, (exception, timeSpan) => { // do something }); // Retry, waiting a specified duration between each retry, // calling an action on each retry with the current exception, // duration and context provided to Execute() Policy .Handle<DivideByZeroException>() .WaitAndRetry(new[] { 1.Seconds(), 2.Seconds(), 3.Seconds() }, (exception, timeSpan, context) => { // do something }); // Retry a specified number of times, using a function to // calculate the duration to wait between retries based on // the current retry attempt (allows for exponential backoff) // In this case will wait for // 2 ^ 1 = 2 seconds then // 2 ^ 2 = 4 seconds then // 2 ^ 3 = 8 seconds then // 2 ^ 4 = 16 seconds then // 2 ^ 5 = 32 seconds Policy .Handle<DivideByZeroException>() .WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)) ); // Retry a specified number of times, using a function to // calculate the duration to wait between retries based on // the current retry attempt, calling an action on each retry // with the current exception, duration and context provided // to Execute() Policy .Handle<DivideByZeroException>() .WaitAndRetry( 5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)), (exception, timeSpan, context) => { // do something } );
Circuit Breaker
// Break the circuit after the specified number of exceptions // and keep circuit broken for the specified duration Policy .Handle<DivideByZeroException>() .CircuitBreaker(2, TimeSpan.FromMinutes(1));
Step 3 : Execute the policy
// Execute an action var policy = Policy .Handle<DivideByZeroException>() .Retry(); policy.Execute(() => DoSomething()); // Execute an action passing arbitrary context data var policy = Policy .Handle<DivideByZeroException>() .Retry(3, (exception, retryCount, context) => { var methodThatRaisedException = context["methodName"]; Log(exception, methodThatRaisedException); }); policy.Execute( () => DoSomething(), new Dictionary<string, object>() {{ "methodName", "some method" }} ); // Execute a function returning a result var policy = Policy .Handle<DivideByZeroException>() .Retry(); var result = policy.Execute(() => DoSomething()); // Execute a function returning a result passing arbitrary context data var policy = Policy .Handle<DivideByZeroException>() .Retry(3, (exception, retryCount, context) => { object methodThatRaisedException = context["methodName"]; Log(exception, methodThatRaisedException) }); var result = policy.Execute( () => DoSomething(), new Dictionary<string, object>() {{ "methodName", "some method" }} ); // You can of course chain it all together Policy .Handle<SqlException>(ex => ex.Number == 1205) .Or<ArgumentException>(ex => ex.ParamName == "example") .Retry() .Execute(() => DoSomething());>
Polly – https://github.com/App-vNext/Polly
4 Trackbacks