Flip stands for Fault Injection Point. Its a generic framework for injecting fault into the code. It provides a framework, where actual fault could be injected outside the application.
To induce the system to take an alternate path of the code, we simulate faults. One typical way to achieve this is by explicitly writing a injected fault and make it compile time option to trigger, something like.
io_status IO::submit(packet_t *pkt) {
...
#ifdef INJECT_SUBMIT_FAULT
return io_status::error;
#endif
....
return real_submit(pkt);
}Unfortunately this is extremely limited because every time a fault needs to triggered it needs to be recompiled. Another limitation to this approach is that it hits on every call to a method irrespective of any condition. This makes it lot of hand holding and affect automation of the test cases. Flip tries to eliminate these issues and provide a generic framework to ease out fault injection.
Following are some of the important features of Flip:
- Multiple fault injection points: Flip supports multiple fault injection points in a single program. Each fault injection point is associated with an unique name. Example:
Flip flip;
io_status IO::submit(packet_t *pkt) {
if (flip.test_flip("fail_all_writes")) {
// Do your failure generation here
return io_status::error;
}
return real_submit(pkt);
}-
Trigger fault externally: One of the major design goal is the ability to trigger these faults externally. Flip provides a protobuf based serialization, which can be used by any RPC mechanism.
-
Parameterized faults: In the above example, if there should be a provision for different types of packets to be injected at different instances, there would be a fault injection for every type. This will quickly become unscalabale approach as more and more packet types could be added and generic framework idea will be lost. Flip provides parameterized definition of faults. The above example could be extended to
Flip flip;
io_status IO::submit(packet_t *pkt) {
if (flip.test_flip("fail_specific_writes", pkt->op_code)) {
// Do your failure generation here
return io_status::error;
}
return real_submit(pkt);
}Here the pkt->op_code is the parameter which could be controlled externally in-order to inject the fault. Flip supports filtering various conditions (not just ==) to the value of the parameter. Hence, while triggering in the above example one can trigger all OP_TYPE_CREATE or anything but OP_TYPE_CREATE etc.
There are no limits to number of parameters, but the 2 conditions will be anded. The above example could be expanded to
Flip flip;
io_status IO::submit(packet_t *pkt) {
if (flip.test_flip("fail_specific_writes", pkt->op_code, pkt->size)) {
// Do your failure generation here
return io_status::error;
}
return real_submit(pkt);
}- Return specific values: It is useful to inject an alternate path, but what will be more useful is whats the error it should generate as configurable. This will avoid multiple flip points for different types of error generation. Extending above example, if one wants to simulate return of different errors from IO::submit, one can write similar flip.test_flip("") for all possible error codes, but it will become too verbose. Flip supports parameterized return as well. Hence it one could do the following
Flip flip;
io_status IO::submit(packet_t *pkt) {
auto ret = flip.get_test_flip("fail_specific_writes", pkt->op_code, pkt->size);
if (ret) {
// Do your failure generation here
return ret.get();
}
return real_submit(pkt);
}Now fault injection externally can make flip return specific errors, not just io_status::error, but say io_status::corrupted_data etc..
- Async delay injection: One more typical use of fault injection is to delay execution or simulate delays to test timeout code paths etc.. In a sync code, it is easy to put a sleep to reproduce the delay. However, more and more applications do async operation or code being completely async. In these cases, there needs a timer routine to keep track of the delay. Flip covers this and creates a simple async delay injection framework.
Flip flip;
void IO::process_response(packet_t *pkt) {
if (flip.delay_flip("delay_response", [this, pkt]() {
IO::real_process_response(pkt);
}, pkt->op_code)) {
return;
}
IO::real_process_response(pkt);Above example, provide the fault injection to delay specific opcode. After the configured delay (externally controllable) it calls the closure. As always number of parameters are unlimited and it is exactly similar to the other types of fault injections explained above. Also like other faults, it can be controlled externally on how many times and how frequent the faults have to be triggered.
Flip supports combining delay_flip and simulated_value generation flip, so it can generate a specific value after imparting delay. This will be useful, since after delay an application typically return a timeout error or other types of errors which will have different behavior in apps, which needed to be tested.
Flip flip;
void IO::process_response(packet_t *pkt) {
if (flip.get_delay_flip("delay_response", [this, pkt](io_status generated_error) {
pkt->status = generated_error;
IO::real_process_response(pkt);
}, pkt->op_code)) {
return;
}
IO::real_process_response(pkt);In the above example after delay the value injected externally is passed to the closure, which could be used to simulate various error scenarios.
- Callback injection: For more complex fault scenarios where you need to execute custom logic at the fault injection point,
Flip provides callback-based fault injection. Unlike predefined actions (return value, delay), callbacks allow you to run arbitrary
code when a flip is triggered. This is useful for:
- Complex state manipulation
- Multi-step fault scenarios
- Synchronization among multiple systems
- Conditional fault logic based on runtime state
- Integration with testing frameworks or mocks
- Enhance visibility in the test codes
Flip flip;
io_status IO::submit(packet_t *pkt) {
// Callback flip: no return value
if (flip.callback_flip("corrupt_packet", pkt->op_code, pkt->size)) {
// Callback was triggered and executed
return io_status::error;
}
return real_submit(pkt);
}In this example, when the flip is triggered, it executes a user-defined callback function that was registered during the injection phase. The callback can perform any custom logic like logging, state manipulation, or even more complex fault scenarios.
Flip also supports callbacks with return values, which is useful when the callback needs to compute a specific error value:
Flip flip;
io_status IO::submit(packet_t *pkt) {
// Callback flip: with return value
auto error = flip.get_callback_flip<io_status>("compute_error", pkt->op_code);
if (error) {
return error.get(); // Return value computed by callback
}
return real_submit(pkt);
}Key features of callback flips:
- Type safety: Callbacks are type-checked at compile time with template parameters
- Flexible signatures: Support arbitrary parameter lists (int, string, custom types, etc.)
- Thread safety: Callbacks are cloned before execution to avoid holding locks while running user code
- No deadlock: Callbacks execute outside of Flip's internal locks, so they can safely call other Flip methods
- Exception safety: If a callback throws an exception, it's caught and logged without crashing
Current limitations:
- Local client only: Callback flips are currently only supported through the local FlipClient. RPC-based injection (gRPC/Python client) does not support callbacks. Use callback flips for unit tests.
Flip usage has 2 phases
- Definition phase: This is the phase where the declaration of which place and what action needs to be taken in application code with the fault. It needs to be written before compiling the application code.
- Injection phase: This is the phase where the faults are injected or triggered either through local flip client or external client.
There are 4 important parameters that needs to be determined for a fault point. The proto file proto/flip_spec.proto defines these parameters:
Flip name: Unique name identifies this flip point. There can be multiple instances (with different parameters) for same flip point, say "fail_writes" flip for opcode=1, opcode=2 can coexist (with the same name) in above examples. The name needs to be declared during definition phase and addressed with that during injection phase.
Flip Parameters: The list of parameters that needs to filter out a flip. During definition phase, application code needs to decide what are the possible filtering attributes to control with. Thus it is advisable to write a flip in possible common portion of the code and let injection phase decide what it needs to filter on. In above example flip.get_test_flip("fail_specific_writes", pkt->op_code, pkt->size) allows the injection phase to filter on opcode and pkt_size. Note that if there are multiple parameters each filter conditions are and'd.
During injection phase, user can supply values and operator for each parameter. Flip as of now only supports primitive types (int, long, double, bool, uint64) and string, const char* as parameter types. It supports all operators (==, >, <, >=, <=, !=) and one more called "*" to ignore the check for this parameter.
Flip Action: If the fault is triggered what action the application should take. Flip supports 6 types of action
- No explicit action: Flip does not take any other action other than returning the fault is hit. Application code will then write the error simulation code.
- Return a value action: A value decided during injection phase (of type determined during definition phase) will be returned as part of flip hit.
- Delay action: Introduce a time delay determined during the injection phase.
- Delay and return a value action: Combining above 2.
- Callback action: Execute a user-defined callback function when the flip is triggered.
- Callback with return value action: Execute a user-defined callback function and return its result.
Flip frequency: This is actioned only during injection phase, which determines how frequently and how much the flip has to hit or trigger the fault.
Count: How many times it needs to hit. Frequency: either percentage of times it needs to hit (to randomly hit for this much percentage) or every nth time
Flip can be initialized with default constructor. In future it will accept parameters like having application own timer routine and also specific grpc instance. As of now flip when called with default constructor creates/uses its own timer and thread for delay and does not provide any RPC service to call.
template< class... Args >
bool test_flip(std::string flip_name, Args &&... args);Test if flip is triggered or not. Parameters are
- flip_name: Name of the flip
- args: variable list of arguments to filter. Arguments can be of primitive types or std::string or const char *
Returns: If flip is hit or not. Flip is hit only if it matches the filter criteria and frequency of injection criteria. This API can be called on any of the 4 types of flip.
template< typename T, class... Args >
boost::optional< T > get_test_flip(std::string flip_name, Args &&... args);Test if flip is triggered and if triggered, returns injected value.
- flip_name: Name of the flip
- args: variable list of arguments to filter. Arguments can be of primitive types or std::string or const char *
Returns: If flip is not hit, returns boost::none, otherwise returns the injected value. The injected value cannot be of one of the primitive types or std::string. This API is only valid for "return a value action" flip type.
template< class... Args >
bool delay_flip(std::string flip_name, const std::function<void()> &closure, Args &&... args);Test if flip is triggered and if triggered, calls the supplied closure callback after injected delay time in microseconds.
- flip_name: Name of the flip
- closure: The callback closure to call after the delay, if flip is hit
- args: variable list of arguments to filter. Arguments can be of primitive types or std::string or const char *
Returns: If the flip is hit or not. Whether flip is hit or not is immediately known.
template<typename T, class... Args >
bool get_delay_flip(std::string flip_name, const std::function<void(T)> &closure, Args &&... args);Test if flip is triggered and if triggered, calls the supplied closure callback after injected delay time in microseconds with the injected value as a parameter.
- flip_name: Name of the flip
- closure: The callback closure to call after the delay, if flip is hit. The closure should accept the parameter of type which the fault could be injected with.
- args: variable list of arguments to filter. Arguments can be of primitive types or std::string or const char *
Returns: If the flip is hit or not. Whether flip is hit or not is immediately known.
template< class... Args >
bool callback_flip(std::string flip_name, Args &&... args);Test if flip is triggered and if triggered, executes the registered callback function.
- flip_name: Name of the flip
- args: variable list of arguments to filter. Arguments can be of primitive types or std::string or const char *
Returns: true if the flip was triggered and callback executed successfully, false if the flip did not match or was rate-limited.
Important notes:
- The callback is executed synchronously in the calling thread
- The callback executes outside of Flip's internal locks to prevent deadlocks
- Callbacks can safely call other Flip methods (add, remove, test_flip, etc.)
- If the callback throws an exception, it's caught and logged, and the method returns
false
template< typename T, class... Args >
boost::optional< T > get_callback_flip(std::string flip_name, Args &&... args);Test if flip is triggered and if triggered, executes the registered callback function and returns its result.
- flip_name: Name of the flip
- args: variable list of arguments to filter. Arguments can be of primitive types or std::string or const char *
Returns: If flip is not hit, returns boost::none, otherwise returns the value returned by the callback function. The return type T
must match the callback's return type.
Example:
// Callback that computes an error code
auto result = flip.get_callback_flip<int>("error_code_flip", pkt->op_code);
if (result) {
int error_code = result.get(); // Value returned by callback
return handle_error(error_code);
}Flip is a header only framework and hence will be included and compiled along with application binary. It uses a protobuf to serialize the message about how faults can be triggered. The protobuf could be used against any RPCs the application provide.
Flip supports an optional GRPC server which can be started using
Flip::start_rpc_server()If application uses GRPC and if another GRPC server creation is to be avoided, then application can instead add the following grpc definition to its RPC call to the grpc service proto.
// Inject a fault rpc
rpc InjectFault (flip.FlipSpec) returns (flip.FlipResponse);Flip needs a client to trigger the faults externally. At present there are 2 forms of flip client, one GRPC client, which allows flip faults can be injected remotely from external application. Second form is local flip client, which means flip will be triggered by the same application binary which is to be fault tested.
There is python grpc client library through which it can be triggered remotely. To setup the python client, execute
bash ./setup_python_client.sh
Python library is available under src/client/python/flip_rpc_client.py
Libraries are defined in class FlipRPCClient. Examples of how to use library is provided by the python script src/client/python/flip_client_example.py
If the code that needs to be fault injected and tested is a library in itself and that there is separate unit tests which runs the code, we don't need an RPC, but local calls to trigger the faults. For example, if the tested code runs with GTest or other unit test framework, the test code runs in the same context as actual code. Flip provides a FlipClient class to trigger faults.
Following are the APIs for FlipClient
FlipClient(Flip *f)Constructs a flipclient providing the flip instances of actual code. Typically FlipClient and Flip could be singleton instances.
template< typename T>
void create_condition(const std::string& param_name, flip::Operator oper, const T& value, FlipCondition *out_condition);Parameters are:
- param_name: This is not used for any cross verification but just for logging purposes
- oper: One of the flip operators (==, >, <, >=, <= !=, *)
- value: Value parameter of the filter criteria
- condition: Returns the FlipCondition that will be passed in subsequent APIs.
bool inject_noreturn_flip(std::string flip_name, std::span< const FlipCondition > conditions, const FlipFrequency &freq);
template <typename T>
bool inject_retval_flip(std::string flip_name, std::span< const FlipCondition > conditions,
const FlipFrequency& freq, const T& retval);
bool inject_delay_flip(std::string flip_name, std::span< const FlipCondition > conditions,
const FlipFrequency& freq, uint64_t delay_usec);
template <typename T>
bool inject_delay_and_retval_flip(std::string flip_name, std::span< const FlipCondition > conditions,
const FlipFrequency &freq, uint64_t delay_usec, const T& retval);
template <typename... Args>
bool inject_callback_flip(std::string flip_name, std::span< const FlipCondition > conditions,
const FlipFrequency& freq, std::function<void(Args...)> callback);
template <typename T, typename... Args>
bool inject_callback_retval_flip(std::string flip_name, std::span< const FlipCondition > conditions,
const FlipFrequency& freq, std::function<T(Args...)> callback);Parameters are:
- flip_name: Name of the flip to inject the fault to
- conditions: Vector of conditions which will be and'd. Each condition can be created using create_condition API
- freq: Flip frequency determines how much and how frequent. Can be constructed using FlipFrequency::set_count(), FlipFrequency::set_percent(), FlipFrequency::set_every_nth()
- retval: (for inject_retval_flip and inject_delay_and_retval_flip): What is the injected value to be returned or called back respecitvely
- delay_usec: (for inject_delay_flip and inject_delay_and_retval_flip): How much delay to inject
- callback: (for inject_callback_flip and inject_callback_retval_flip): User-defined callback function to execute when flip is triggered.
- For
inject_callback_flip: callback signature isvoid(Args...)where Args match the parameters passed tocallback_flip() - For
inject_callback_retval_flip: callback signature isT(Args...)where T is the return type
- For
Returns:
- If successfully injected the fault or not.
Example code
Flip flip;
FlipClient fclient(&flip);
...
FlipCondition cond1, cond2;
fclient.create_condition("cmd_type", flip::Operator::EQUAL, (int)1, &cond1);
fclient.create_condition("size_bytes", flip::Operator::LESS_THAN_OR_EQUAL, (long)2048, &cond2);
FlipFrequency freq;
freq.set_count(2); freq.set_percent(100);
fclient.inject_delay_flip("delay_flip", std::vector{cond1, cond2}, freq, 100000 /* delay in usec */);Above examples, trigger a flip called delay flip to inject a delay of 100ms if cmd_type == 1 and size_bytes <= 2048