Table of Contents
Understanding Filter Rules: A Simple Guide
Filter Rules help clean up all your data. When you upload data to our platform, these rules act like a broom, sweeping away anything you don't need and keeping your data tidy.
Filters are used to remove any unnecessary data from the source data upload.
How to Create Your Own Filter Rules
Step 1: Accessing the Rules Section
Click on the "+ Add Filter Rules" button found in the "Set Filter Rules" dialog, under the Map Data section.
If further data cleaning/transformation needs to be done on the source data then use the expression builder, accessed by clicking on the "+ Add Filter Rules" button in the "Set Filter Rules" dialogue in the Map Data section, option 2.
Step 2: Building a Rule
When you open the Filter rules section, a "Target Expression Builder" will pop up. It's like a tool to make your own rules. This will allow you to build "Expressions" and create rules, you'll see some ready-to-use functions to get you started.
An expression is a piece of code that transforms source data. It is used to assist at the mapping stage. The expression is written using the Groovy coding language or using a handful of built-in functions.
For example Left( {Policy Id}, 4 ) will return the first 4 characters in the "Policy Id" source field.
Use the source fields and available functions to build your rule. As you do this, you'll see your creation take shape in the code section.
What is a Function?
"Functions" refer to predefined operations that can be applied to data expressions to filter, transform, or manipulate data. These functions are akin to Excel functions, offering a familiar syntax and methodology for users looking to refine their data sets. Below are examples of some commonly used functions within the platform:
trim(): This function removes any leading or trailing spaces from a text string. It is particularly useful for cleaning up data that may have inconsistent formatting or has been imported from various sources.
left(text, number): Extracts a specified number of characters from the beginning of a text string. This function is useful for isolating specific data within a string, such as extracting area codes from phone numbers or abbreviations from longer text.
right(text, number): Similar to the left() function, but extracts characters from the end of a text string. This can be used for retrieving specific information like file extensions from filenames or the last few digits of a numeric string.
These functions can be combined and nested within expressions to perform complex data transformations, enabling users to tailor their data exactly as needed for analysis, reporting, or further data processing tasks.
What is a Source Field?
"Source Fields" refer to the column headers from an inputted file. These fields represent the foundational elements of the dataset, acting as identifiers for the data contained within each column.
Key Features and Usage:
Drag-and-Drop Interface: Source fields can be easily integrated into expressions by using a user-friendly drag-and-drop interface. This intuitive design facilitates the efficient construction of expressions, enabling users to seamlessly combine source fields with functions for advanced data manipulation.
Combination with Functions: Users can enhance their data manipulation capabilities by combining source fields with functions. This combination allows for the application of transformations such as trimming whitespace, extracting specific portions of data, or applying conditional logic directly to the data referenced by the source fields.
Data Transformation and Filtering: By utilizing source fields in expressions, users can perform sophisticated data transformations and filtering. This includes operations like concatenating fields, converting data types, or applying formulas to compute new values.
Step 3: Adding more Functions
Need more tools? Click on the Functions button. You can simply drag and drop these into your code.
Step 4: Testing your Rule
Click on "Parse Expression" and then "Test Expression." You can use a sample data record, which is provided by default from the source file or your own sample value.
Note - You will not be able to "Test" your expression until you have parsed it. Once you successfully parse your expression, test it.
After clicking on "Parse Expression", you will receive a message identifying the "variable" and the "value"
Then you should click "Test Expression". If the expression value is correct you will receive a "true" sample output.
Step 5: Saving your Rule
Once you're happy with your rule, hit "Save" to apply it to your data mapping.
Common Issues to Watch Out For
If your rule isn't working, it might be because it's referring to a data field that doesn't exist. This will show up as a "startup..." error in the validation error report. Double-check your rule if you see this and check the filter expression.
Practical Example
Let's say we don't want any empty spaces in a crucial field like the Policy Number. Here's a simple rule you could write:
{Policy Number} != null && {Policy Number} != ""
Let's break down the rule
Policy Number
Source Field
!=
Is not equal to
&&
and
This means the Policy Number in your data should never be empty or just a blank space.
And that's it! You're now ready to use Filter Rules. Remember, it's all about keeping your data clean and useful. If you have any questions or need more help, feel free to ask.