Pipelines are the collections of transformations applied to a data payload. Every Pipeline requires at least one (1) transformation and may be assigned a virtually unlimited number of transformations.
These transformations are grouped into four (4) categories:
Transform (Technical Transformations) - modify the data layout of the payload.
Filter - Remove rows from the data payload based on a criterion/criteria.
Data - Sort data by specific fields and apply summation to derive data points.
Mapping (Functional Transformations) - build relationships between data fields.
Adding transformations to a Pipeline is a very guided, but not prescriptive process. This ensures the ultimate flexibility to address your transformation needs.
📓 Prior to adding transformations to your Pipeline, be sure to pin a sample file.
The First Transformation
Transformations are added from the Transformation Tray. The tray can be pivoted to a vertical orientation by clicking the "List View" button next to the "Publish" button. Pivot back to a horizontal view by clicking the "Flow" button to the left of the "List View".
The Transformation Selector Form displays the available transformations. They are grouped by category - Transfer, Filter, Data, Mapping.
Click the blue plus (+) sign in the bottom-middle of the Transformation Tray.
Navigate to the appropriate transformation type in the Transformation Selector.
Select the transformation to add to the Pipeline.
Configure a Transformation
Once a transformation has been added to the Pipeline, it must be configured. The Transformation form displays to the left of the data grid displaying the sample file contents. Each transformation has configuration fields that are specific to the action the transformation will perform.
Any field marked with an asterisk (*) is required and must be completed. Even if a field is not specified as required, be sure to review the field as it may contain information that will be critical to the transformation performing as expected.
For example, when adding a new column with a static value to the Pipeline using the Insert Column transformation, the column Data Type dropdown should be reviewed or updated to ensure that the new column is set as the appropriate data type.
Once the transformation form is complete, click the Save button to apply the transformation to the Pipeline. The transformation is added to the transformation tray and the result of the transformation is visible in the data grid. Toggle the transformation configuration form on-and-off by clicking the target icon in the header.
Additional Transformation Functions
Add a Transformation
Additional transformations can be added to a Pipeline by clicking the green plus signs to the right or left of an existing transformation. Clicking the plus-sign to the left or right of an existing transformation adds a new transformation before or after it, respectively.
Copy a Transformation
Any existing transformation in the Pipeline can be duplicated by clicking the copy icon that is under the transformation. A copy of the transformation is created and the transformation form is opened. Update the transformation inputs as needed and save the copied transformation.
Delete a Transformation
Any transformation in a Pipeline can be deleted by clicking the trash can icon under the transformation. When prompted to confirm the delete, type the required text and then click the Delete button. Click the Back button on the confirmation form to cancel the deletion.
When deleting a transformation that creates a new column in the Pipeline, any subsequent transformation that utilizes the column that was created by the now-deleted transformation will be marked as invalid and the transformation icon (blue circle) will turn red to indicate that the transformation is no longer valid.
⚠️ Deleted transformations cannot be restored.
The order of the transformations can be changed by clicking the left and right arrows under a specific transformation. By clicking the left arrow, the transformation is shifted to occur immediately before the transformation that previously proceeded it. The right arrow shifts the transformation to occur after the transformation that previously followed.
📓 A transformation can be shifted as many places to the left or right as needed. However, as with deleting a transformation, any subsequent transformations that depend on the result of the transformation being shifted must continue to follow.