Requeue Missing Message Data
Sometimes a workflow will fail to fetch data, for example, when the remote service is temporarily offline. To recover missing data or to re-queue data that’s already been processed, advanced date filters can be applied to a workflow trigger. This, along with the “Ignore Last Run” option can be used to add messages to the queue that were either missed entirely, or simply need to be re-processed.
The video above walks through the process of configuring a workflow trigger so that message data can be added which would normally be disregarded.
- Go to the workflow’s “Workflow Trigger” page.
- Update trigger settings to fetch data that may have already been processed or may have been missed during a remote service outage. Each Connector and Source will have different filter options, but most have:
Ignore Last Run- when checked, this option will load message data based on the provided filters. Use with caution, as this will repeatedly load the same data, depending on filters.
- Various date filters - when configured, these will load data only within a certain date range. Use these filters in combination with “Ignore Last Run” to fetch historical data for re-processing.
- Update the schedule to “Manual” to ensure this data is only loaded as needed.
- Save changes, then restart the engine to ensure schedule updates are applied.
- Go to the “Messages” screen and Run the workflow to load the newly filtered data.
- Review and troubleshoot.
- When ready, restore the workflow settings:
Ignore Last Run
- Remove date filters
- Restore the workflow schedule back to normal
- Restart the engine.
For high-volume workflows, the following tips may be helpful:
- Smaller date ranges may be needed (for example a day at a time) depending on the vendor’s support for data retrieval. Confirm vendor-specific data limits before fetching large amounts of data (this is typically documented in the vendor’s API)
- If running the workflow multiple times, an additional round of filtering may need to be applied to handle “today’s” data while the above changes were run.