Requeue Missing Message Data

Sometimes a workflow will fail to fetch data, for example, when the remote service is temporarily offline. To recover missing data or to re-queue data that’s already been processed, advanced date filters can be applied to a workflow trigger. This, along with the “Ignore Last Run” option can be used to add messages to the queue that were either missed entirely, or simply need to be re-processed.

The video above walks through the process of configuring a workflow trigger so that message data can be added which would normally be disregarded.

In summary:

  • Go to the workflow’s “Workflow Trigger” page.
  • Update trigger settings to fetch data that may have already been processed or may have been missed during a remote service outage. Each Connector and Source will have different filter options, but most have:
    • Ignore Last Run - when checked, this option will load message data based on the provided filters. Use with caution, as this will repeatedly load the same data, depending on filters.
    • Various date filters - when configured, these will load data only within a certain date range. Use these filters in combination with “Ignore Last Run” to fetch historical data for re-processing.
  • Update the schedule to “Manual” to ensure this data is only loaded as needed.
  • Save changes, then restart the engine to ensure schedule updates are applied.
  • Go to the “Messages” screen and Run the workflow to load the newly filtered data.
  • Review and troubleshoot.
  • When ready, restore the workflow settings:
    • Un-check Ignore Last Run
    • Remove date filters
    • Restore the workflow schedule back to normal
  • Restart the engine.

For high-volume workflows, the following tips may be helpful:

  • Smaller date ranges may be needed (for example a day at a time) depending on the vendor’s support for data retrieval. Confirm vendor-specific data limits before fetching large amounts of data (this is typically documented in the vendor’s API)
  • If running the workflow multiple times, an additional round of filtering may need to be applied to handle “today’s” data while the above changes were run.