Skip to content

External Pipeline Flows

Introduction

The pipeline flow schema describes two ways of storing a sub-flow pipelines for a supernode: local or external. Local sub-flows are stored within the pipeline flow document whereas external pipelines are stored in a separate pipeline flow document (usually as the primary pipeline in that pipeline flow). The supernode that references an external pipeline contains a url property (in <node>.subflow_ref.url) and a pipeline ID property (in <node>.subflow_ref.pipeline_id_ref) to identify the external pipeline flow and the pipeline within it.

Common Canvas supports external pipelines but the host application UI code needs to manage the storage and retrieval of any external pipeline flows. To do this the app UI code needs to:

  • Manage the storage of newly created external pipeline flows.
  • Respond to calls from Common Canvas to provide the external pipeline flow(s) when requested.
  • Respond to call from Common Canvas when a local pipeline is changed to external or an external is changed to local.

Common Canvas supports lazy loading of external pipeline flows so they will only be requested from your code when the user performs some gesture that requires the pipeline (from within the external pipeline flow) to be displayed.

Creating an external sub-flow

When the common-canvas config property enableExternalPipelineFlows is set to true (the default) and, when a set of objects are selected from which a super node can be created, the default common-canvas context menu will include a Create External Supernode option.

When the Create External Supernode option is clicked the createSuperNodeExternal action is executed.

If your application doesn’t use the default context menu you can define your own context menu (returned from contextMenuHandler) to contain an option which maps to the createSuperNodeExternal action. See the section on the contextMenuHandler for details on how to do this.

When the createSuperNodeExternal action is executed, the beforeEditActionHandler callback is called before the external sub-flow is created. The beforeEditActionHandler is called where the first parameter data has two properties externalUrl and externalPipelineFlowId which will both be set to empty string. Your code must set these to whatever values you want for the url and pipeline flow ID. The url will be assigned to the subflow_ref.url property of the supernode that is being created. The pipeline flow ID will be assigned to the newly created pipeline flow.

Your code must return the data parameter from the beforeEditActionHandler callback if you want the action to proceed and create the sub-flow. If you need to do any asynchronous activity at this point see the documentation on the beforeEditActionHanlder for details on how to do that.

When the sub-flow has been created, Common Canvas will call the editActionHandler callback with the createSuperNodeExternal action. In this callback you can, if you wish, retrieve the pipeline flow document that has been created internally in Common Canvas using CanvasController.getExternalPipelineFlow(url). Your code can then save it to your repository. Alternatively, you can wait until some later time, like perhaps during an auto-save, to retrieve and store the pipeline flow externally in your repository.

Loading an external sub-flow

When the main pipeline flow, displayed by Common Canvas, contains a super node that references an external sub-flow it will need to be loaded whenever the user performs a gesture that causes it to be displayed or processed in some way - for example displaying it ‘in-place’ or converting it from an external to a local supernode. An external sub-flow will also need to be loaded if the top-level pipeline being displayed has a supernode, that refers to an external pipeline, that is already expanded in-place in the saved pipeline flow JSON document being displayed. So, actions that can cause the external pipeline to be loaded are:

  • loadPipelineFlow
  • expandSuperNodeInPlace
  • displaySubPipeline
  • convertSuperNodeExternalToLocal
  • deconstructSuperNode

When any of these actions are performed Common Canvas will call the beforeEditActionHandler callback with the data parameter as the first parameter. The data object will have the following properties:

  • editType - The name of the action being performed.
  • externalPipelineFlowLoad - This is a boolean which indicates whether the pipeline flow needs to be provided by your code.
  • externalUrl - This is the string which identifies the external pipeline flow document.
  • externalPipelineId - This is the ID of the pipeline being loaded
  • externalPipelineFlow - If externalPipelineFlowLoad if true this will be undefined. Otherwise it will be contain the previously loaded external pipeline flow.

You need to implement the beforeEditActionHandler so that:

  • when the actions above are being performed and externalPipelineFlowLoad is true, you retrieve the external pipeline flow from your repository
  • you then assign it to the externalPipelineFlow property of the data object
  • you then return the data object from the callback.

Your code must return the data parameter from the beforeEditActionHandler callback if you want the action to proceed and load the external pipeline flow. If you need to do any asynchronous activity at this point see the documentation on the beforeEditActionHanlder for details on how to do that.

Converting a local supernode to an external supernode

When the common-canvas config property enableExternalPipelineFlows is set to true, and a local supernode’s is right clicked, the default common-canvas context menu will include a Convert local to external option. This will execute the convertSuperNodeLocalToExternal action.

The convertSuperNodeLocalToExternal action is similar to the createSuperNodeExternal action in that a new external pipeline flow is being created. Consequently, you can follow the instructions in the Creating an external sub-flow section for providing the appropriate properties of the data object in the beforeEditActionHandler and editActionHandler callbacks.

Converting an external supernode to a local supernode

When the common-canvas config property enableExternalPipelineFlows is set to true, and an external supernode’s is right clicked, the default common-canvas context menu will include a Convert external to local option. This will execute the convertSuperNodeExternalToLocal action.

The convertSuperNodeExternalToLocal action is similar to the expandSuperNodeInPlace action in that an external pipeline flow may need to be retrieved from your repository. Consequently, you can follow the instructions in the Loading an external sub-flow section for providing the appropriate properties of the data object in the beforeEditActionHandler callback.

Deleting an external supernode/sub-flow

When the user deletes an external supernode/sub-flow the supernode will be removed from the canvas. You code does not need to do anything unless you want to also remove the external pipeline flow from your repository.

Clipboard support

When a supernode, that refers to an external pipeline, is cut/copied and pasted, the pasted supernode refers to the same external pipeline as the supernode that was cut or copied. This means that if an external supernode is copied to the clipboard and then pasted into the same canvas the result will be two supernodes that refer to the same external pipeline.  

The same situation can occur if a supernode, that refers to an external pipeline, is in the palette and that node is dragged multiple times from the palette onto the canvas.

Manipulating objects in external pipelines using the Canvas Controller API

Objects (nodes, links comments, etc.) in an external pipeline can be updated by the host application calling the CanvasController API. However, such changes are only effective within Common Canvas. It is the host application’s responsibility to make sure these changes are persisted in the external pipeline flow document (if that is the behavior that is required). This can be done by the host application calling CanvasController.getExternalPipelineFlow(url) and then saving the returned document to the appropriate repository.

External pipelines in the Elyra Canvas Test Harness

The Test Harness supports external pipeline flows but will only persist any saved flows for the current session. (It just stores them in memory). You can examine the beforeEditActionHandler and editActionHandler in App.js in the test harness to see how it handles the different actions for managing external pipeline flows.