Creating a New Pipeline

1. In the Graph Studio application, expand the Onboard menu and click Unstructured Data. Graph Studio displays the Pipelines screen, which lists any existing unstructured pipelines. For example:

   

2. Click the Add Unstructured Pipeline button and select Distributed Unstructured Pipeline. Graph Studio opens the Create Distributed Unstructured Pipeline dialog box. For example:

   

3. In the Title field, type a name for the pipeline.

   The title serves as a key to identify this pipeline and its corpus in multiple contexts. Specify a title that is unique and stable. The pipeline's corpus dataset name is derived from this value.

4. Type an optional description for the pipeline in the Description field.

5. If necessary, click the Target Graph Studio Data Store field and select the Graph Studio Data Store for this pipeline.

6. If the environment is configured for dynamic Kubernetes-based deployments of the infrastructure, select the Deploy Unstructured Infrastructure Dynamically checkbox and leave the Static Elasticsearch Config field blank.

7. If necessary, click the Static Elasticsearch Config field and select the Elasticsearch connection to use for this pipeline. If you use dynamic deployments to deploy Elasticsearch instances on-demand, leave this field blank. Graph Studio prompts the user to choose a Cloud Location when the pipeline is run.

8. Click Save to create the pipeline. Graph Studio displays the pipeline Overview screen. For example:

   

   A pipeline saves automatically and constantly undergoes validation to make sure that it is valid based on the current configuration. Graph Studio displays validation issues in red on the top of the screen. The warnings will disappear as you add components to the pipeline.

9. If necessary, click Advanced to configure the advanced pipeline settings. For details about the advanced settings, see Pipeline Settings Reference.

10. Next, follow the instructions in Add Crawlers to the Pipeline to add one or more crawlers to the pipeline.