Creating a Dataset Template

A dataset template is a directory on a shared file store where file-based linked data sets can be written by Graph Studio, such as when unstructured data is onboarded or data is exported to disk from a graphmart. A dataset template also defines write properties such as the maximum file size and whether files should be compressed. One dataset template, called Default Graph Studio Shared Filesystem Template, is created automatically on the shared file system during the Graph Studio installation. You can configure all pipelines and workflows to write to that directory or you can create multiple templates to use for different datasets. Follow the instructions below if you want to create a dataset template.

1. In the Administration application, expand the Connections menu and click Graph Studio Data Store. Graph Studio displays the Graph Studio Data Store screen, which lists any existing data stores. For example:

   

   The Server Graph Studio Data Store is a default data store that points to the local Graph Studio file system. This store exists so that first-time users can quickly test the onboarding process. It is not meant to be used in production. Do not change the Data Location to a shared file store; reconfiguring this Data Store can cause unexpected consequences when upgrading or migrating the system. It is safe to delete this store so that it is not presented as an option when users configure ingestion pipelines.

2. On the Graph Studio Data Store screen, click the Add Graph Studio Data Store button and select Add Graph Studio Data Store. Graph Studio opens the Create Graph Studio Data Store screen.

   

3. Type a Title and optional Description for the data store.

4. Click in the Data Location field. Graph Studio opens the File Location dialog box.

   

5. On the left side of the screen, select the File Store on which to create this data store. On the right side of the screen, navigate to the directory that you want to designate as the data location. Select a directory, and then click OK. Or click Create New Folder to create a new directory. Each time a pipeline is run for this data store, a new subdirectory is created under the specified data location.

   The Data Location needs to be a directory on the file store that is shared between Graph Studio, Graph Lakehouse, and any Graph Studio Unstructured, or Elasticsearch servers. If you want Graph Studio to generate files for this data store in one location and then load the files into Graph Lakehouse from another location, specify the file generation location in this field, and then specify the Graph Lakehouse load location in the Alternate Data Location field that is displayed on the Details screen after you save the data store.

6. If necessary, you can modify the maximum limit for the size of the files that are created by pipelines that write to this data store by specifying the size (in bytes) in the Max File Size Before Compression (Bytes) field. The value applies to files before they are compressed.

   Altair recommends that you do not set this value unless instructed to do so by Altair Support.

7. Specify whether to compress the generated load files. By default, the Compress output checkbox is selected, indicating that Graph Studio generates .ttl.gz files when writing to this graph data source. If you clear the checkbox, Graph Studio generates uncompressed .ttl files. To preserve disk space and reduce read times when loading data into memory, Altair recommends that you accept the default configuration and compress load files.

8. Click Save to create the data store. Graph Studio saves the configuration and displays the details view. For example:

   

   You can click the Edit icon () to modify any of the options. Click the check mark icon () to save changes to an option, or click the X icon () to clear the value for an option.

9. If you plan to load files into Graph Lakehouse from a location that is different than the Data Location that you specified, edit the Alternate Data Location field and select the location for Graph Lakehouse load files.