Apache Arrow Library

Apache Arrow is a software development platform for building high performance applications that process and transport large data sets. It is designed to improve both the performance of analytical algorithms and the efficiency of moving data from one system to another. One important feature of Apache Arrow is its in-memory columnar format, a standardized, language-agnostic specification for representing structured, table-like data sets in memory.

Graph Lakehouse can act as an Apache Arrow client for graph query driven exports and imports of in-memory data sets. Graph Lakehouse provides a collection of services that support the Arrow Flight protocol for integration with leading ML and other Big Data Ecosystems, including Python Pandas, Spark MLLIB and Google Tensorflow, Cassandra, Kudu, and Hadoop. This topic provides details about the Arrow services:

• arrow_flight_get: Returns data about a specific flight from the flight server.
• arrow_flight_get_info: Returns metadata about a specific flight.
• arrow_flight_list: Lists all of the available flights known to the flight server.
• arrow_flight_push: Pushes query results directly to the flight server.
• arrow_flight_push_csv: Pushes a CSV file to the flight server.

arrow_flight_get

This service returns data for a specific flight from the flight server.

Syntax

SERVICE http://cambridgesemantics.com/anzograph/inmemory#arrow\_flight\_get ("IP", port, flight_type, "flight_name", "root_cert_file")

Argument	Type	Description
IP	string	Flight server IP address.
port	int	Server port number.
flight_type	int	Flight descriptor's type: 1-PATH, 2-CMD .
flight_name	string	Name of the flight.
root_cert_file	string	Path to a trusted TLS root certificate. For non-TLS operation, specify an empty path.

The function returns the column data for the specified flight.

Example 1

Returns CMD flight data using a custom name.

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# SELECT * WHERE { SERVICE exfun:arrow_flight_get("10.117.2.36", 5005, 2, "tickit_total_price", "/tmp/arrow/keys/root-ca.pem") { } } ORDER BY ?ticketPrice ?numtickets LIMIT 3

ticketPrice | numtickets | totalPrice ------------+------------+------------ 20.000000 | 1 | 20.000000 20.000000 | 1 | 20.000000 20.000000 | 1 | 20.000000

Example 2

Returns PATH flight.

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# SELECT * WHERE { SERVICE exfun:arrow_flight_get("10.117.2.36", 5005, 1, "/tmp/arrow/data/iris.csv", "/tmp/arrow/keys/root-ca.pem") { } } ORDER BY ?sepal_length ?sepal_width ?petal_length ?petal_width LIMIT 3

sepal_length | sepal_width | petal_length | petal_width | variety -------------+-------------+---------------+-------------+-------- 4.300000 | 3.000000 | 1.100000 | 0.100000 | Setosa 4.400000 | 2.900000 | 1.400000 | 0.200000 | Setosa 4.400000 | 3.000000 | 1.300000 | 0.200000 | Setosa

Example 3

Returns CMD flight using the query name.

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# SELECT * WHERE { SERVICE exfun:arrow_flight_get("10.117.2.36", 5005, 2, "SELECT ?ticketPrice ?numtickets ?totalPrice WHERE { ?sell ?ticketPrice; ?numtickets; ?totalPrice . }", "/tmp/arrow/keys/root-ca.pem") { } } ORDER BY ?ticketPrice ?numtickets LIMIT 3

ticketPrice | numtickets | totalPrice -------------+------------+------------ 20.000000 | 1 | 20.000000 20.000000 | 1 | 20.000000 20.000000 | 1 | 20.000000

Example 4

Create a graph from a CMD flight with a custom name.

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# INSERT { graph <flight_graph1> { ?insUri ?ticketPrice; ?numtickets; ?totalPrice. } } WHERE { SERVICE exfun:arrow_flight_get("10.117.2.36", 5005, 2, "tickit_total_price", "/tmp/arrow/keys/root-ca.pem") { } BIND(IRI(CONCAT("http:/CSI.COM/", STRUUID())) as ?insUri) }

To display the graph data, you can run the following query:

SELECT ?ticketPrice ?numtickets ?totalPrice FROM <flight_graph1> WHERE { ?insUri ?ticketPrice; ?numtickets; ?totalPrice. } ORDER BY ?ticketPrice ?numtickets LIMIT 3

ticketPrice | numtickets | totalPrice ------------+------------+------------ 20.000000 | 1 | 20.000000 20.000000 | 1 | 20.000000 20.000000 | 1 | 20.000000

Example 5

Creates a graph from a PATH flight.

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# INSERT { graph <flight_graph2> { ?insUri <sepal_length> ?sepal_length; <sepal_width> ?sepal_width; <petal_length> ?petal_length; <petal_width> ?petal_width; ?variety. } } WHERE { SERVICE exfun:arrow_flight_get("10.117.2.36", 5005, 1, "/tmp/arrow/data/iris.csv", "/tmp/arrow/keys/root-ca.pem") { } BIND(IRI(CONCAT("http:/CSI.COM/", STRUUID())) as ?insUri) }

Example 6

Creates a graph from a CMD flight with a query name.

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# INSERT { graph <flight_graph3> { ?insUri ?ticketPrice; ?numtickets; ?totalPrice. } } WHERE { SERVICE exfun:arrow_flight_get("10.117.2.36", 5005, 2, "SELECT ?ticketPrice ?numtickets ?totalPrice WHERE { ?sell ?ticketPrice; ?numtickets; ?totalPrice . }", "/tmp/arrow/keys/root-ca.pem") { } BIND(IRI(CONCAT("http:/CSI.COM/", STRUUID())) as ?insUri) }

arrow_flight_get_info

This service returns metadata about a specific flight.

Syntax

SERVICE http://cambridgesemantics.com/anzograph/inmemory#arrow\_flight\_get\_info ("IP", port, "type", "path", "root_cert_file")

Argument	Type	Description
IP	string	Flight server IP address.
port	int	Server port number.
type	string
Flight descriptor's type: 1-PATH, 2-CMD .

| | path | string | Flight command or path. | | root_cert_file | string | Path to a trusted TLS root certificate. For non-TLS operation, specify an empty path. |

Returns

Type	Description
string	Flight's schema.
long	Total number of records (rows) in the dataset.
long	Total number of bytes in the dataset.

Example 1

This example returns metadata.

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# SELECT (row_opt_string(?flight_info,0) as ?schema) (row_get_long(?flight_info,1) as ?records) (row_get_long(?flight_info,2) as ?bytes) WHERE { SELECT (exfun:arrow_flight_get_info("10.117.2.36", 5005, 2, "SELECT ?ticketPrice ?numtickets ?totalPrice WHERE { ?sell ?ticketPrice; ?numtickets; ?totalPrice . }", "/tmp/arrow/keys/root-ca.pem") as ?flight_info) }

schema | records | bytes -------------------------------------------------------+---------+--------- ?ticketPrice:double ?numtickets:int ?totalPrice:double | 192497 | 3942600

Example 2

Returns PATH name flight’s metadata.

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# SELECT (row_opt_string(?flight_info,0) as ?schema) (row_get_long(?flight_info,1) as ?records) (row_get_long(?flight_info,2) as ?bytes) WHERE { SELECT (exfun:arrow_flight_get_info("10.117.2.36", 5005, 1, "/tmp/arrow/data/iris.csv", "/tmp/arrow/keys/root-ca.pem") as ?flight_info) }

schema | records | bytes ---------------------------------------------------------------------------------------------------+---------+------- ?sepal_length:double ?sepal_width:double ?petal_length:double ?petal_width:double ?variety:String | 150 | 7368

Example 3

Returns custom name flight’s metadata.

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# SELECT (row_opt_string(?flight_info,0) as ?schema) (row_get_long(?flight_info,1) as ?records) (row_get_long(?flight_info,2) as ?bytes) WHERE { SELECT (exfun:arrow_flight_get_info("10.117.2.36", 5005, 2, "tickit_total_price", "/tmp/arrow/keys/root-ca.pem") as ?flight_info) }

schema | records | bytes --------------------------------------------------------+-----------+--------- ?ticketPrice:double ?numtickets:int ?totalPrice:double | 192497 | 3942600

arrow_flight_list

This service lists all of the available flights known to the flight server.

Syntax

SERVICE http://cambridgesemantics.com/anzograph/inmemory#arrow\_flight\_list ("IP", port, "root_cert_file")

Argument	Type	Description
IP	string	Flight server IP address.
port	int	Server port number.
root_cert_file	string	Path to a trusted TLS root certificate. For non-TLS operation, specify an empty path.

Returns

Type	Description
long	Flight number.
long	Total number of endpoints associated with the flight (dataset).
int	Endpoint number in the flight.
string	Type of flight. CMD-Command, PATH-Path.
string	Name of the flight.
string	List of locations where ticket can be redeemed.

Example

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# SELECT * WHERE { SERVICE exfun:arrow_flight_list("10.117.2.36", 5005, "/tmp/arrow/keys/root-ca.pem") {} }

flight | total_endpoints | endpoint_num | flight_type | flight_name | location --------+-----------------+--------------+-------------+----------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------- 1 | 1 | 1 | PATH | /tmp/arrow/data/iris.csv | grpc+tls://10.117.2.36:5005 2 | 1 | 1 | CMD | tickit_total_price | grpc+tls://10.117.2.36:5005 3 | 1 | 1 | CMD | SELECT ?ticketPrice ?numtickets ?totalPrice WHERE { ?sell ?ticketPrice; ?numtickets; ?totalPrice . } | grpc+tls://10.117.2.36:5005

arrow_flight_push

This service pushes out the result of a query directly to the flight server.

Syntax

SERVICE http://cambridgesemantics.com/anzograph/inmemory#arrow\_flight\_push ("IP", port, "cmd_name", "root_cert_file")

Argument	Type	Description
IP	string	Server IP address.
port	int	Server port number.
cmd_name	string	Flight name that can be a SPARQL command or custom name. Specify "CMD" to use SPARQL command as name, otherwise provide a custom name.
root_cert_file	string	Path to a trusted TLS root certificate. For non-TLS operation, specify an empty path.

Returns

Type	Description
long	The number of rows pushed to the server.
string	Name of the flight.

Example 1

Returns the flight name as user-defined name, "tickit_total_price".

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# SELECT ?num_rows ?flight_name FROM WHERE { { SELECT ?ticketPrice ?numtickets ?totalPrice WHERE { ?sell ?ticketPrice; ?numtickets; ?totalPrice .
} } SERVICE exfun:arrow_flight_push("10.117.2.36", 5005, "tickit_total_price", "/tmp/arrow/keys/root-ca.pem") { } }

num_rows | flight_name ----------+-------------------- 192497 | tickit_total_price

Example 2

This example returns the flight name as a SPARQL query.

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# SELECT ?num_rows ?flight_name FROM WHERE { { SELECT ?ticketPrice ?numtickets ?totalPrice WHERE { ?sell ?ticketPrice; ?numtickets; ?totalPrice . } } SERVICE exfun:arrow_flight_push("10.117.2.36", 5005, "CMD", "/tmp/arrow/keys/root-ca.pem") { } }

num_rows | flight_name ---------+-------------------- 192497 | SELECT ?ticketPrice ?numtickets ?totalPrice WHERE { ?sell ?ticketPrice; ?numtickets; ?totalPrice . }

arrow_flight_push_csv

This service pushes a CSV file to the flight server.

Syntax

SERVICE http://cambridgesemantics.com/anzograph/inmemory#arrow\_flight\_push\_csv ("IP", port, "file", "root_cert_file")

Argument	Type	Description
IP	string	Server IP address.
port	int	Server port number.
file	string	CSV file path.
root_cert_file	string	Path to a trusted TLS root certificate. For non-TLS operation, specify an empty path.

Returns

Type	Description
long	The number of rows pushed to the server.
string	Flight path.

Example

PREFIX exfun: http://cambridgesemantics.com/anzograph/inmemory# SELECT * WHERE { SERVICE exfun:arrow_flight_push_csv("10.117.2.36", 5005, "/tmp/arrow/data/iris.csv", "/tmp/arrow/keys/root-ca.pem") {} }

num_rows | path ----------+-------------------------- 150 | /tmp/arrow/data/iris.csv